Globus Toolkit on HP TruCluster systems

The advantages and disadvantages of a Node-specific Installation are as follows:

• + A client may select a specific node for job submission.
• + MDS may be configured to run on every node.
• - Each node requires unique certificates.
• - Each node must be installed to a separate location.
• - No failover for Gatekeeper and GridFTP services.

Before installing Globus on any cluster node, create a CDSL for the "/etc/grid-security" directory as well as one for Globus installation directory (i.e., identified by $GLOBUS_LOCATION). This procedure must be done once, as the superuser, on any cluster node. An example command sequence (where the $GLOBUS_LOCATION variable is installation-dependent) follows:

    $ GLOBUS_LOCATION=/opt/globus/nodespec
    $ mkdir -m 755 -p /etc/grid-security ${GLOBUS_LOCATION}
    $ mkcdsl -a /etc/grid-security
    $ mkcdsl -a ${GLOBUS_LOCATION}

Additionally, when registering an installation through the GTbuild script, the registration directory itself should also be a CDSL. This procedure must be done once, as the superuser, on any cluster node. An example command sequence (where the $GTKREG_DIR variable is installation-dependent) follows:

    $ GTKREG_DIR=/globus
    $ mkdir -m 755 -p ${GTKREG_DIR}
    $ mkcdsl -a ${GTKREG_DIR}

Follow the instructions for a Globus on Multi-homed host installation on every node of the cluster, Note that, since "/etc/inetd.conf" is a shared file, it is only necessary to modify it on one cluster node. However, after making the change, the inetd server must be restarted on every node:

    $ nodes=`clu_get_info -raw | grep '^M.*:UP:' | \
           awk -F: '{ print $3 }'`
    $ for mem in $nodes; do
    >   rsh $mem '/usr/sbin/inetd -h'
    > done

Use the cluster alias for the GCHN and follow the instructions for installing Globus on Multi-homed hosts (i.e., the $GLOBUS_HOSTNAME variable and all certificate requests should use the cluster alias). Next, do the following:

• Edit the "/etc/clua_services" file and add these lines:

      gsigatekeeper   2119/tcp        in_multi,out_alias,static
      gsiftp          2811/tcp        in_multi,out_alias,static
  

  The "in_multi" option allows incoming connections to be handled by any cluster node. The "out_alias" option will ensure that the outgoing IP address is that of the cluster alias (since this is the host name used in the certificate subject) and "static" ensures that the port will not be dynamically assigned.
• On each cluster node, reload the "/etc/clua_services" file ("/usr/sbin/cluamgr -f") and restart the inetd server ("/usr/sbin/inetd -h"). This procedure must be done once, as the superuser, on any cluster node. An example command sequence follows:

      $ nodes=`clu_get_info -raw | grep '^M.*:UP:' | awk -F: '{ print $3 }'`
      $ for mem in $nodes; do
      >   rsh $mem '/usr/sbin/cluamgr -f; sleep 1; /usr/sbin/inetd -h'
      > done
  
  

• Instruct everyone to use the cluster alias to communicate with the Cluster.

As mentioned above, one disadvantage to the Cluster-wide Installation is that there is no way to directly submit a job to a specific node. One could use a job manager (e.g., OpenPBS or LSF) to steer an application, or alternatively, the Globus fork job manager could be coerced into running a job on a specific node by vectoring the submission through the rsh(1) command.

If Globus was installed on a shared disk, the post-install procedure for the setup-globus-gram-job-manager script will display messages of the form:

    Creating state file directory.
    WARNING: It looks like [...]/tmp may not be on a local filesystem.
    WARNING: The test for local file systems is not 100% reliable.
             Ignore the below if this is a false positive.
    WARNING: The jobmanager requires state dir to be on a local filesystem
    WARNING: Rerun the jobmanager setup script with
             the -state-dir= option.

This message can be ignored unless the installation directory is actually being shared with other nodes. If this is the case, follow the instructions above for configuring Globus directories using CDSLs and then run the following command to reconfigure the GRAM job manager:

    $ ${GLOBUS_LOCATION}/setup/globus/setup-globus-gram-job-manager --state-dir=${GLOBUS_LOCATION}/tmp