I wonder if you can try this:
use the PE_HOSTFILE that is generated by the qsh -pe my_pe for tight
integration, so the host names may be the ones without the "x". But then
in your mpirun command, specify the interfaces you would like to
exclude, e.g. all except for the gigE interface. See if that works instead.
-mca btl_tcp_if_exclude lo,all_my_non_gigE_if
Orion Poplawski wrote:
> Pak Lui wrote:
>> Hi Orion and Reuti,
>> Let me see if I can understand the issue by breaking them down first:
>> (1) First, I am curious to know why you would need to create a
>> PE_HOSTFILE yourself, because that file is generated by SGE/N1GE when
>> you specify you are running a parallel job under SGE/N1GE, by doing
>> something like this with qsub/qsh/qrsh, etc:
> In our setup (which I don't believe is very unique) the nodes are
> connected by two networks: an "admin" network which allows for
> connections from outside the cluster and an "MPI" network that is a
> private GigE network connecting the nodes for MPI traffic:
> +---------admin net (192.168.0.X)--------+
> | | |
> +-----------+ +--------+ +--------+
> | SGE Master| | coop00 | | coop01 |
> | | | coop00x| | coop01x|
> +-----------+ +--------+ +--------+
> | |
> MPI net (192.168.1.X)
> So the "x" suffix names are the addresses on the MPI network.
> Currently (loose integration), we create machines files like:
> coop00x.cora.nwra.com cpu=2
> coop01x.cora.nwra.com cpu=2
> which makes the MPI traffic travel over the MPI network. I'm trying to
> duplicate this under "tight" integration.
>> (2) As for the following error message:
>> > error: commlib error: access denied (client IP resolved to host name
>> > "coop01x.cora.nwra.com". This is not identical to clients host name
>> > "coop01.cora.nwra.com")
>> As you mentioned in your setup, each node has 2 interfaces. And this
>> message is an SGE error and it seems to tell you that SGE cannot
>> resolve the host name.
> No, it says that the IP resolved to "coop01x.cora.nwra.com", but the
> machines' primary hostname is "coop01.cora.nwra.com".
>> (4) As for what you have mentioned here:
>> > Now, looking at the OpenMPI gridengine code, it looks like it gets the
>> > node name from the first entry in the pe_hostfile, and never really
>> > the queue name for anything.
>> > ptr = strtok_r(buf, " \n", &tok);
>> > num = strtok_r(NULL, " \n", &tok);
>> > queue = strtok_r(NULL, " \n", &tok);
>> > arch = strtok_r(NULL, " \n", &tok);
>> > ...
>> > node->node_name = strdup(ptr);
>> > node->node_arch = strdup(arch);
>> > Perhaps it can be modified it uses the queue name hostname when doing
>> > SGE/qrsh calls, but the first hostname when doing MPI communication.
>> > Not really sure what the intent of the two fields in SGE's pe_hostfile
>> > is, or if OpenMPI can handle the idea of two hostnames for different
>> > purposes.
>> Once it is in a parallel environment of SGE (e.g. when you have
>> started a parallel job with "qsh/qsub/qrsh -pe name_of_pe"), in ORTE
>> would use the -inherit flag of qrsh to tell qrsh to start a task in a
>> already scheduled parallel job, therefore we cannot assign another
>> queue to the job
> I don't want to assign another queue to the job. Assuming a pe_hostfile
> coop01x.cora.nwra.com 2 coop.q_at_[hidden] <NULL>
> coop00x.cora.nwra.com 2 coop.q_at_[hidden] <NULL>
> I would like ORTE to use "coop01.cora.nwra.com" and
> "coop00.cora.nwra.com" (taken from the hostname parts of the queue
> names) for use with qrsh, and "coop01x.cora.nwra.com" and
> "coop00x.cora.nwra.com" as the hostnames to use for MPI traffic.
> Again, this is just a stab at some way to hack what I want to achieve,
> which is SGE/admin traffic traveling over one network, and MPI traffic
> traveling over another. I don't care how it is done.
> - Orion
- Pak Lui