Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Keith Refson (Keith.lists_at_[hidden])
Date: 2006-07-18 06:20:31


Dear Brian,

Thanks for the help

Brian Barrett wrote:

> > The arguments you want would look like:
> >
> > mpirun -np X -mca btl gm,sm,self -mca btl_base_verbose 1 -mca
> > btl_gm_debug 1 <other arguments>

Aha. I think I had misunderstood the syntax slightly, which explains why
I previously saw no debugging information. I has also omitted the "sm"
btl - though I'm not sure what that one is....

I am now getting some debugging output

[scarf-cn008.rl.ac.uk:04291] [0,1,0] gm_port 017746B0, board 545460846592, global 3712550725 node
180388626433 port 180388626436
[scarf-cn010.rl.ac.uk:13964] [0,1,2] gm_port 017746B0, board 545460846592, global 3712549034 node
180388626433 port 180388626436
[scarf-cn010.rl.ac.uk:13965] [0,1,3] gm_port 017746D0, board 545460846592, global 3712549034 node
180388626433 port 180388626437
[scarf-cn008.rl.ac.uk:04292] [0,1,1] gm_port 017746D0, board 545460846592, global 3712550725 node
180388626433 port 180388626437
[scarf-cn010.rl.ac.uk:13965] [0,1,3] mapped global id 3712550725 to node id 28
[scarf-cn010.rl.ac.uk:13965] [0,1,3] mapped global id 3712550725 to node id 180388626460
[scarf-cn010.rl.ac.uk:13965] [0,1,3] mapped global id 3712549034 to node id 180388626433
[scarf-cn008.rl.ac.uk:04292] [0,1,1] mapped global id 3712550725 to node id 1
[scarf-cn008.rl.ac.uk:04292] [0,1,1] mapped global id 3712549034 to node id 180388626455
[scarf-cn008.rl.ac.uk:04292] [0,1,1] mapped global id 3712549034 to node id 180388626455
[scarf-cn010.rl.ac.uk:13964] [0,1,2] mapped global id 3712550725 to node id 28
[scarf-cn010.rl.ac.uk:13964] [0,1,2] mapped global id 3712550725 to node id 180388626460
[scarf-cn010.rl.ac.uk:13964] [0,1,2] mapped global id 3712549034 to node id 180388626433
[scarf-cn008.rl.ac.uk:04291] [0,1,0] mapped global id 3712550725 to node id 1
[scarf-cn008.rl.ac.uk:04291] [0,1,0] mapped global id 3712549034 to node id 180388626455
[scarf-cn008.rl.ac.uk:04291] [0,1,0] mapped global id 3712549034 to node id 180388626455

which I home means that I am using the GM btl. The run is also about 20% quicker than
before which may suggest that I was not previously using gm.

I have also noticed that if I simply specify --mca btl ^tcp + the debugging options
the run works apparently uses gm, and as quickly. It was (and is) the combination
  -mca btl gm,sm,self,^tcp
that fails with
   No available btl components were found!

> >
>> >> Q3: Is there a way to make openmpi work with the LSF commands? So
>> >> far
>> >> I have constructed a hostfile from the LSF environment variable
>> >> LSB_HOSTS and used the openmpi mpirun command to start the
>> >> parallel executable.
> >
> > Currently, we do not have tight LSF integration for Open MPI, like we
> > do for PBS, SLURM, and BProc. This is mainly because the only LSF
> > machines the development team regularly uses are BProc machines,
> > which do not use the traditional startup and allocation mechanisms of
> > LSF. I believe it is on our feature request list, but I also don't
> > believe we have a timeline for implementation.

OK. It is actually quite easy to construct a hostfile from the LSF
environment and start the processes using the openmpi mpirun command.
I don't know how this will interact with for larger scale usage,
job termination etc but I plan to experiment.

One further question. My run times are still noticably longer than
with mpich_gm. I saw in the mailing list archives that there was
a new implementation of the collective routines in 1.0, (which my application
depends on rather heavil. Is this the default in openmpi 1.1 or is
it still necessary to specify this manually? And if anyone has a comparison
of MPI_AlltoallV performance with other MPI implementations I'd like to
hear the numbers.

Thanks again for all the work. Openmpi looks very promising and it is
definitely the easiest to install and get running of any MPI implementation
I have tried so far.

Keith Refson

-- 
Dr Keith Refson,
Building R3
Rutherford Appleton Laboratory
Chilton
Didcot
Oxfordshire OX11 0QX