Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Problem with openmpi and infiniband
From: Biagio Lucini (B.Lucini_at_[hidden])
Date: 2008-12-23 18:33:43


Hi Dorian,

thank you for your message.

doriankrause wrote:

>>
>> The trouble is with an MPI code that runs fine with an openmpi 1.1.2
>> library compiled without infiniband support (I have tested the
>> scalability of the code up to 64 cores, the nodes are 4 or 8 cores,
>> the results are exactly what I expect), but if I try to use a version
>> compiled for infiniband, then only a subset of comunications (the ones
>> connecting cores in the same node) are enabled, and because of this
>> the program fails (gets stuck in a perennial waiting phase, in
>> particular). This happens with any combination of compilers/library
>> releases (1.1.2, 1.2.7, 1.2.8) I have tried. On other codes, and in
>> particular on benchmarks downloaded from the net, openmpi over
>> infiniband seems to work (I compared the latency with the tcp btl, so
>> I am pretty sure that infiniband works). The two variables I kept
>> fixed are SGE and the OFED module stack. I would like not to touch
>> them, if possible, because the cluster seems to run fine for other
>> purposes.

> Does the problem only show up with openmpi? Did you tried to use mvapich
> (http://mvapich.cse.ohio-state.edu/) to test whether it is a hardware or
> software problem? (I don't know any other open-source MPI implementation
> which supports infiniband)
>

I have had bad experiences with mpich, on which mvapich is based. The
short answer to your question is yes, and it did not work for other
reasons (not even over ethernet). The interesting development today is
that Intel MPI (which should be more or less mvapich2 if I am not wrong)
seems to work (I will verify this also with mvapich2). This seems to
point towards a problem with the OpenMPI libraries, but I have
reservations: they seem to work for even complicated benchmarking tests
(like the Intel Benchmark) AND I have troubles also with mpich, which I
did not sort out. A possibility is that the problem is generated by the
interaction MPI-SGE-my code. I would love if someone more experienced
than me would give a look at the code (which unfortunately is fortran).
I will try to trim down the over 4000 lines to a manageable proof of
concept, if anyone is interested in following this up, but it is
unlikely to happen before new year :-)

Thanks again,
Biagio