Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OMPI-1.3.2, openib/iWARP(cxgb3) problem: PML add procs failed (Unreachable)
From: Ken Cain (kcain_at_[hidden])
Date: 2009-05-06 16:45:02


Jon Mason wrote:
> On Wed, May 06, 2009 at 01:20:48PM -0400, Ken Cain wrote:
>> Thanks Jon. I have some responses inline.
>>
>> Jon Mason wrote:
>>> On Wed, May 06, 2009 at 12:15:19PM -0400, Ken Cain wrote:
>>>> I am trying to run NetPIPE-3.7.1 NPmpi using Open MPI version 1.3.2
>>>> with the openib btl in an OFED-1.4 environment. The system
>>>> environment is two Linux (2.6.27) ppc64 blades, each with one
>>>> Chelsio RNIC device, interconnected by a 10GbE switch. The problem
>>>> is that I cannot (using Open MPI) establish connections between the
>>>> two MPI ranks.
>>>>
>>>> I have already read the OMPI FAQ entries and searched for similar
>>>> problems reported to this email list without success. I do have a
>>>> compressed config.log that I can provide separately (it is 80KB in
>>>> size so I'll spare everyone here). I also have the output of
>>>> ompi_info --all that I can share.
>>>>
>>>> I can successfully run small diagnostic programs such as rping,
>>>> ib_rdma_bw, ib_rdma_lat, etc. between the same two blades. I can also
>>>> run NPmpi using another MPI library (MVAPICH2) and the Chelsio/iWARP
>>>> interface.
>>>>
>>>> Here is the one example mpirun command line I used:
>>>> mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self
>>>> --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 >
>>>> outfile1 2>&1
>>>>
>>>> and its output:
>>>>> --------------------------------------------------------------------------
>>>>> No OpenFabrics connection schemes reported that they were able to be
>>>>> used on a specific port. As such, the openib BTL (OpenFabrics
>>>>> support) will be disabled for this port.
>>>>>
>>>>> Local host: aae1
>>>>> Local device: cxgb3_0
>>>>> CPCs attempted: oob, xoob, rdmacm
>>>>> --------------------------------------------------------------------------
>>>>> [snip]
>>>>>
>>>>> Process 1 ([[3115,1],0]) is on host: aae4
>>>>> Process 2 ([[3115,1],1]) is on host: aae1
>>>>> BTLs attempted: self
>>>>>
>>>>> Your MPI job is now going to abort; sorry.
>>>>> --------------------------------------------------------------------------
>>>>> [snip]
>>>>> PML add procs failed
>>>>> --> Returned "Unreachable" (-12) instead of "Success" (0)
>>>>> --------------------------------------------------------------------------
>>>>> *** An error occurred in MPI_Init
>>>>> --------------------------------------------------------------------------
>>>>> [snip]
>>>>> *** An error occurred in MPI_Init
>>>>> *** before MPI was initialized
>>>>> *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
>>>>> *** before MPI was initialized
>>>>> [aae1:6598] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!
>>>>> *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
>>>>> [aae4:19434] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!
>>>>> --------------------------------------------------------------------------
>>>>> mpirun has exited due to process rank 0 with PID 19434 on
>>>>> node aae4 exiting without calling "finalize". This may
>>>>> have caused other processes in the application to be
>>>>> terminated by signals sent by mpirun (as reported here).
>>>>> --------------------------------------------------------------------------
>>>>
>>>> Here is the another mpirun command I used (adding verbosity and more
>>>> specific btl parameters):
>>>>
>>>> mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self,sm
>>>> --mca btl_base_verbose 10 --mca btl_openib_verbose 10 --mca
>>>> btl_openib_if_include cxgb3_0:1 --mca btl_openib_cpc_include rdmacm
>>>> --mca btl_openib_device_type iwarp --mca btl_openib_max_btls 1 --mca
>>>> mpi_leave_pinned 1 --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0
>>>> -l 1 -u 1024 > ~/outfile2 2>&1
>>> It looks like you are only using 1 port on the Chelsio RNIC. Based on
>>> the messages above, It looks like it might be the wrong port. Is there
>>> a reason why you are excluding it? Also, you might try the TCP btl and
>>> verify that it works correctly in the testcase (as a point of
>>> reference).
>>>
>>> Thanks,
>>> Jon
>>>
>> Yes we only have one port connected. The cxgb3 device is associated with
>> eth2 and eth3. Only eth2 is configured with a static IP address. To be
>> sure I didn't choose the wrong OFED device I tried the same command
>> line, changing btl_openib_if_include to cxgb3_0:2 (instead of my
>> original choice cxgb3_0:1). I got the same result in this experiment.
>> The same result occurs when I ask only for cxgb3_0 (no particular port).
>>
>> To test with the TCP btl I changed both of my mpirun commands so that I
>> added tcp to the --mca btl list (keeping openib and the others). In both
>> cases the NPmpi application runs to completion (using a TCP/IP transport
>> not iWARP). In the first (simpler) mpirun command I get the same "No
>> OpenFabrics connection schemes ..." warning message (followed by
>> successful run to completion as noted). In the second mpirun command I
>> get no particular warning messages and run to completion.
>
> Hmm...If you are just adding tcp and keeping openib in the btl grouping,
> it might be possible that it is using the openib btl with the simplier
> commandline. Can you try the simplier commandline without tcp in the
> btl? Also, please show the commands you are running (working and
> non-working).
>

Is it possible for OMPI to generate output at runtime indicating exactly
what btl(s) will be used?

Removing tcp below brings me back to the original simple command line
that fails with the output shown above (indicating that openib btl will
be disabled):

mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self --hostfile
~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024

Adding tcp below allows the application to run (but still indicating in
the output warning messages that the openib btl will be disabled):

mpirun --mca orte_base_help_aggregate 0 --mca btl tcp,openib,self
--hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024

> Is there a reason why the mpirun command you are using is so complex?
>
> Thanks,
> Jon

I guess I just went a little wild while reading the FAQ and the output
of "ompi_info --param btl openib".

>>>> and its output:
>>>>> [snip]
>>>>
>>>> Thanks for any advice/help you can offer.
>>>>
>>>>
>>>> -Ken
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Thanks,
>>
>> -Ken
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-Ken

This message is intended only for the designated recipient(s) and may
contain confidential or proprietary information of Mercury Computer
Systems, Inc. This message is solely intended to facilitate business
discussions and does not constitute an express or implied offer to sell
or purchase any products, services, or support. Any commitments must be
made in writing and signed by duly authorized representatives of each
party.