Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Myricom MX2G Segmentation fault on OMPI 1.6
From: Yong Qin (yong.qin_at_[hidden])
Date: 2012-06-28 20:04:00


Thanks to Jeff, we now have a bug registered with the segv issue.

Now we are moving to the testing with a myrinet onboard, but we are
also having some unexpected issues.

1. If we don't specify which BTL to use, my understanding is that it
should pick up the mx btl if openib is not available. However it
doesn't look like this is the case. So it falls back to tcp from the
following example.

[yqin_at_n0026.hbar]$ mpirun -np 2 -H n0026.hbar,n0027.hbar ./osu_latency
CMA: no RDMA devices found
--------------------------------------------------------------------------
[[59254,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: n0026.hbar

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
CMA: no RDMA devices found
# OSU MPI Latency Test (Version 2.0)
# Size Latency (us)
0 45.86
1 46.57
2 46.61
4 46.79
...

2. If we specify the mx btl, it spits an error although it seems to
run fine after that.

[yqin_at_n0026.hbar]$ mpirun -np 2 -H n0026.hbar,n0027.hbar -mca btl
mx,sm,self ./osu_latency
mx_finalize() called while some endpoints are still open.
[n0026.hbar:02045] Error in mx_finalize (error Busy)
mx_finalize() called while some endpoints are still open.
[n0027.hbar:02175] Error in mx_finalize (error Busy)
# OSU MPI Latency Test (Version 2.0)
# Size Latency (us)
0 3.85
1 4.27
2 4.27
4 4.31
...

3. If we disable openib btl, it falls back to #2.

[yqin_at_n0026.hbar]$ mpirun -np 2 -H n0026.hbar,n0027.hbar -mca btl
^openib ./osu_latency
mx_finalize() called while some endpoints are still open.
[n0026.hbar:02051] Error in mx_finalize (error Busy)
mx_finalize() called while some endpoints are still open.
[n0027.hbar:02198] Error in mx_finalize (error Busy)
# OSU MPI Latency Test (Version 2.0)
# Size Latency (us)
0 3.84
1 4.24
2 4.25
4 4.29
...

Note this is also different behavior than the old gm btl because we
have always had both btl's built in and the proper one will be picked
up based on which device is actually onboard. Can anybody whether this
is a also a bug, or some parameters that we can set so that it will do
the select automatically? Also that error message "mx_finalize"
doesn't look right either.

Thanks,

Yong

On Fri, Jun 15, 2012 at 6:41 AM, Jeff Squyres <jsquyres_at_[hidden]> wrote:
> On Jun 11, 2012, at 7:48 PM, Yong Qin wrote:
>
>> ah, I guess my original understanding of PML was wrong. Adding "-mca
>> pml ob1" does help to ease the problem.
>
> See the README for a little more discussion about this issue.  There can only be 1 PML in use by a given MPI job -- using "--mca pml ob1" forces the use of the "ob1" PML (i.e., the BTLs), as opposed to the "cm" MTL (i.e., the MTLs).
>
>> But the question still
>> remains. Why ompi decided to use the mx BTL in the first place, given
>> there's no physical device onboard at all? This behavior is completely
>> different than the original gm BTL.
>
> That's not what is actually happening.
>
> Open MPI *built* with MX support, and it therefore assumes that you will likely want to use it.  So it *warns* you when there is no MX device available.
>
> That being said, I have recently run into the issue you are seeing: if OMPI 1.6 warns you that there is no high-speed device available (openib in my case), it then segv's (which it obviously shouldn't -- it should warn and then die gracefully).  I'll open a ticket on this behavior.  It's not a common scenario, but we still shouldn't segv.
>
> My first guess is that this has something to do with the memory manager... but that's a guess.
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users