Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] openib unloaded before last mem dereg
From: Steve Wise (swise_at_[hidden])
Date: 2013-01-29 10:24:04


Tests good on 1.6.3 too.

Thanks Josh!

On 1/29/2013 9:17 AM, Steve Wise wrote:
> I applied it to 1.6.4rc2 and it fixed the seg fault issue. Lemme try
> 1.6.3 too.
>
>
> On 1/29/2013 9:11 AM, Joshua Ladd wrote:
>> It should apply cleanly to 1.6.3 branch, I tested it this morning.
>> From top level OMPI directory do
>>
>> patch -p1 < yossi_openib.patch
>>
>>
>> Best,
>>
>> Josh
>>
>>
>>
>>
>> Josh
>> -----Original Message-----
>> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]]
>> On Behalf Of Jeff Squyres (jsquyres)
>> Sent: Tuesday, January 29, 2013 10:05 AM
>> To: Steve Wise
>> Cc: Open MPI Developers
>> Subject: Re: [OMPI devel] openib unloaded before last mem dereg
>>
>> It's on the ticket that I just assigned to you. :-)
>>
>>
>> On Jan 29, 2013, at 10:03 AM, Steve Wise
>> <swise_at_[hidden]> wrote:
>>
>>> Will do...once I get a patch.
>>>
>>> STeve
>>> On 1/29/2013 7:40 AM, Jeff Squyres (jsquyres) wrote:
>>>> Thanks Josh.
>>>>
>>>> Steve -- if you can confirm that this fixes your problem in the
>>>> v1.6 series, we'll go ahead and commit the patch.
>>>>
>>>> FWIW: the OpenFabrics startup code got a little cleanup/revamp on
>>>> the trunk/v1.7 -- I suspect that's why you're not seeing the
>>>> problem on trunk/v1.7 (e.g., look at the utility routines that were
>>>> abstracted out to ompi/mca/common/verbs).
>>>>
>>>>
>>>>
>>>> On Jan 29, 2013, at 2:41 AM, Joshua Ladd <joshual_at_[hidden]> wrote:
>>>>
>>>>> So, we (Mellanox) have observed this ourselves when no suitable
>>>>> CPC can be found. Seems the BTL associated with this port is not
>>>>> destroyed and the ref count is not decreased. Not sure why you
>>>>> don't see the problem in 1.7. But we have a patch that I'll CMR
>>>>> today. Please review our symptoms, diagnosis, and proposed change.
>>>>> Ralph, maybe I can list you as a reviewer of the patch? I've
>>>>> reviewed myself and it looks fine, but wouldn't mind having
>>>>> another set of eyes on it since I don't want to be responsible for
>>>>> breaking the OpenIB BTL.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Josh Ladd
>>>>>
>>>>>
>>>>> Reported by Yossi:
>>>>> Hi,
>>>>>
>>>>> There is a bug in open mpi (openib component) when one of the
>>>>> active ports is Ethernet.
>>>>> The fix is attached, probably needs to be reviewed and submitted to
>>>>> ompi
>>>>>
>>>>> Error flow:
>>>>> 1. Openib component creates a btl instance for every active
>>>>> port (including Ethernet)
>>>>> 2. Every btl holds a reference count to the device
>>>>> (mca_btl_openib_device_t::btls)
>>>>> 3. Openib tries to create a "connection module" for every btl
>>>>> 4. It fails to create connection module for the Ethernet port
>>>>> 5. The btl for Ethernet port is not returned by openib
>>>>> component, in the list of btl modules
>>>>> 6. The btl for Ethernet port is not destroyed during openib
>>>>> component finalize
>>>>> 7. The device is not destroyed, because of the reference count
>>>>> 8. The memory pool created by the device is not destroyed
>>>>> 9. Later, rdma mpool module cleans up remaining pools during
>>>>> its finalize
>>>>> 10. The memory pool created by openib is destroyed by rdma
>>>>> mpool component finalize
>>>>> 11. The memory pool points to a function (openib_dereg_mr)
>>>>> which is already unloaded from memory (because mca_btl_openib.so
>>>>> was unloaded)
>>>>> 12. Segfault because of a call to invalid function
>>>>>
>>>>> The fix: If a btl module is not going to be returned from openib
>>>>> component init, destroy it.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]]
>>>>> On Behalf Of Ralph Castain
>>>>> Sent: Monday, January 28, 2013 8:35 PM
>>>>> To: Steve Wise
>>>>> Cc: Open MPI Developers
>>>>> Subject: Re: [OMPI devel] openib unloaded before last mem dereg
>>>>>
>>>>> Out of curiosity, could you tell us how you configured OMPI?
>>>>>
>>>>>
>>>>> On Jan 28, 2013, at 12:46 PM, Steve Wise
>>>>> <swise_at_[hidden]> wrote:
>>>>>
>>>>>> On 1/28/2013 2:04 PM, Ralph Castain wrote:
>>>>>>> On Jan 28, 2013, at 11:55 AM, Steve Wise
>>>>>>> <swise_at_[hidden]> wrote:
>>>>>>>
>>>>>>>> Do you know if the rdmacm CPC is really being used for your
>>>>>>>> connection setup (vs other CPCs supported by IB)? Cuz iwarp
>>>>>>>> only supports rdmacm. Maybe that's the difference?
>>>>>>> Dunno for certain, but I expect it is using the OOB cm since I
>>>>>>> didn't direct it to do anything different. Like I said, I
>>>>>>> suspect the problem is that the cluster doesn't have iWARP on it.
>>>>>> Definitely, or it could be the different CPC used for IWvs IB is
>>>>>> tickling the issue.
>>>>>>
>>>>>>>> Steve.
>>>>>>>>
>>>>>>>> On 1/28/2013 1:47 PM, Ralph Castain wrote:
>>>>>>>>> Nope - still works just fine. I didn't receive that warning at
>>>>>>>>> all, and it ran to completion without problem.
>>>>>>>>>
>>>>>>>>> I suspect the problem is that the system I can use just isn't
>>>>>>>>> configured like yours, and so I can't trigger the problem.
>>>>>>>>> Afraid I can't be of help after all... :-(
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jan 28, 2013, at 11:25 AM, Steve Wise
>>>>>>>>> <swise_at_[hidden]> wrote:
>>>>>>>>>
>>>>>>>>>> On 1/28/2013 12:48 PM, Ralph Castain wrote:
>>>>>>>>>>> Hmmm...afraid I cannot replicate this using the current
>>>>>>>>>>> state of the 1.6 branch (which is the 1.6.4rcN) on the only
>>>>>>>>>>> IB-based cluster I can access.
>>>>>>>>>>>
>>>>>>>>>>> Can you try it with a 1.6.4 tarball and see if you still see
>>>>>>>>>>> the problem? Could be someone already fixed it.
>>>>>>>>>> I still hit it on 1.6.4rc2.
>>>>>>>>>>
>>>>>>>>>> Note iWARP != IB so you may not have this issue on IB systems
>>>>>>>>>> for various reasons. Did you use the same mpirun line?
>>>>>>>>>> Namely using this:
>>>>>>>>>>
>>>>>>>>>> --mca btl_openib_ipaddr_include "192.168.170.0/24"
>>>>>>>>>>
>>>>>>>>>> (adjusted to your network config).
>>>>>>>>>>
>>>>>>>>>> Because if I don't use ipaddr_include, then I don't see this
>>>>>>>>>> issue on my setup.
>>>>>>>>>>
>>>>>>>>>> Also, did you see these logged:
>>>>>>>>>>
>>>>>>>>>> Right after starting the job:
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------
>>>>>>>>>> ---
>>>>>>>>>> -------- No OpenFabrics connection schemes reported that they
>>>>>>>>>> were able to be used on a specific port. As such, the openib
>>>>>>>>>> BTL (OpenFabrics
>>>>>>>>>> support) will be disabled for this port.
>>>>>>>>>>
>>>>>>>>>> Local host: hpc-hn1.ogc.int
>>>>>>>>>> Local device: cxgb4_0
>>>>>>>>>> Local port: 2
>>>>>>>>>> CPCs attempted: oob, xoob, rdmacm
>>>>>>>>>> ---------------------------------------------------------------
>>>>>>>>>> ---
>>>>>>>>>> --------
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> At the end of the job:
>>>>>>>>>>
>>>>>>>>>> [hpc-hn1.ogc.int:07850] 5 more processes have sent help message
>>>>>>>>>> help-mpi-btl-openib-cpc-base.txt / no cpcs for port
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think these are benign, but prolly indicate a bug: the
>>>>>>>>>> mpirun is restricting the job to use port 1 only, so the CPCs
>>>>>>>>>> shouldn't be attempting port 2...
>>>>>>>>>>
>>>>>>>>>> Steve.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Jan 28, 2013, at 10:03 AM, Steve Wise
>>>>>>>>>>> <swise_at_[hidden]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On 1/28/2013 11:48 AM, Ralph Castain wrote:
>>>>>>>>>>>>> On Jan 28, 2013, at 9:12 AM, Steve Wise
>>>>>>>>>>>>> <swise_at_[hidden]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 1/25/2013 12:19 PM, Steve Wise wrote:
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm tracking an issue I see in openmpi-1.6.3. Running
>>>>>>>>>>>>>>> this command on my chelsio iwarp/rdma setup causes a seg
>>>>>>>>>>>>>>> fault every time:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /usr/mpi/gcc/openmpi-1.6.3-dbg/bin/mpirun --np 2 --host
>>>>>>>>>>>>>>> hpc-hn1,hpc-cn2 --mca btl openib,sm,self --mca
>>>>>>>>>>>>>>> btl_openib_ipaddr_include "192.168.170.0/24"
>>>>>>>>>>>>>>> /usr/mpi/gcc/openmpi-1.6.3/tests/IMB-3.2/IMB-MPI1 pingpong
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The segfault is during finalization, and I've debugged
>>>>>>>>>>>>>>> this to the point were I see a call to dereg_mem() after
>>>>>>>>>>>>>>> the openib blt is unloaded via dlclose(). dereg_mem()
>>>>>>>>>>>>>>> dereferences a function pointer to call the btl-specific
>>>>>>>>>>>>>>> dereg function, in this case it is openib_dereg_mr().
>>>>>>>>>>>>>>> However, since that btl has already been unloaded, the
>>>>>>>>>>>>>>> deref causes a seg fault. Happens every time with the
>>>>>>>>>>>>>>> above mpi job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Now, I tried this same experiment with openmpi-1.7rc6
>>>>>>>>>>>>>>> and I don't see the seg fault, and I don't see a call to
>>>>>>>>>>>>>>> dereg_mem() after the openib btl is unloaded. That's
>>>>>>>>>>>>>>> all well good. :) But I'd like to get this fix pushed
>>>>>>>>>>>>>>> into 1.6 since that is the current stable release.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Question: Can someone point me to the fix in 1.7?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Steve.
>>>>>>>>>>>>>> It appears that in ompi_mpi_finalize(),
>>>>>>>>>>>>>> mca_pml_base_close() is called which unloads the openib
>>>>>>>>>>>>>> btl. Then further down in ompi_mpi_finalize(),
>>>>>>>>>>>>>> mca_mpool_base_close() is called which ends up calling
>>>>>>>>>>>>>> dereg_mem() which seg faults trying to call into the
>>>>>>>>>>>>>> unloaded openib btl.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> That definitely sounds like a bug
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anybody have thoughts? Anybody care? :)
>>>>>>>>>>>>> I care! It needs to be fixed - I'll take a look. Probably
>>>>>>>>>>>>> something that forgot to be cmr'd.
>>>>>>>>>>>> Great! If you want me to try out a fix or gather more
>>>>>>>>>>>> debug, just hollar.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Steve.
>>>>>>>>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> devel_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> devel_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> --
>> Jeff Squyres
>> jsquyres_at_[hidden]
>> For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel