Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] [openib] segfault when using openib btl
From: Eloi Gaudry (eg_at_[hidden])
Date: 2010-09-24 10:54:06


Terry,

You were right, the error indeed seems to come from the message coalescing feature.
If I turn it off using the "--mca btl_openib_use_message_coalescing 0", I'm not able to observe the "hdr->tag=0" error.

There are some trac requests associated to very similar error (https://svn.open-mpi.org/trac/ompi/search?q=coalescing) but they are all closed (except https://svn.open-mpi.org/trac/ompi/ticket/2352
that might be related), aren't they ? What would you suggest Terry ?

Eloi

On Friday 24 September 2010 16:00:26 Terry Dontje wrote:
> Eloi Gaudry wrote:
> > Terry,
> >
> > No, I haven't tried any other values than P,65536,256,192,128 yet.
> >
> > The reason why is quite simple. I've been reading and reading again this
> > thread to understand the btl_openib_receive_queues meaning and I can't
> > figure out why the default values seem to induce the hdr-
> >
> >> tag=0 issue
> >> (http://www.open-mpi.org/community/lists/users/2009/01/7808.php).
>
> Yeah, the size of the fragments and number of them really should not
> cause this issue. So I too am a little perplexed about it.
>
> > Do you think that the default shared received queue parameters are
> > erroneous for this specific Mellanox card ? Any help on finding the
> > proper parameters would actually be much appreciated.
>
> I don't necessarily think it is the queue size for a specific card but
> more so the handling of the queues by the BTL when using certain sizes.
> At least that is one gut feel I have.
>
> In my mind the tag being 0 is either something below OMPI is polluting
> the data fragment or OMPI's internal protocol is some how getting messed
> up. I can imagine (no empirical data here) the queue sizes could change
> how the OMPI protocol sets things up. Another thing may be the
> coalescing feature in the openib BTL which tries to gang multiple
> messages into one packet when resources are running low. I can see
> where changing the queue sizes might affect the coalescing. So, it
> might be interesting to turn off the coalescing. You can do that by
> setting "--mca btl_openib_use_message_coalescing 0" in your mpirun line.
>
> If that doesn't solve the issue then obviously there must be something
> else going on :-).
>
> Note, the reason I am interested in this is I am seeing a similar error
> condition (hdr->tag == 0) on a development system. Though my failing
> case fails with np=8 using the connectivity test program which is mainly
> point to point and there are not a significant amount of data transfers
> going on either.
>
> --td
>
> > Eloi
> >
> > On Friday 24 September 2010 14:27:07 you wrote:
> >> That is interesting. So does the number of processes affect your runs
> >> any. The times I've seen hdr->tag be 0 usually has been due to protocol
> >> issues. The tag should never be 0. Have you tried to do other
> >> receive_queue settings other than the default and the one you mention.
> >>
> >> I wonder if you did a combination of the two receive queues causes a
> >> failure or not. Something like
> >>
> >> P,128,256,192,128:P,65536,256,192,128
> >>
> >> I am wondering if it is the first queuing definition causing the issue
> >> or possibly the SRQ defined in the default.
> >>
> >> --td
> >>
> >> Eloi Gaudry wrote:
> >>> Hi Terry,
> >>>
> >>> The messages being send/received can be of any size, but the error
> >>> seems to happen more often with small messages (as an int being
> >>> broadcasted or allreduced). The failing communication differs from one
> >>> run to another, but some spots are more likely to be failing than
> >>> another. And as far as I know, there are always located next to a
> >>> small message (an int being broadcasted for instance) communication.
> >>> Other typical messages size are
> >>>
> >>>> 10k but can be very much larger.
> >>>
> >>> I've been checking the hca being used, its' from mellanox (with
> >>> vendor_part_id=26428). There is no receive_queues parameters associated
> >>> to it.
> >>>
> >>> $ cat share/openmpi/mca-btl-openib-device-params.ini as well:
> >>> [...]
> >>>
> >>> # A.k.a. ConnectX
> >>> [Mellanox Hermon]
> >>> vendor_id = 0x2c9,0x5ad,0x66a,0x8f1,0x1708,0x03ba,0x15b3
> >>> vendor_part_id =
> >>> 25408,25418,25428,26418,26428,25448,26438,26448,26468,26478,26488
> >>> use_eager_rdma = 1
> >>> mtu = 2048
> >>> max_inline_data = 128
> >>>
> >>> [..]
> >>>
> >>> $ ompi_info --param btl openib --parsable | grep receive_queues
> >>>
> >>> mca:btl:openib:param:btl_openib_receive_queues:value:P,128,256,192,128
> >>> :S ,2048,256,128,32:S,12288,256,128,32:S,65536,256,128,32
> >>> mca:btl:openib:param:btl_openib_receive_queues:data_source:default
> >>> value mca:btl:openib:param:btl_openib_receive_queues:status:writable
> >>> mca:btl:openib:param:btl_openib_receive_queues:help:Colon-delimited,
> >>> comma delimited list of receive queues: P,4096,8,6,4:P,32768,8,6,4
> >>> mca:btl:openib:param:btl_openib_receive_queues:deprecated:no
> >>>
> >>> I was wondering if these parameters (automatically computed at openib
> >>> btl init for what I understood) were not incorrect in some way and I
> >>> plugged some others values: "P,65536,256,192,128" (someone on the list
> >>> used that values when encountering a different issue) . Since that, I
> >>> haven't been able to observe the segfault (occuring as hrd->tag = 0 in
> >>> btl_openib_component.c:2881) yet.
> >>>
> >>> Eloi
> >>>
> >>>
> >>> /home/pp_fr/st03230/EG/Softs/openmpi-custom-1.4.2/bin/
> >>>
> >>> On Thursday 23 September 2010 23:33:48 Terry Dontje wrote:
> >>>> Eloi, I am curious about your problem. Can you tell me what size of
> >>>> job it is? Does it always fail on the same bcast, or same process?
> >>>>
> >>>> Eloi Gaudry wrote:
> >>>>> Hi Nysal,
> >>>>>
> >>>>> Thanks for your suggestions.
> >>>>>
> >>>>> I'm now able to get the checksum computed and redirected to stdout,
> >>>>> thanks (I forgot the "-mca pml_base_verbose 5" option, you were
> >>>>> right). I haven't been able to observe the segmentation fault (with
> >>>>> hdr->tag=0) so far (when using pml csum) but I 'll let you know when
> >>>>> I am.
> >>>>>
> >>>>> I've got two others question, which may be related to the error
> >>>>> observed:
> >>>>>
> >>>>> 1/ does the maximum number of MPI_Comm that can be handled by OpenMPI
> >>>>> somehow depends on the btl being used (i.e. if I'm using openib, may
> >>>>> I use the same number of MPI_Comm object as with tcp) ? Is there
> >>>>> something as MPI_COMM_MAX in OpenMPI ?
> >>>>>
> >>>>> 2/ the segfaults only appears during a mpi collective call, with very
> >>>>> small message (one int is being broadcast, for instance) ; i followed
> >>>>> the guidelines given at http://icl.cs.utk.edu/open-
> >>>>> mpi/faq/?category=openfabrics#ib-small-message-rdma but the
> >>>>> debug-build of OpenMPI asserts if I use a different min-size that
> >>>>> 255. Anyway, if I deactivate eager_rdma, the segfaults remains. Does
> >>>>> the openib btl handle very small message differently (even with
> >>>>> eager_rdma
> >>>>> deactivated) than tcp ?
> >>>>
> >>>> Others on the list does coalescing happen with non-eager_rdma? If so
> >>>> then that would possibly be one difference between the openib btl and
> >>>> tcp aside from the actual protocol used.
> >>>>
> >>>>> is there a way to make sure that large messages and small messages
> >>>>> are handled the same way ?
> >>>>
> >>>> Do you mean so they all look like eager messages? How large of
> >>>> messages are we talking about here 1K, 1M or 10M?
> >>>>
> >>>> --td
> >>>>
> >>>>> Regards,
> >>>>> Eloi
> >>>>>
> >>>>> On Friday 17 September 2010 17:57:17 Nysal Jan wrote:
> >>>>>> Hi Eloi,
> >>>>>> Create a debug build of OpenMPI (--enable-debug) and while running
> >>>>>> with the csum PML add "-mca pml_base_verbose 5" to the command line.
> >>>>>> This will print the checksum details for each fragment sent over the
> >>>>>> wire. I'm guessing it didnt catch anything because the BTL failed.
> >>>>>> The checksum verification is done in the PML, which the BTL calls
> >>>>>> via a callback function. In your case the PML callback is never
> >>>>>> called because the hdr->tag is invalid. So enabling checksum
> >>>>>> tracing also might not be of much use. Is it the first Bcast that
> >>>>>> fails or the nth Bcast and what is the message size? I'm not sure
> >>>>>> what could be the problem at this moment. I'm afraid you will have
> >>>>>> to debug the BTL to find out more.
> >>>>>>
> >>>>>> --Nysal
> >>>>>>
> >>>>>> On Fri, Sep 17, 2010 at 4:39 PM, Eloi Gaudry <eg_at_[hidden]> wrote:
> >>>>>>> Hi Nysal,
> >>>>>>>
> >>>>>>> thanks for your response.
> >>>>>>>
> >>>>>>> I've been unable so far to write a test case that could illustrate
> >>>>>>> the hdr->tag=0 error.
> >>>>>>> Actually, I'm only observing this issue when running an internode
> >>>>>>> computation involving infiniband hardware from Mellanox (MT25418,
> >>>>>>> ConnectX IB DDR, PCIe 2.0
> >>>>>>> 2.5GT/s, rev a0) with our time-domain software.
> >>>>>>>
> >>>>>>> I checked, double-checked, and rechecked again every MPI use
> >>>>>>> performed during a parallel computation and I couldn't find any
> >>>>>>> error so far. The fact that the very
> >>>>>>> same parallel computation run flawlessly when using tcp (and
> >>>>>>> disabling openib support) might seem to indicate that the issue is
> >>>>>>> somewhere located inside the
> >>>>>>> openib btl or at the hardware/driver level.
> >>>>>>>
> >>>>>>> I've just used the "-mca pml csum" option and I haven't seen any
> >>>>>>> related messages (when hdr->tag=0 and the segfaults occurs).
> >>>>>>> Any suggestion ?
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Eloi
> >>>>>>>
> >>>>>>> On Friday 17 September 2010 16:03:34 Nysal Jan wrote:
> >>>>>>>> Hi Eloi,
> >>>>>>>> Sorry for the delay in response. I haven't read the entire email
> >>>>>>>> thread, but do you have a test case which can reproduce this
> >>>>>>>> error? Without that it will be difficult to nail down the cause.
> >>>>>>>> Just to clarify, I do not work for an iwarp vendor. I can
> >>>>>>>> certainly try to reproduce it on an IB system. There is also a
> >>>>>>>> PML called csum, you can use it via "-mca pml csum", which will
> >>>>>>>> checksum the MPI messages and verify it at the receiver side for
> >>>>>>>> any data
> >>>>>>>> corruption. You can try using it to see if it is able
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>>> catch anything.
> >>>>>>>>
> >>>>>>>> Regards
> >>>>>>>> --Nysal
> >>>>>>>>
> >>>>>>>> On Thu, Sep 16, 2010 at 3:48 PM, Eloi Gaudry <eg_at_[hidden]> wrote:
> >>>>>>>>> Hi Nysal,
> >>>>>>>>>
> >>>>>>>>> I'm sorry to intrrupt, but I was wondering if you had a chance to
> >>>>>>>>> look
> >>>>>>>
> >>>>>>> at
> >>>>>>>
> >>>>>>>>> this error.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Eloi
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Eloi Gaudry
> >>>>>>>>>
> >>>>>>>>> Free Field Technologies
> >>>>>>>>> Company Website: http://www.fft.be
> >>>>>>>>> Company Phone: +32 10 487 959
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ---------- Forwarded message ----------
> >>>>>>>>> From: Eloi Gaudry <eg_at_[hidden]>
> >>>>>>>>> To: Open MPI Users <users_at_[hidden]>
> >>>>>>>>> Date: Wed, 15 Sep 2010 16:27:43 +0200
> >>>>>>>>> Subject: Re: [OMPI users] [openib] segfault when using openib btl
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> I was wondering if anybody got a chance to have a look at this
> >>>>>>>>> issue.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Eloi
> >>>>>>>>>
> >>>>>>>>> On Wednesday 18 August 2010 09:16:26 Eloi Gaudry wrote:
> >>>>>>>>>> Hi Jeff,
> >>>>>>>>>>
> >>>>>>>>>> Please find enclosed the output (valgrind.out.gz) from
> >>>>>>>>>> /opt/openmpi-debug-1.4.2/bin/orterun -np 2 --host pbn11,pbn10
> >>>>>>>>>> --mca
> >>>>>>>
> >>>>>>> btl
> >>>>>>>
> >>>>>>>>>> openib,self --display-map --verbose --mca mpi_warn_on_fork 0
> >>>>>>>>>> --mca btl_openib_want_fork_support 0 -tag-output
> >>>>>>>>>> /opt/valgrind-3.5.0/bin/valgrind --tool=memcheck
> >>>>>>>>>> --suppressions=/opt/openmpi-debug-1.4.2/share/openmpi/openmpi-
> >>>>>>>>>> valgrind.supp --suppressions=./suppressions.python.supp
> >>>>>>>>>> /opt/actran/bin/actranpy_mp ...
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Eloi
> >>>>>>>>>>
> >>>>>>>>>> On Tuesday 17 August 2010 09:32:53 Eloi Gaudry wrote:
> >>>>>>>>>>> On Monday 16 August 2010 19:14:47 Jeff Squyres wrote:
> >>>>>>>>>>>> On Aug 16, 2010, at 10:05 AM, Eloi Gaudry wrote:
> >>>>>>>>>>>>> I did run our application through valgrind but it couldn't
> >>>>>>>>>>>>> find any "Invalid write": there is a bunch of "Invalid read"
> >>>>>>>>>>>>> (I'm using
> >>>>>>>>>
> >>>>>>>>> 1.4.2
> >>>>>>>>>
> >>>>>>>>>>>>> with the suppression file), "Use of uninitialized bytes" and
> >>>>>>>>>>>>> "Conditional jump depending on uninitialized bytes" in
> >>>>>>>
> >>>>>>> different
> >>>>>>>
> >>>>>>>>> ompi
> >>>>>>>>>
> >>>>>>>>>>>>> routines. Some of them are located in btl_openib_component.c.
> >>>>>>>>>>>>> I'll send you an output of valgrind shortly.
> >>>>>>>>>>>>
> >>>>>>>>>>>> A lot of them in btl_openib_* are to be expected --
> >>>>>>>>>>>> OpenFabrics uses OS-bypass methods for some of its memory,
> >>>>>>>>>>>> and therefore valgrind is unaware of them (and therefore
> >>>>>>>>>>>> incorrectly marks them as
> >>>>>>>>>>>> uninitialized).
> >>>>>>>>>>>
> >>>>>>>>>>> would it help if i use the upcoming 1.5 version of openmpi ? i
> >>>>>>>
> >>>>>>> read
> >>>>>>>
> >>>>>>>>> that
> >>>>>>>>>
> >>>>>>>>>>> a huge effort has been done to clean-up the valgrind output ?
> >>>>>>>>>>> but maybe that this doesn't concern this btl (for the reasons
> >>>>>>>>>>> you mentionned).
> >>>>>>>>>>>
> >>>>>>>>>>>>> Another question, you said that the callback function pointer
> >>>>>>>>>
> >>>>>>>>> should
> >>>>>>>>>
> >>>>>>>>>>>>> never be 0. But can the tag be null (hdr->tag) ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> The tag is not a pointer -- it's just an integer.
> >>>>>>>>>>>
> >>>>>>>>>>> I was worrying that its value could not be null.
> >>>>>>>>>>>
> >>>>>>>>>>> I'll send a valgrind output soon (i need to build libpython
> >>>>>>>>>>> without pymalloc first).
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Eloi
> >>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for your help,
> >>>>>>>>>>>>> Eloi
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 16/08/2010 18:22, Jeff Squyres wrote:
> >>>>>>>>>>>>>> Sorry for the delay in replying.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Odd; the values of the callback function pointer should
> >>>>>>>>>>>>>> never
> >>>>>>>
> >>>>>>> be
> >>>>>>>
> >>>>>>>>> 0.
> >>>>>>>>>
> >>>>>>>>>>>>>> This seems to suggest some kind of memory corruption is
> >>>>>>>>>>>>>> occurring.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I don't know if it's possible, because the stack trace looks
> >>>>>>>>>>>>>> like you're calling through python, but can you run this
> >>>>>>>>>>>>>> application through valgrind, or some other memory-checking
> >>>>>>>>>>>>>> debugger?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Aug 10, 2010, at 7:15 AM, Eloi Gaudry wrote:
> >>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> sorry, i just forgot to add the values of the function
> >>>>>>>>>
> >>>>>>>>> parameters:
> >>>>>>>>>>>>>>> (gdb) print reg->cbdata
> >>>>>>>>>>>>>>> $1 = (void *) 0x0
> >>>>>>>>>>>>>>> (gdb) print openib_btl->super
> >>>>>>>>>>>>>>> $2 = {btl_component = 0x2b341edd7380, btl_eager_limit =
> >>>>>>>
> >>>>>>> 12288,
> >>>>>>>
> >>>>>>>>>>>>>>> btl_rndv_eager_limit = 12288, btl_max_send_size = 65536,
> >>>>>>>>>>>>>>> btl_rdma_pipeline_send_length = 1048576,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> btl_rdma_pipeline_frag_size = 1048576,
> >>>>>>>>>
> >>>>>>>>> btl_min_rdma_pipeline_size
> >>>>>>>>>
> >>>>>>>>>>>>>>> = 1060864, btl_exclusivity = 1024, btl_latency = 10,
> >>>>>>>>>>>>>>> btl_bandwidth = 800, btl_flags = 310, btl_add_procs =
> >>>>>>>>>>>>>>> 0x2b341eb8ee47<mca_btl_openib_add_procs>, btl_del_procs =
> >>>>>>>>>>>>>>> 0x2b341eb90156<mca_btl_openib_del_procs>, btl_register =
> >>>>>>>>>>>>>>> 0, btl_finalize =
> >>>>>>>>>>>>>>> 0x2b341eb93186<mca_btl_openib_finalize>,
> >>>>>>>>>
> >>>>>>>>> btl_alloc
> >>>>>>>>>
> >>>>>>>>>>>>>>> = 0x2b341eb90a3e<mca_btl_openib_alloc>, btl_free =
> >>>>>>>>>>>>>>> 0x2b341eb91400<mca_btl_openib_free>, btl_prepare_src =
> >>>>>>>>>>>>>>> 0x2b341eb91813<mca_btl_openib_prepare_src>,
> >>>>>>>>>>>>>>> btl_prepare_dst
> >>>>>>>
> >>>>>>> =
> >>>>>>>
> >>>>>>>>>>>>>>> 0x2b341eb91f2e<mca_btl_openib_prepare_dst>, btl_send =
> >>>>>>>>>>>>>>> 0x2b341eb94517<mca_btl_openib_send>, btl_sendi =
> >>>>>>>>>>>>>>> 0x2b341eb9340d<mca_btl_openib_sendi>, btl_put =
> >>>>>>>>>>>>>>> 0x2b341eb94660<mca_btl_openib_put>, btl_get =
> >>>>>>>>>>>>>>> 0x2b341eb94c4e<mca_btl_openib_get>, btl_dump =
> >>>>>>>>>>>>>>> 0x2b341acd45cb<mca_btl_base_dump>, btl_mpool = 0xf3f4110,
> >>>>>>>>>>>>>>> btl_register_error =
> >>>>>>>>>>>>>>> 0x2b341eb90565<mca_btl_openib_register_error_cb>,
> >>>>>>>>>>>>>>> btl_ft_event
> >>>>>>>>>
> >>>>>>>>> =
> >>>>>>>>>
> >>>>>>>>>>>>>>> 0x2b341eb952e7<mca_btl_openib_ft_event>}
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> (gdb) print hdr->tag
> >>>>>>>>>>>>>>> $3 = 0 '\0'
> >>>>>>>>>>>>>>> (gdb) print des
> >>>>>>>>>>>>>>> $4 = (mca_btl_base_descriptor_t *) 0xf4a6700
> >>>>>>>>>>>>>>> (gdb) print reg->cbfunc
> >>>>>>>>>>>>>>> $5 = (mca_btl_base_module_recv_cb_fn_t) 0
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Eloi
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tuesday 10 August 2010 16:04:08 Eloi Gaudry wrote:
> >>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Here is the output of a core file generated during a
> >>>>>>>>>
> >>>>>>>>> segmentation
> >>>>>>>>>
> >>>>>>>>>>>>>>>> fault observed during a collective call (using openib):
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> #0 0x0000000000000000 in ?? ()
> >>>>>>>>>>>>>>>> (gdb) where
> >>>>>>>>>>>>>>>> #0 0x0000000000000000 in ?? ()
> >>>>>>>>>>>>>>>> #1 0x00002aedbc4e05f4 in btl_openib_handle_incoming
> >>>>>>>>>>>>>>>> (openib_btl=0x1902f9b0, ep=0x1908a1c0, frag=0x190d9700,
> >>>>>>>>>>>>>>>> byte_len=18) at btl_openib_component.c:2881 #2
> >>>>>>>>>>>>>>>> 0x00002aedbc4e25e2 in handle_wc (device=0x19024ac0, cq=0,
> >>>>>>>>>>>>>>>> wc=0x7ffff279ce90) at
> >>>>>>>>>>>>>>>> btl_openib_component.c:3178 #3 0x00002aedbc4e2e9d in
> >>>>>>>>>
> >>>>>>>>> poll_device
> >>>>>>>>>
> >>>>>>>>>>>>>>>> (device=0x19024ac0, count=2) at
> >>>>>>>>>>>>>>>> btl_openib_component.c:3318
> >>>>>>>
> >>>>>>> #4
> >>>>>>>
> >>>>>>>>>>>>>>>> 0x00002aedbc4e34b8 in progress_one_device
> >>>>>>>
> >>>>>>> (device=0x19024ac0)
> >>>>>>>
> >>>>>>>>>>>>>>>> at btl_openib_component.c:3426 #5 0x00002aedbc4e3561 in
> >>>>>>>>>>>>>>>> btl_openib_component_progress () at
> >>>>>>>>>>>>>>>> btl_openib_component.c:3451
> >>>>>>>>>
> >>>>>>>>> #6
> >>>>>>>>>
> >>>>>>>>>>>>>>>> 0x00002aedb8b22ab8 in opal_progress () at
> >>>>>>>>>>>>>>>> runtime/opal_progress.c:207 #7 0x00002aedb859f497 in
> >>>>>>>>>>>>>>>> opal_condition_wait (c=0x2aedb888ccc0, m=0x2aedb888cd20)
> >>>>>>>>>>>>>>>> at ../opal/threads/condition.h:99 #8
> >>>>>>>>>>>>>>>> 0x00002aedb859fa31 in ompi_request_default_wait_all
> >>>>>>>
> >>>>>>> (count=2,
> >>>>>>>
> >>>>>>>>>>>>>>>> requests=0x7ffff279d0e0, statuses=0x0) at
> >>>>>>>>>>>>>>>> request/req_wait.c:262 #9 0x00002aedbd7559ad in
> >>>>>>>>>>>>>>>> ompi_coll_tuned_allreduce_intra_recursivedoubling
> >>>>>>>>>>>>>>>> (sbuf=0x7ffff279d444, rbuf=0x7ffff279d440, count=1,
> >>>>>>>>>>>>>>>> dtype=0x6788220, op=0x6787a20,
> >>>>>>>>>>>>>>>> comm=0x19d81ff0, module=0x19d82b20) at
> >>>>>>>>>
> >>>>>>>>> coll_tuned_allreduce.c:223
> >>>>>>>>>
> >>>>>>>>>>>>>>>> #10 0x00002aedbd7514f7 in
> >>>>>>>>>>>>>>>> ompi_coll_tuned_allreduce_intra_dec_fixed
> >>>>>>>>>>>>>>>> (sbuf=0x7ffff279d444, rbuf=0x7ffff279d440, count=1,
> >>>>>>>>>>>>>>>> dtype=0x6788220, op=0x6787a20, comm=0x19d81ff0,
> >>>>>>>>>>>>>>>> module=0x19d82b20) at
> >>>>>>>>>>>>>>>> coll_tuned_decision_fixed.c:63
> >>>>>>>>>>>>>>>> #11 0x00002aedb85c7792 in PMPI_Allreduce
> >>>>>>>>>
> >>>>>>>>> (sendbuf=0x7ffff279d444,
> >>>>>>>>>
> >>>>>>>>>>>>>>>> recvbuf=0x7ffff279d440, count=1, datatype=0x6788220,
> >>>>>>>>>
> >>>>>>>>> op=0x6787a20,
> >>>>>>>>>
> >>>>>>>>>>>>>>>> comm=0x19d81ff0) at pallreduce.c:102 #12
> >>>>>>>>>>>>>>>> 0x0000000004387dbf
> >>>>>>>
> >>>>>>> in
> >>>>>>>
> >>>>>>>>>>>>>>>> FEMTown::MPI::Allreduce (sendbuf=0x7ffff279d444,
> >>>>>>>>>>>>>>>> recvbuf=0x7ffff279d440, count=1, datatype=0x6788220,
> >>>>>>>>>
> >>>>>>>>> op=0x6787a20,
> >>>>>>>>>
> >>>>>>>>>>>>>>>> comm=0x19d81ff0) at stubs.cpp:626 #13 0x0000000004058be8
> >>>>>>>>>>>>>>>> in FEMTown::Domain::align (itf=
> >>>>>>>>>
> >>>>>>>>> {<FEMTown::Boost::shared_base_ptr<FEMTown::Domain::Int
> >>>>>>>>>
> >>>>>>>>>>>>>>>> er fa ce>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> = {_vptr.shared_base_ptr = 0x7ffff279d620, ptr_ = {px =
> >>>>>>>>>>>>>>>> 0x199942a4, pn = {pi_ = 0x6}}},<No data fields>}) at
> >>>>>>>>>>>>>>>> interface.cpp:371 #14 0x00000000040cb858 in
> >>>>>>>>>>>>>>>> FEMTown::Field::detail::align_itfs_and_neighbhors (dim=2,
> >>>>>>>>>
> >>>>>>>>> set={px
> >>>>>>>>>
> >>>>>>>>>>>>>>>> = 0x7ffff279d780, pn = {pi_ = 0x2f279d640}},
> >>>>>>>>>>>>>>>> check_info=@0x7ffff279d7f0) at check.cpp:63 #15
> >>>>>>>>>
> >>>>>>>>> 0x00000000040cbfa8
> >>>>>>>>>
> >>>>>>>>>>>>>>>> in FEMTown::Field::align_elements (set={px =
> >>>>>>>>>>>>>>>> 0x7ffff279d950, pn
> >>>>>>>>>
> >>>>>>>>> =
> >>>>>>>>>
> >>>>>>>>>>>>>>>> {pi_ = 0x66e08d0}}, check_info=@0x7ffff279d7f0) at
> >>>>>>>>>>>>>>>> check.cpp:159 #16 0x00000000039acdd4 in
> >>>>>>>>>>>>>>>> PyField_align_elements (self=0x0, args=0x2aaab0765050,
> >>>>>>>>>>>>>>>> kwds=0x19d2e950) at check.cpp:31 #17 0x0000000001fbf76d in
> >>>>>>>>>>>>>>>> FEMTown::Main::ExErrCatch<_object* (*)(_object*, _object*,
> >>>>>>>>>>>>>>>> _object*)>::exec<_object>
> >>>>>>>>>>>>>>>> (this=0x7ffff279dc20, s=0x0, po1=0x2aaab0765050,
> >>>>>>>>>>>>>>>> po2=0x19d2e950) at
> >>>>>>>>>>>>>>>> /home/qa/svntop/femtown/modules/main/py/exception.hpp:463
> >>>>>>>
> >>>>>>> #18
> >>>>>>>
> >>>>>>>>>>>>>>>> 0x00000000039acc82 in PyField_align_elements_ewrap
> >>>>>>>
> >>>>>>> (self=0x0,
> >>>>>>>
> >>>>>>>>>>>>>>>> args=0x2aaab0765050, kwds=0x19d2e950) at check.cpp:39 #19
> >>>>>>>>>>>>>>>> 0x00000000044093a0 in PyEval_EvalFrameEx (f=0x19b52e90,
> >>>>>>>>>>>>>>>> throwflag=<value optimized out>) at Python/ceval.c:3921
> >>>>>>>>>>>>>>>> #20 0x000000000440aae9 in PyEval_EvalCodeEx
> >>>>>>>>>>>>>>>> (co=0x2aaab754ad50, globals=<value optimized out>,
> >>>>>>>>>>>>>>>> locals=<value optimized out>, args=0x3, argcount=1,
> >>>>>>>>>>>>>>>> kws=0x19ace4a0, kwcount=2,
> >>>>>>>>>>>>>>>> defs=0x2aaab75e4800, defcount=2, closure=0x0) at
> >>>>>>>>>>>>>>>> Python/ceval.c:2968
> >>>>>>>>>>>>>>>> #21 0x0000000004408f58 in PyEval_EvalFrameEx
> >>>>>>>>>>>>>>>> (f=0x19ace2d0, throwflag=<value optimized out>) at
> >>>>>>>>>>>>>>>> Python/ceval.c:3802 #22 0x000000000440aae9 in
> >>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab7550120, globals=<value
> >>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, args=0x7,
> >>>>>>>>>>>>>>>> argcount=1, kws=0x19acc418, kwcount=3,
> >>>>>>>>>>>>>>>> defs=0x2aaab759e958, defcount=6, closure=0x0) at
> >>>>>>>>>>>>>>>> Python/ceval.c:2968
> >>>>>>>>>>>>>>>> #23 0x0000000004408f58 in PyEval_EvalFrameEx
> >>>>>>>>>>>>>>>> (f=0x19acc1c0, throwflag=<value optimized out>) at
> >>>>>>>>>>>>>>>> Python/ceval.c:3802 #24 0x000000000440aae9 in
> >>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab8b5e738, globals=<value
> >>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, args=0x6,
> >>>>>>>>>>>>>>>> argcount=1, kws=0x19abd328, kwcount=5,
> >>>>>>>>>>>>>>>> defs=0x2aaab891b7e8, defcount=3, closure=0x0) at
> >>>>>>>>>>>>>>>> Python/ceval.c:2968
> >>>>>>>>>>>>>>>> #25 0x0000000004408f58 in PyEval_EvalFrameEx
> >>>>>>>>>>>>>>>> (f=0x19abcea0, throwflag=<value optimized out>) at
> >>>>>>>>>>>>>>>> Python/ceval.c:3802 #26 0x000000000440aae9 in
> >>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab3eb4198, globals=<value
> >>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, args=0xb,
> >>>>>>>>>>>>>>>> argcount=1, kws=0x19a89df0, kwcount=10, defs=0x0,
> >>>>>>>>>>>>>>>> defcount=0, closure=0x0) at Python/ceval.c:2968
> >>>>>>>>>>>>>>>> #27 0x0000000004408f58 in PyEval_EvalFrameEx
> >>>>>>>>>>>>>>>> (f=0x19a89c40, throwflag=<value optimized out>) at
> >>>>>>>>>>>>>>>> Python/ceval.c:3802 #28 0x000000000440aae9 in
> >>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab3eb4288, globals=<value
> >>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, args=0x1,
> >>>>>>>>>>>>>>>> argcount=0, kws=0x19a89330, kwcount=0,
> >>>>>>>>>>>>>>>> defs=0x2aaab8b66668, defcount=1, closure=0x0) at
> >>>>>>>>>>>>>>>> Python/ceval.c:2968
> >>>>>>>>>>>>>>>> #29 0x0000000004408f58 in PyEval_EvalFrameEx
> >>>>>>>>>>>>>>>> (f=0x19a891b0, throwflag=<value optimized out>) at
> >>>>>>>>>>>>>>>> Python/ceval.c:3802 #30 0x000000000440aae9 in
> >>>>>>>>>>>>>>>> PyEval_EvalCodeEx (co=0x2aaab8b6a738, globals=<value
> >>>>>>>>>>>>>>>> optimized out>, locals=<value optimized out>, args=0x0,
> >>>>>>>>>>>>>>>> argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
> >>>>>>>>>>>>>>>> closure=0x0) at
> >>>>>>>>>>>>>>>> Python/ceval.c:2968
> >>>>>>>>>>>>>>>> #31 0x000000000440ac02 in PyEval_EvalCode (co=0x1902f9b0,
> >>>>>>>>>>>>>>>> globals=0x0, locals=0x190d9700) at Python/ceval.c:522 #32
> >>>>>>>>>>>>>>>> 0x000000000442853c in PyRun_StringFlags (str=0x192fd3d8
> >>>>>>>>>>>>>>>> "DIRECT.Actran.main()", start=<value optimized out>,
> >>>>>>>>>>>>>>>> globals=0x192213d0, locals=0x192213d0, flags=0x0) at
> >>>>>>>>>>>>>>>> Python/pythonrun.c:1335 #33 0x0000000004429690 in
> >>>>>>>>>>>>>>>> PyRun_SimpleStringFlags (command=0x192fd3d8
> >>>>>>>>>>>>>>>> "DIRECT.Actran.main()", flags=0x0) at
> >>>>>>>>>>>>>>>> Python/pythonrun.c:957 #34 0x0000000001fa1cf9 in
> >>>>>>>>>>>>>>>> FEMTown::Python::FEMPy::run_application
> >>>>>>>
> >>>>>>> (this=0x7ffff279f650)
> >>>>>>>
> >>>>>>>>>>>>>>>> at fempy.cpp:873 #35 0x000000000434ce99 in
> >>>>>>>>>
> >>>>>>>>> FEMTown::Main::Batch::run
> >>>>>>>>>
> >>>>>>>>>>>>>>>> (this=0x7ffff279f650) at batch.cpp:374 #36
> >>>>>>>
> >>>>>>> 0x0000000001f9aa25
> >>>>>>>
> >>>>>>>>>>>>>>>> in main (argc=8, argv=0x7ffff279fa48) at main.cpp:10 (gdb)
> >>>>>>>>>>>>>>>> f 1 #1 0x00002aedbc4e05f4 in btl_openib_handle_incoming
> >>>>>>>>>>>>>>>> (openib_btl=0x1902f9b0, ep=0x1908a1c0, frag=0x190d9700,
> >>>>>>>>>>>>>>>> byte_len=18) at btl_openib_component.c:2881 2881
> >>>>>>>>>>>>>>>> reg->cbfunc( &openib_btl->super, hdr->tag, des,
> >>>>>>>>>>>>>>>> reg->cbdata
> >>>>>>>
> >>>>>>> );
> >>>>>>>
> >>>>>>>>>>>>>>>> Current language: auto; currently c
> >>>>>>>>>>>>>>>> (gdb)
> >>>>>>>>>>>>>>>> #1 0x00002aedbc4e05f4 in btl_openib_handle_incoming
> >>>>>>>>>>>>>>>> (openib_btl=0x1902f9b0, ep=0x1908a1c0, frag=0x190d9700,
> >>>>>>>>>>>>>>>> byte_len=18) at btl_openib_component.c:2881 2881
> >>>>>>>>>>>>>>>> reg->cbfunc( &openib_btl->super, hdr->tag, des,
> >>>>>>>>>>>>>>>> reg->cbdata
> >>>>>>>
> >>>>>>> );
> >>>>>>>
> >>>>>>>>>>>>>>>> (gdb) l 2876
> >>>>>>>>>>>>>>>> 2877 if(OPAL_LIKELY(!(is_credit_msg =
> >>>>>>>>>>>>>>>> is_credit_message(frag)))) { 2878 /* call
> >>>>>>>>>>>>>>>> registered callback */
> >>>>>>>>>>>>>>>> 2879 mca_btl_active_message_callback_t* reg;
> >>>>>>>>>>>>>>>> 2880 reg = mca_btl_base_active_message_trigger
> >>>>>>>>>>>>>>>> + hdr->tag; 2881
> >>>>>>>>>>>>>>>> reg->cbfunc(&openib_btl->super, hdr->tag, des,
> >>>>>>>>>>>>>>>> reg->cbdata ); 2882
> >>>>>>>>>>>>>>>> if(MCA_BTL_OPENIB_RDMA_FRAG(frag)) { 2883
> >>>>>>>>>>>>>>>> cqp
> >>>>>>>
> >>>>>>> =
> >>>>>>>
> >>>>>>>>>>>>>>>> (hdr->credits>> 11)& 0x0f;
> >>>>>>>>>>>>>>>> 2884 hdr->credits&= 0x87ff;
> >>>>>>>>>>>>>>>> 2885 } else {
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>> Eloi
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Friday 16 July 2010 16:01:02 Eloi Gaudry wrote:
> >>>>>>>>>>>>>>>>> Hi Edgar,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The only difference I could observed was that the
> >>>>>>>>>>>>>>>>> segmentation fault appeared sometimes later during the
> >>>>>>>>>>>>>>>>> parallel computation.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'm running out of idea here. I wish I could use the
> >>>>>>>>>>>>>>>>> "--mca
> >>>>>>>>>
> >>>>>>>>> coll
> >>>>>>>>>
> >>>>>>>>>>>>>>>>> tuned" with "--mca self,sm,tcp" so that I could check
> >>>>>>>>>>>>>>>>> that the issue is not somehow limited to the tuned
> >>>>>>>>>>>>>>>>> collective routines.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> Eloi
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thursday 15 July 2010 17:24:24 Edgar Gabriel wrote:
> >>>>>>>>>>>>>>>>>> On 7/15/2010 10:18 AM, Eloi Gaudry wrote:
> >>>>>>>>>>>>>>>>>>> hi edgar,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> thanks for the tips, I'm gonna try this option as well.
> >>>>>>>
> >>>>>>> the
> >>>>>>>
> >>>>>>>>>>>>>>>>>>> segmentation fault i'm observing always happened during
> >>>>>>>>>>>>>>>>>>> a collective communication indeed... does it basically
> >>>>>>>
> >>>>>>> switch
> >>>>>>>
> >>>>>>>>> all
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>> collective communication to basic mode, right ?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> sorry for my ignorance, but what's a NCA ?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> sorry, I meant to type HCA (InifinBand networking card)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>> Edgar
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> thanks,
> >>>>>>>>>>>>>>>>>>> éloi
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thursday 15 July 2010 16:20:54 Edgar Gabriel wrote:
> >>>>>>>>>>>>>>>>>>>> you could try first to use the algorithms in the basic
> >>>>>>>>>
> >>>>>>>>> module,
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> e.g.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> mpirun -np x --mca coll basic ./mytest
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> and see whether this makes a difference. I used to
> >>>>>>>
> >>>>>>> observe
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>> sometimes a (similar ?) problem in the openib btl
> >>>>>>>>>>>>>>>>>>>> triggered from the tuned collective component, in
> >>>>>>>>>>>>>>>>>>>> cases where the ofed libraries were installed but no
> >>>>>>>>>>>>>>>>>>>> NCA was found on a node. It used to work however with
> >>>>>>>>>>>>>>>>>>>> the basic component.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>>> Edgar
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 7/15/2010 3:08 AM, Eloi Gaudry wrote:
> >>>>>>>>>>>>>>>>>>>>> hi Rolf,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> unfortunately, i couldn't get rid of that annoying
> >>>>>>>>>>>>>>>>>>>>> segmentation fault when selecting another bcast
> >>>>>>>>>>>>>>>>>>>>> algorithm. i'm now going to replace MPI_Bcast with a
> >>>>>>>>>>>>>>>>>>>>> naive
> >>>>>>>>>>>>>>>>>>>>> implementation (using MPI_Send and MPI_Recv) and see
> >>>>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>
> >>>>>>>>> that
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> helps.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> regards,
> >>>>>>>>>>>>>>>>>>>>> éloi
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Wednesday 14 July 2010 10:59:53 Eloi Gaudry wrote:
> >>>>>>>>>>>>>>>>>>>>>> Hi Rolf,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> thanks for your input. You're right, I miss the
> >>>>>>>>>>>>>>>>>>>>>> coll_tuned_use_dynamic_rules option.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I'll check if I the segmentation fault disappears
> >>>>>>>>>>>>>>>>>>>>>> when
> >>>>>>>>>
> >>>>>>>>> using
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> the basic bcast linear algorithm using the proper
> >>>>>>>>>>>>>>>>>>>>>> command line you provided.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>> Eloi
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Tuesday 13 July 2010 20:39:59 Rolf vandeVaart
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Hi Eloi:
> >>>>>>>>>>>>>>>>>>>>>>> To select the different bcast algorithms, you need
> >>>>>>>>>>>>>>>>>>>>>>> to add an extra mca parameter that tells the
> >>>>>>>>>>>>>>>>>>>>>>> library to use dynamic selection. --mca
> >>>>>>>>>>>>>>>>>>>>>>> coll_tuned_use_dynamic_rules 1
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> One way to make sure you are typing this in
> >>>>>>>>>>>>>>>>>>>>>>> correctly is
> >>>>>>>>>
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> use it with ompi_info. Do the following:
> >>>>>>>>>>>>>>>>>>>>>>> ompi_info -mca coll_tuned_use_dynamic_rules 1
> >>>>>>>>>>>>>>>>>>>>>>> --param
> >>>>>>>>>
> >>>>>>>>> coll
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> You should see lots of output with all the
> >>>>>>>>>>>>>>>>>>>>>>> different algorithms that can be selected for the
> >>>>>>>>>>>>>>>>>>>>>>> various collectives. Therefore, you need this:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> --mca coll_tuned_use_dynamic_rules 1 --mca
> >>>>>>>>>>>>>>>>>>>>>>> coll_tuned_bcast_algorithm 1
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Rolf
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On 07/13/10 11:28, Eloi Gaudry wrote:
> >>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I've found that "--mca coll_tuned_bcast_algorithm
> >>>>>>>>>>>>>>>>>>>>>>>> 1" allowed to switch to the basic linear
> >>>>>>>>>>>>>>>>>>>>>>>> algorithm. Anyway whatever the algorithm used,
> >>>>>>>>>>>>>>>>>>>>>>>> the segmentation fault remains.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Does anyone could give some advice on ways to
> >>>>>>>
> >>>>>>> diagnose
> >>>>>>>
> >>>>>>>>> the
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> issue I'm facing ?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Eloi
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Monday 12 July 2010 10:53:58 Eloi Gaudry wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I'm focusing on the MPI_Bcast routine that seems
> >>>>>>>>>>>>>>>>>>>>>>>>> to randomly segfault when using the openib btl.
> >>>>>>>>>>>>>>>>>>>>>>>>> I'd
> >>>>>>>
> >>>>>>> like
> >>>>>>>
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> know if there is any way to make OpenMPI switch
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> different algorithm than the default one being
> >>>>>>>>>>>>>>>>>>>>>>>>> selected for MPI_Bcast.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your help,
> >>>>>>>>>>>>>>>>>>>>>>>>> Eloi
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Friday 02 July 2010 11:06:52 Eloi Gaudry wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I'm observing a random segmentation fault during
> >>>>>>>
> >>>>>>> an
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> internode parallel computation involving the
> >>>>>>>
> >>>>>>> openib
> >>>>>>>
> >>>>>>>>> btl
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> and OpenMPI-1.4.2 (the same issue can be
> >>>>>>>>>>>>>>>>>>>>>>>>>> observed with OpenMPI-1.3.3).
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> mpirun (Open MPI) 1.4.2
> >>>>>>>>>>>>>>>>>>>>>>>>>> Report bugs to
> >>>>>>>>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/community/help/
> >>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] *** Process received signal ***
> >>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] Signal: Segmentation fault (11)
> >>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] Signal code: Address not mapped
> >>>>>>>
> >>>>>>> (1)
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] Failing at address: (nil)
> >>>>>>>>>>>>>>>>>>>>>>>>>> [pbn08:02624] [ 0] /lib64/libpthread.so.0
> >>>>>>>>>>>>>>>>>>>>>>>>>> [0x349540e4c0] [pbn08:02624] *** End of error
> >>>>>>>>>
> >>>>>>>>> message
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> ***
> >>>>>>>>>>>>>>>>>>>>>>>>>> sh: line 1: 2624 Segmentation fault
> >>>>>>>>>
> >>>>>>>>> \/share\/hpc3\/actran_suite\/Actran_11\.0\.rc2\.41872\/R
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> ed Ha tE L\ -5 \/ x 86 _6 4\ /bin\/actranpy_mp
> >>>>>>>>>
> >>>>>>>>> '--apl=/share/hpc3/actran_suite/Actran_11.0.rc2.41872/Re
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> dH at EL -5 /x 86 _ 64 /A c tran_11.0.rc2.41872'
> >>>>>>>>>
> >>>>>>>>> '--inputfile=/work/st25652/LSF_130073_0_47696_0/Case1_3D
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> re al _m 4_ n2 .d a t'
> >>>>>>>>>
> >>>>>>>>> '--scratch=/scratch/st25652/LSF_130073_0_47696_0/scratch
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> ' '--mem=3200' '--threads=1'
> >>>>>>>>>>>>>>>>>>>>>>>>>> '--errorlevel=FATAL' '--t_max=0.1'
> >>>>>>>>>>>>>>>>>>>>>>>>>> '--parallel=domain'
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> If I choose not to use the openib btl (by using
> >>>>>>>>>>>>>>>>>>>>>>>>>> --mca btl self,sm,tcp on the command line, for
> >>>>>>>>>>>>>>>>>>>>>>>>>> instance), I don't encounter any problem and the
> >>>>>>>>>>>>>>>>>>>>>>>>>> parallel computation runs flawlessly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I would like to get some help to be able:
> >>>>>>>>>>>>>>>>>>>>>>>>>> - to diagnose the issue I'm facing with the
> >>>>>>>>>>>>>>>>>>>>>>>>>> openib btl - understand why this issue is
> >>>>>>>>>>>>>>>>>>>>>>>>>> observed only when
> >>>>>>>>>
> >>>>>>>>> using
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> the openib btl and not when using self,sm,tcp
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Any help would be very much appreciated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> The outputs of ompi_info and the configure
> >>>>>>>>>>>>>>>>>>>>>>>>>> scripts of OpenMPI are enclosed to this email,
> >>>>>>>>>>>>>>>>>>>>>>>>>> and some
> >>>>>>>>>
> >>>>>>>>> information
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> on the infiniband drivers as well.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Here is the command line used when launching a
> >>>>>>>>>
> >>>>>>>>> parallel
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> computation
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> using infiniband:
> >>>>>>>>>>>>>>>>>>>>>>>>>> path_to_openmpi/bin/mpirun -np $NPROCESS
> >>>>>>>>>>>>>>>>>>>>>>>>>> --hostfile host.list --mca
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> btl openib,sm,self,tcp --display-map --verbose
> >>>>>>>>>>>>>>>>>>>>>>>>>> --version --mca mpi_warn_on_fork 0 --mca
> >>>>>>>>>>>>>>>>>>>>>>>>>> btl_openib_want_fork_support 0 [...]
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> and the command line used if not using infiniband:
> >>>>>>>>>>>>>>>>>>>>>>>>>> path_to_openmpi/bin/mpirun -np $NPROCESS
> >>>>>>>>>>>>>>>>>>>>>>>>>> --hostfile host.list --mca
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> btl self,sm,tcp --display-map --verbose
> >>>>>>>>>>>>>>>>>>>>>>>>>> --version
> >>>>>>>>>
> >>>>>>>>> --mca
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> mpi_warn_on_fork 0 --mca
> >>>>>>>>>>>>>>>>>>>>>>>>>> btl_openib_want_fork_support
> >>>>>>>>>
> >>>>>>>>> 0
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> [...]
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Eloi
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
> >>>>>
> >>>>> _______________________________________________
> >>>>> users mailing list
> >>>>> users_at_[hidden]
> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Eloi Gaudry
Free Field Technologies
Company Website: http://www.fft.be
Company Phone:   +32 10 487 959