Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] How to debug segv
From: Shamis, Pavel (shamisp_at_[hidden])
Date: 2012-04-25 14:00:14


Alex,
+1 vote for core. It is good starting point.

* If you can't (from some reason) generate the core file, you may drop while (1) somewhere in the init code and attach the gdb later.
* If you are looking for more user-friendly experience, you may try Allinea DDT (they have 30day trial version).

Regards,
Pasha.

> Another thing to try is to load up the core file in gdb and see if that gives you a valid stack trace of where exactly the segv occurred.
>
>
> On Apr 25, 2012, at 9:30 AM, Alex Margolin wrote:
>
>> On 04/25/2012 02:57 PM, Ralph Castain wrote:
>>> Strange that your code didn't generate any symbols - is that a mosix thing? Have you tried just adding opal_output (so it goes to a special diagnostic output channel) statements in your code to see where the segfault is occurring?
>>>
>>> It looks like you are getting thru orte_init. You could add -mca grpcomm_base_verbose 5 to see if you are getting in/thru the modex - if so, then you are probably failing in add_procs.
>>>
>> I guess the symbols are a mosix thing, but it should still show some sort of segmentation fault trace, no? maybe only the assembly opcode... It seems that the SEGV is detected, rather then caught. This may also be related to mosix - I'll check it with the mosix developer.
>>
>> I added the parameter you suggested and appended the output. Modex seems to be working because I use it to exchange the IP and PID, and as you can see at the bottom these are received OK. I'll try debug printouts specifically in add_procs. Thanks for the advice!
>>
>> alex_at_singularity:~/huji/benchmarks/mpi/npb$ mpirun -mca grpcomm_base_verbose 5 -mca btl self,mosix -mca btl_base_verbose 100 -n 4 ft.S.4
>> [singularity:08915] mca:base:select:(grpcomm) Querying component [bad]
>> [singularity:08915] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
>> [singularity:08915] mca:base:select:(grpcomm) Selected component [bad]
>> [singularity:08915] [[37778,0],0] grpcomm:base:receive start comm
>> [singularity:08915] [[37778,0],0] grpcomm:bad:xcast sent to job [37778,0] tag 1
>> [singularity:08915] [[37778,0],0] grpcomm:xcast:recv:send_relay
>> [singularity:08915] [[37778,0],0] grpcomm:base:xcast updating nidmap
>> [singularity:08915] [[37778,0],0] orte:daemon:send_relay - recipient list is empty!
>> [singularity:08916] mca:base:select:(grpcomm) Querying component [bad]
>> [singularity:08916] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
>> [singularity:08916] mca:base:select:(grpcomm) Selected component [bad]
>> [singularity:08916] [[37778,1],0] grpcomm:base:receive start comm
>> [singularity:08919] mca:base:select:(grpcomm) Querying component [bad]
>> [singularity:08919] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
>> [singularity:08919] mca:base:select:(grpcomm) Selected component [bad]
>> [singularity:08919] [[37778,1],2] grpcomm:base:receive start comm
>> [singularity:08917] mca:base:select:(grpcomm) Querying component [bad]
>> [singularity:08917] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
>> [singularity:08917] mca:base:select:(grpcomm) Selected component [bad]
>> [singularity:08917] [[37778,1],1] grpcomm:base:receive start comm
>> [singularity:08921] mca:base:select:(grpcomm) Querying component [bad]
>> [singularity:08921] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
>> [singularity:08921] mca:base:select:(grpcomm) Selected component [bad]
>> [singularity:08921] [[37778,1],3] grpcomm:base:receive start comm
>> [singularity:08916] [[37778,1],0] grpcomm:set_proc_attr: setting attribute MPI_THREAD_LEVEL data size 1
>> [singularity:08916] [[37778,1],0] grpcomm:set_proc_attr: setting attribute OMPI_ARCH data size 11
>> [singularity:08919] [[37778,1],2] grpcomm:set_proc_attr: setting attribute MPI_THREAD_LEVEL data size 1
>> [singularity:08919] [[37778,1],2] grpcomm:set_proc_attr: setting attribute OMPI_ARCH data size 11
>> [singularity:08917] [[37778,1],1] grpcomm:set_proc_attr: setting attribute MPI_THREAD_LEVEL data size 1
>> [singularity:08917] [[37778,1],1] grpcomm:set_proc_attr: setting attribute OMPI_ARCH data size 11
>> [singularity:08921] [[37778,1],3] grpcomm:set_proc_attr: setting attribute MPI_THREAD_LEVEL data size 1
>> [singularity:08921] [[37778,1],3] grpcomm:set_proc_attr: setting attribute OMPI_ARCH data size 11
>> [singularity:08916] mca: base: components_open: Looking for btl components
>> [singularity:08916] mca: base: components_open: opening btl components
>> [singularity:08916] mca: base: components_open: found loaded component mosix
>> [singularity:08916] mca: base: components_open: component mosix register function successful
>> [singularity:08916] mca: base: components_open: component mosix open function successful
>> [singularity:08916] mca: base: components_open: found loaded component self
>> [singularity:08916] mca: base: components_open: component self has no register function
>> [singularity:08916] mca: base: components_open: component self open function successful
>> [singularity:08919] mca: base: components_open: Looking for btl components
>> [singularity:08917] mca: base: components_open: Looking for btl components
>> [singularity:08919] mca: base: components_open: opening btl components
>> [singularity:08919] mca: base: components_open: found loaded component mosix
>> [singularity:08919] mca: base: components_open: component mosix register function successful
>> [singularity:08919] mca: base: components_open: component mosix open function successful
>> [singularity:08919] mca: base: components_open: found loaded component self
>> [singularity:08919] mca: base: components_open: component self has no register function
>> [singularity:08919] mca: base: components_open: component self open function successful
>> [singularity:08921] mca: base: components_open: Looking for btl components
>> [singularity:08917] mca: base: components_open: opening btl components
>> [singularity:08917] mca: base: components_open: found loaded component mosix
>> [singularity:08917] mca: base: components_open: component mosix register function successful
>> [singularity:08917] mca: base: components_open: component mosix open function successful
>> [singularity:08917] mca: base: components_open: found loaded component self
>> [singularity:08917] mca: base: components_open: component self has no register function
>> [singularity:08917] mca: base: components_open: component self open function successful
>> [singularity:08921] mca: base: components_open: opening btl components
>> [singularity:08921] mca: base: components_open: found loaded component mosix
>> [singularity:08921] mca: base: components_open: component mosix register function successful
>> [singularity:08921] mca: base: components_open: component mosix open function successful
>> [singularity:08921] mca: base: components_open: found loaded component self
>> [singularity:08921] mca: base: components_open: component self has no register function
>> [singularity:08921] mca: base: components_open: component self open function successful
>> [singularity:08916] select: initializing btl component mosix
>> [singularity:08916] [[37778,1],0] grpcomm:set_proc_attr: setting attribute btl.mosix.1.7 data size 20
>> [singularity:08919] select: initializing btl component mosix
>> [singularity:08916] select: init of component mosix returned success
>> [singularity:08916] select: initializing btl component self
>> [singularity:08916] select: init of component self returned success
>> [singularity:08916] [[37778,1],0] grpcomm:base:modex: performing modex
>> [singularity:08916] [[37778,1],0] grpcomm:base:pack_modex: reporting 3 entries
>> [singularity:08916] [[37778,1],0] grpcomm:base:full:modex: executing allgather
>> [singularity:08916] [[37778,1],0] grpcomm:bad entering allgather
>> [singularity:08915] [[37778,0],0] COLLECTIVE RECVD FROM [[37778,1],0]
>> [singularity:08915] [[37778,0],0] WORKING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] ADDING [[37778,1],WILDCARD] TO PARTICIPANTS
>> [singularity:08915] [[37778,0],0] PROGRESSING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] PROGRESSING COLL id 0
>> [singularity:08915] [[37778,0],0] ALL LOCAL PROCS CONTRIBUTE 4
>> [singularity:08916] [[37778,1],0] grpcomm:bad allgather underway
>> [singularity:08916] [[37778,1],0] grpcomm:base:modex: modex posted
>> [singularity:08919] [[37778,1],2] grpcomm:set_proc_attr: setting attribute btl.mosix.1.7 data size 20
>> [singularity:08917] select: initializing btl component mosix
>> [singularity:08917] [[37778,1],1] grpcomm:set_proc_attr: setting attribute btl.mosix.1.7 data size 20
>> [singularity:08921] select: initializing btl component mosix
>> [singularity:08921] [[37778,1],3] grpcomm:set_proc_attr: setting attribute btl.mosix.1.7 data size 20
>> [singularity:08919] select: init of component mosix returned success
>> [singularity:08919] select: initializing btl component self
>> [singularity:08919] select: init of component self returned success
>> [singularity:08919] [[37778,1],2] grpcomm:base:modex: performing modex
>> [singularity:08919] [[37778,1],2] grpcomm:base:pack_modex: reporting 3 entries
>> [singularity:08919] [[37778,1],2] grpcomm:base:full:modex: executing allgather
>> [singularity:08919] [[37778,1],2] grpcomm:bad entering allgather
>> [singularity:08915] [[37778,0],0] COLLECTIVE RECVD FROM [[37778,1],2]
>> [singularity:08915] [[37778,0],0] WORKING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] PROGRESSING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] PROGRESSING COLL id 0
>> [singularity:08915] [[37778,0],0] ALL LOCAL PROCS CONTRIBUTE 4
>> [singularity:08919] [[37778,1],2] grpcomm:bad allgather underway
>> [singularity:08919] [[37778,1],2] grpcomm:base:modex: modex posted
>> [singularity:08917] select: init of component mosix returned success
>> [singularity:08917] select: initializing btl component self
>> [singularity:08917] select: init of component self returned success
>> [singularity:08917] [[37778,1],1] grpcomm:base:modex: performing modex
>> [singularity:08917] [[37778,1],1] grpcomm:base:pack_modex: reporting 3 entries
>> [singularity:08917] [[37778,1],1] grpcomm:base:full:modex: executing allgather
>> [singularity:08917] [[37778,1],1] grpcomm:bad entering allgather
>> [singularity:08915] [[37778,0],0] COLLECTIVE RECVD FROM [[37778,1],1]
>> [singularity:08915] [[37778,0],0] WORKING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] PROGRESSING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] PROGRESSING COLL id 0
>> [singularity:08915] [[37778,0],0] ALL LOCAL PROCS CONTRIBUTE 4
>> [singularity:08917] [[37778,1],1] grpcomm:bad allgather underway
>> [singularity:08917] [[37778,1],1] grpcomm:base:modex: modex posted
>> [singularity:08921] select: init of component mosix returned success
>> [singularity:08921] select: initializing btl component self
>> [singularity:08921] select: init of component self returned success
>> [singularity:08921] [[37778,1],3] grpcomm:base:modex: performing modex
>> [singularity:08921] [[37778,1],3] grpcomm:base:pack_modex: reporting 3 entries
>> [singularity:08921] [[37778,1],3] grpcomm:base:full:modex: executing allgather
>> [singularity:08921] [[37778,1],3] grpcomm:bad entering allgather
>> [singularity:08915] [[37778,0],0] COLLECTIVE RECVD FROM [[37778,1],3]
>> [singularity:08915] [[37778,0],0] WORKING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] PROGRESSING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] PROGRESSING COLL id 0
>> [singularity:08915] [[37778,0],0] ALL LOCAL PROCS CONTRIBUTE 4
>> [singularity:08915] [[37778,0],0] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
>> [singularity:08915] [[37778,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[37778,0],0]
>> [singularity:08915] [[37778,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
>> [singularity:08915] [[37778,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 4
>> [singularity:08915] [[37778,0],0] grpcomm:bad:xcast sent to job [37778,1] tag 30
>> [singularity:08915] [[37778,0],0] grpcomm:xcast:recv:send_relay
>> [singularity:08915] [[37778,0],0] orte:daemon:send_relay - recipient list is empty!
>> [singularity:08921] [[37778,1],3] grpcomm:bad allgather underway
>> [singularity:08921] [[37778,1],3] grpcomm:base:modex: modex posted
>> [singularity:08921] [[37778,1],3] grpcomm:base:receive processing collective return for id 0
>> [singularity:08921] [[37778,1],3] CHECKING COLL id 0
>> [singularity:08921] [[37778,1],3] STORING MODEX DATA
>> [singularity:08921] [[37778,1],3] grpcomm:base:store_modex adding modex entry for proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:base:store_modex adding modex entry for proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:base:receive processing collective return for id 0
>> [singularity:08916] [[37778,1],0] grpcomm:base:receive processing collective return for id 0
>> [singularity:08916] [[37778,1],0] CHECKING COLL id 0
>> [singularity:08917] [[37778,1],1] CHECKING COLL id 0
>> [singularity:08916] [[37778,1],0] STORING MODEX DATA
>> [singularity:08917] [[37778,1],1] STORING MODEX DATA
>> [singularity:08921] [[37778,1],3] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],2]
>> [singularity:08916] [[37778,1],0] grpcomm:base:store_modex adding modex entry for proc [[37778,1],0]
>> [singularity:08917] [[37778,1],1] grpcomm:base:store_modex adding modex entry for proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:base:store_modex adding modex entry for proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],0]
>> [singularity:08917] [[37778,1],1] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:base:store_modex adding modex entry for proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:base:store_modex adding modex entry for proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],2]
>> [singularity:08916] [[37778,1],0] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:base:store_modex adding modex entry for proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:base:store_modex adding modex entry for proc [[37778,1],1]
>> [singularity:08917] [[37778,1],1] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],1]
>> [singularity:08917] [[37778,1],1] grpcomm:base:store_modex adding modex entry for proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:base:store_modex adding modex entry for proc [[37778,1],3]
>> [singularity:08917] [[37778,1],1] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],3]
>> [singularity:08921] [[37778,1],3] grpcomm:base:store_modex adding modex entry for proc [[37778,1],3]
>> [singularity:08921] [[37778,1],3] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],3]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],0]
>> [singularity:08915] [[37778,0],0] COLLECTIVE RECVD FROM [[37778,1],3]
>> [singularity:08915] [[37778,0],0] WORKING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] ADDING [[37778,1],WILDCARD] TO PARTICIPANTS
>> [singularity:08915] [[37778,0],0] PROGRESSING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] PROGRESSING COLL id 1
>> [singularity:08915] [[37778,0],0] ALL LOCAL PROCS CONTRIBUTE 4
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],1]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],1]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],2]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],2]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],1]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],1]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],2]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],2]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],3]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],3]
>> [singularity:08921] [[37778,1],3] grpcomm:bad entering barrier
>> [singularity:08921] [[37778,1],3] grpcomm:bad barrier underway
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],2]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],2]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],0]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],0]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],0]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],3]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],3]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],0]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],0]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],1]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],1]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],3]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],2]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:bad entering barrier
>> [singularity:08917] [[37778,1],1] grpcomm:bad entering barrier
>> [singularity:08915] [[37778,0],0] COLLECTIVE RECVD FROM [[37778,1],0]
>> [singularity:08915] [[37778,0],0] WORKING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] PROGRESSING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] PROGRESSING COLL id 1
>> [singularity:08915] [[37778,0],0] ALL LOCAL PROCS CONTRIBUTE 4
>> [singularity:08915] [[37778,0],0] COLLECTIVE RECVD FROM [[37778,1],1]
>> [singularity:08915] [[37778,0],0] WORKING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] PROGRESSING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] PROGRESSING COLL id 1
>> [singularity:08915] [[37778,0],0] ALL LOCAL PROCS CONTRIBUTE 4
>> [singularity:08917] [[37778,1],1] grpcomm:bad barrier underway
>> [singularity:08916] [[37778,1],0] grpcomm:bad barrier underway
>> [singularity:08919] [[37778,1],2] grpcomm:base:receive processing collective return for id 0
>> [singularity:08919] [[37778,1],2] CHECKING COLL id 0
>> [singularity:08919] [[37778,1],2] STORING MODEX DATA
>> [singularity:08919] [[37778,1],2] grpcomm:base:store_modex adding modex entry for proc [[37778,1],0]
>> [singularity:08919] [[37778,1],2] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],0]
>> [singularity:08919] [[37778,1],2] grpcomm:base:store_modex adding modex entry for proc [[37778,1],2]
>> [singularity:08919] [[37778,1],2] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],2]
>> [singularity:08919] [[37778,1],2] grpcomm:base:store_modex adding modex entry for proc [[37778,1],1]
>> [singularity:08919] [[37778,1],2] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],1]
>> [singularity:08919] [[37778,1],2] grpcomm:base:store_modex adding modex entry for proc [[37778,1],3]
>> [singularity:08919] [[37778,1],2] grpcomm:base:update_modex_entries: adding 3 entries for proc [[37778,1],3]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],0]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],0]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],1]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],1]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr OMPI_ARCH on proc [[37778,1],3]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 11 bytes for attr OMPI_ARCH on proc [[37778,1],3]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],0]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],0]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],1]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],1]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],2]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],2]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr btl.mosix.1.7 on proc [[37778,1],3]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 20 bytes for attr btl.mosix.1.7 on proc [[37778,1],3]
>> [singularity:08919] [[37778,1],2] grpcomm:bad entering barrier
>> [singularity:08915] [[37778,0],0] COLLECTIVE RECVD FROM [[37778,1],2]
>> [singularity:08915] [[37778,0],0] WORKING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] PROGRESSING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] PROGRESSING COLL id 1
>> [singularity:08915] [[37778,0],0] ALL LOCAL PROCS CONTRIBUTE 4
>> [singularity:08915] [[37778,0],0] COLLECTIVE 1 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
>> [singularity:08915] [[37778,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[37778,0],0]
>> [singularity:08915] [[37778,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1
>> [singularity:08915] [[37778,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 4
>> [singularity:08915] [[37778,0],0] grpcomm:bad:xcast sent to job [37778,1] tag 30
>> [singularity:08915] [[37778,0],0] grpcomm:xcast:recv:send_relay
>> [singularity:08915] [[37778,0],0] orte:daemon:send_relay - recipient list is empty!
>> [singularity:08919] [[37778,1],2] grpcomm:bad barrier underway
>> [singularity:08916] [[37778,1],0] grpcomm:base:receive processing collective return for id 1
>> [singularity:08916] [[37778,1],0] CHECKING COLL id 1
>> [singularity:08917] [[37778,1],1] grpcomm:base:receive processing collective return for id 1
>> [singularity:08921] [[37778,1],3] grpcomm:base:receive processing collective return for id 1
>> [singularity:08921] [[37778,1],3] CHECKING COLL id 1
>> [singularity:08917] [[37778,1],1] CHECKING COLL id 1
>> [singularity:08919] [[37778,1],2] grpcomm:base:receive processing collective return for id 1
>> [singularity:08919] [[37778,1],2] CHECKING COLL id 1
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],0]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],0]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],1]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],1]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],2]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],2]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],3]
>> [singularity:08919] [[37778,1],2] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],3]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],0]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],1]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],1]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],2]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],2]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],3]
>> [singularity:08921] [[37778,1],3] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],3]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],0]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],0]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],0]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],1]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],1]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],2]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],3]
>> [singularity:08917] [[37778,1],1] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],0]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],1]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],2]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],2]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: searching for attr MPI_THREAD_LEVEL on proc [[37778,1],3]
>> [singularity:08916] [[37778,1],0] grpcomm:get_proc_attr: found 1 bytes for attr MPI_THREAD_LEVEL on proc [[37778,1],3]
>>
>>
>> NAS Parallel Benchmarks 3.3 -- FT Benchmark
>>
>> No input file inputft.data. Using compiled defaults
>> Size : 64x 64x 64
>> Iterations : 6
>> Number of processes : 4
>> Processor array : 1x 4
>> Layout type : 1D
>> [singularity:08916] btl: mosix: Establishind TCP link to address 127.0.0.1 and PID #8917
>> [singularity:08917] btl: mosix: Establishind TCP link to address 127.0.0.1 and PID #8921
>> [singularity:08916] btl: mosix: Establishind TCP link to address 127.0.0.1 and PID #8919
>> [singularity:08919] btl: mosix: Establishind TCP link to address 127.0.0.1 and PID #8921
>> [singularity:08921] btl: mosix: Establishind TCP link to address 127.0.0.1 and PID #8919
>> [singularity:08917] btl: mosix: Establishind TCP link to address 127.0.0.1 and PID #8916
>> [singularity:08921] btl: mosix: Establishind TCP link to address 127.0.0.1 and PID #8917
>> [singularity:08915] [[37778,0],0] grpcomm:bad:xcast sent to job [37778,0] tag 1
>> [singularity:08915] [[37778,0],0] grpcomm:xcast:recv:send_relay
>> [singularity:08915] [[37778,0],0] orte:daemon:send_relay - recipient list is empty!
>> --------------------------------------------------------------------------
>> mpirun noticed that process rank 2 with PID 8919 on node singularity exited on signal 11 (Segmentation fault).
>> --------------------------------------------------------------------------
>> [singularity:08915] [[37778,0],0] grpcomm:bad:xcast sent to job [37778,0] tag 1
>> [singularity:08915] [[37778,0],0] grpcomm:xcast:recv:send_relay
>> [singularity:08915] [[37778,0],0] orte:daemon:send_relay - recipient list is empty!
>> alex_at_singularity:~/huji/benchmarks/mpi/npb$
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel