Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OpenMPI and Rmpi/snow
From: Ralph Castain (rhc_at_[hidden])
Date: 2012-07-27 00:09:02


Hmmm...well, it looks from your original error message that Rmpi/snow is using a single "master" process and then comm_spawn'ing a whole bunch of "workers". I tried replicating that on a slurm machine by having a single master comm_spawn a whole bunch of processes, and that worked fine. Of course, this was with the current 1.6 branch, which may have something patched in it as we are getting ready for 1.6.1.

What the error is saying is that a race condition is causing procs to try and communicate before the daemon knows how to route their messages. That shouldn't be possible - the daemon should unpack the routing info prior to starting any local procs. I'll review the code to see if I can spot something.

Meantime, could you try the current 1.6 branch tarball and/or the trunk, per my earlier note? It would help to know if I'm looking for a bug in 1.6.0 or something more systemic.

Thanks!
Ralph

On Jul 26, 2012, at 4:32 PM, Brock Palen wrote:

> I think so, sorry if I gave you the impression that Rmpi changed,
>
> Brock Palen
> www.umich.edu/~brockp
> CAEN Advanced Computing
> brockp_at_[hidden]
> (734)936-1985
>
>
>
> On Jul 26, 2012, at 7:30 PM, Ralph Castain wrote:
>
>> Guess I'm confused - your original note indicated that something had changed in Rmpi that broke things. Are you now saying it was something in OMPI?
>>
>> On Jul 26, 2012, at 4:22 PM, Brock Palen wrote:
>>
>>> Ok will see, Rmpi we had working with 1.4 and has not been updated after 2010, this this kinda stinks.
>>>
>>> I will keep digging into it thanks for the help.
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> CAEN Advanced Computing
>>> brockp_at_[hidden]
>>> (734)936-1985
>>>
>>>
>>>
>>> On Jul 26, 2012, at 7:16 PM, Ralph Castain wrote:
>>>
>>>> Crud - afraid you'll have to ask them, then :-(
>>>>
>>>>
>>>> On Jul 26, 2012, at 3:50 PM, Brock Palen wrote:
>>>>
>>>>> Ralph,
>>>>>
>>>>> Rmpi wraps everything up, so I tried setting them with
>>>>>
>>>>> export OMPI_plm_base_verbose=5
>>>>> export OMPI_dpm_base_verbose=5
>>>>>
>>>>> and I get no extra messages even on helloworld example simple MPI-1.0 code.
>>>>>
>>>>>
>>>>> Brock Palen
>>>>> www.umich.edu/~brockp
>>>>> CAEN Advanced Computing
>>>>> brockp_at_[hidden]
>>>>> (734)936-1985
>>>>>
>>>>>
>>>>>
>>>>> On Jul 26, 2012, at 6:42 PM, Ralph Castain wrote:
>>>>>
>>>>>> Well, it looks like comm_spawn is working on 1.6. Afraid I don't know enough about Rmpi/snow to advise on what changed, but you could add some debug params to get an idea of where the problem is occurring:
>>>>>>
>>>>>> -mca plm_base_verbose 5 -mca dpm_base_verbose 5
>>>>>>
>>>>>> should tell you from an OMPI perspective. I can try to help debug that end, at least.
>>>>>>
>>>>>>
>>>>>> On Jul 26, 2012, at 3:02 PM, Ralph Castain wrote:
>>>>>>
>>>>>>> Weird - looks like it has done a comm_spawn and having trouble connecting between the jobs. I can check the basic code and make sure it is working - I seem to recall someone else recently talking about Rmpi changes causing problems (different ones than this, IIRC), so you might want to search our user archives for rmpi to see what they ran into. Not sure what rmpi changed, or why.
>>>>>>>
>>>>>>> On Jul 26, 2012, at 2:41 PM, Brock Palen wrote:
>>>>>>>
>>>>>>>> I have ran into a problem using Rmpi with OpenMPI (trying to get snow running).
>>>>>>>>
>>>>>>>> I built OpenMPI following another post where I built static:
>>>>>>>>
>>>>>>>> ./configure --prefix=$INSTALL/gcc-4.4.6-static --mandir=$INSTALL/gcc-4.4.6-static/man --with-tm=/usr/local/torque/ --with-openib --with-psm --enable-static CC=gcc CXX=g++ FC=gfortran F77=gfortran
>>>>>>>>
>>>>>>>> Rmpi/snow work fine when I run on a single node. When I span more than one node I get nasty errors (pasted below).
>>>>>>>>
>>>>>>>> I tested this mpi install with a simple hello world and that works. Any thoughts what is different about Rmpi/snow that could cause this?
>>>>>>>>
>>>>>>>> [nyx0400.engin.umich.edu:11927] [[48116,0],4] ORTE_ERROR_LOG: Not found in file routed_binomial.c at line 386
>>>>>>>> [nyx0400.engin.umich.edu:11927] [[48116,0],4]:route_callback tried routing message from [[48116,2],16] to [[48116,1],0]:16, can't find route
>>>>>>>> [nyx0405.engin.umich.edu:07707] [[48116,0],8] ORTE_ERROR_LOG: Not found in file routed_binomial.c at line 386
>>>>>>>> [nyx0405.engin.umich.edu:07707] [[48116,0],8]:route_callback tried routing message from [[48116,2],32] to [[48116,1],0]:16, can't find route
>>>>>>>> [0] func:/home/software/rhel6/openmpi-1.6.0/gcc-4.4.6-static/lib/libopen-rte.so.4(opal_backtrace_print+0x1f) [0x2b7e9209e0df]
>>>>>>>> [1] func:/home/software/rhel6/openmpi-1.6.0/gcc-4.4.6-static/lib/libopen-rte.so.4(+0x9f77a) [0x2b7e9206577a]
>>>>>>>> [2] func:/home/software/rhel6/openmpi-1.6.0/gcc-4.4.6-static/lib/libopen-rte.so.4(mca_oob_tcp_msg_recv_complete+0x27f) [0x2b7e920404af]
>>>>>>>> [3] func:/home/software/rhel6/openmpi-1.6.0/gcc-4.4.6-static/lib/libopen-rte.so.4(+0x7bed2) [0x2b7e92041ed2]
>>>>>>>> [4] func:/home/software/rhel6/openmpi-1.6.0/gcc-4.4.6-static/lib/libopen-rte.so.4(opal_event_base_loop+0x238) [0x2b7e92087e38]
>>>>>>>> [5] func:/home/software/rhel6/openmpi-1.6.0/gcc-4.4.6-static/lib/libopen-rte.so.4(orte_daemon+0x8d8) [0x2b7e92016768]
>>>>>>>> [6] func:orted(main+0x66) [0x400966]
>>>>>>>> [7] func:/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d39c1ecdd]
>>>>>>>> [8] func:orted() [0x400839]
>>>>>>>> [nyx0397.engin.umich.edu:06959] [[48116,0],1] ORTE_ERROR_LOG: Not found in file routed_binomial.c at line 386
>>>>>>>> [nyx0397.engin.umich.edu:06959] [[48116,0],1]:route_callback tried routing message from [[48116,2],7] to [[48116,1],0]:16, can't find route
>>>>>>>> [nyx0401.engin.umich.edu:07782] [[48116,0],5] ORTE_ERROR_LOG: Not found in file routed_binomial.c at line 386
>>>>>>>> [nyx0401.engin.umich.edu:07782] [[48116,0],5]:route_callback tried routing message from [[48116,2],23] to [[48116,1],0]:16, can't find route
>>>>>>>> [nyx0406.engin.umich.edu:07743] [[48116,0],9] ORTE_ERROR_LOG: Not found in file routed_binomial.c at line 386
>>>>>>>> [nyx0406.engin.umich.edu:07743] [[48116,0],9]:route_callback tried routing message from [[48116,2],39] to [[48116,1],0]:16, can't find route
>>>>>>>> [0] func:/home/software/rhel6/openmpi-1.6.0/gcc-4.4.6-static/lib/libopen-rte.so.4(opal_backtrace_print+0x1f) [0x2ae2ad17d0df]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Brock Palen
>>>>>>>> www.umich.edu/~brockp
>>>>>>>> CAEN Advanced Computing
>>>>>>>> brockp_at_[hidden]
>>>>>>>> (734)936-1985
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users