Hmmm....well, according to this, it looks like the process ranks are being incorrectly assigned. Shouldn&#39;t have anything to do with what environ we are in (slurm, rsh, etc).<br><br>I&#39;ll look into it - thanks!<br><br>
<div class="gmail_quote">On Mon, May 17, 2010 at 4:25 PM, Christopher Maestas <span dir="ltr">&lt;<a href="mailto:cdmaestas@gmail.com">cdmaestas@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
OK.  The -np only run:<div>---</div><div><div><div>sh-3.1$ mpirun -np 2 --display-allocation --display-devel-map mpi_hello</div><div><br></div><div>======================   ALLOCATED NODES   ======================</div><div>


<br></div><div> Data for node: Name: cut1n7            Launch id: -1   Arch: ffc91200  State: 2</div><div>        Num boards: 1   Num sockets/board: 2    Num cores/socket: 4</div><div>        Daemon: [[51868,0],0]   Daemon launched: True</div>


<div>        Num slots: 1    Slots in use: 0</div><div>        Num slots allocated: 1  Max slots: 0</div><div>        Username on node: NULL</div><div>        Num procs: 0    Next node_rank: 0</div><div> Data for node: Name: cut1n8            Launch id: -1   Arch: 0 State: 2</div>


<div>        Num boards: 1   Num sockets/board: 2    Num cores/socket: 4</div><div>        Daemon: Not defined     Daemon launched: False</div><div>        Num slots: 0    Slots in use: 0</div><div>        Num slots allocated: 0  Max slots: 0</div>


<div>        Username on node: NULL</div><div>        Num procs: 0    Next node_rank: 0</div><div><br></div></div><div>=================================================================</div><div><br></div><div> Map generated by mapping policy: 0400</div>


<div>        Npernode: 0     Oversubscribe allowed: TRUE     CPU Lists: FALSE</div><div>        Num new daemons: 1      New daemon starting vpid 1</div><div>        Num nodes: 2</div><div><br></div><div> Data for node: Name: cut1n7            Launch id: -1   Arch: ffc91200  State: 2</div>


<div>        Num boards: 1   Num sockets/board: 2    Num cores/socket: 4</div><div>        Daemon: [[51868,0],0]   Daemon launched: True</div><div>        Num slots: 1    Slots in use: 1</div><div>        Num slots allocated: 1  Max slots: 0</div>


<div>        Username on node: NULL</div><div>        Num procs: 1    Next node_rank: 1</div><div>        Data for proc: [[51868,1],0]</div><div>                Pid: 0  Local rank: 0   Node rank: 0</div><div>                State: 0        App_context: 0  Slot list: NULL</div>


<div><br></div><div> Data for node: Name: cut1n8            Launch id: -1   Arch: 0 State: 2</div><div>        Num boards: 1   Num sockets/board: 2    Num cores/socket: 4</div><div>        Daemon: [[51868,0],1]   Daemon launched: False</div>


<div>        Num slots: 0    Slots in use: 1</div><div>        Num slots allocated: 0  Max slots: 0</div><div>        Username on node: NULL</div><div>        Num procs: 1    Next node_rank: 1</div><div>        Data for proc: [[51868,1],1]</div>


<div>                Pid: 0  Local rank: 0   Node rank: 0</div><div>                State: 0        App_context: 0  Slot list: NULL</div><div class="im"><div>Hello, I am node cut1n8 with rank 1</div><div>Hello, I am node cut1n7 with rank 0</div>


</div></div><div><br></div><div>---</div><div><br></div><div>Before the segfault I got (using -npernode):<div>---</div><div><div>sh-3.1$ mpirun -npernode 1 --display-allocation --display-devel-map mpi_hello</div><div><br>
</div>

<div>======================   ALLOCATED NODES   ======================</div><div><br></div><div> Data for node: Name: cut1n7            Launch id: -1   Arch: ffc91200  State: 2</div><div>        Num boards: 1   Num sockets/board: 2    Num cores/socket: 4</div>


<div>        Daemon: [[51942,0],0]   Daemon launched: True</div><div>        Num slots: 1    Slots in use: 0</div><div>        Num slots allocated: 1  Max slots: 0</div><div>        Username on node: NULL</div><div>        Num procs: 0    Next node_rank: 0</div>


<div> Data for node: Name: cut1n8            Launch id: -1   Arch: 0 State: 2</div><div>        Num boards: 1   Num sockets/board: 2    Num cores/socket: 4</div><div>        Daemon: Not defined     Daemon launched: False</div>


<div>        Num slots: 0    Slots in use: 0</div><div>        Num slots allocated: 0  Max slots: 0</div><div>        Username on node: NULL</div><div>        Num procs: 0    Next node_rank: 0</div><div><div>=================================================================</div>


<div><br></div><div> Map generated by mapping policy: 0400</div><div>        Npernode: 1     Oversubscribe allowed: TRUE     CPU Lists: FALSE</div><div>        Num new daemons: 1      New daemon starting vpid 1</div><div>


        Num nodes: 2</div><div><br></div><div> Data for node: Name: cut1n7            Launch id: -1   Arch: ffc91200  State: 2</div><div>        Num boards: 1   Num sockets/board: 2    Num cores/socket: 4</div><div>        Daemon: [[51942,0],0]   Daemon launched: True</div>


<div>        Num slots: 1    Slots in use: 1</div><div>        Num slots allocated: 1  Max slots: 0</div><div>        Username on node: NULL</div><div>        Num procs: 1    Next node_rank: 1</div><div>        Data for proc: [[51942,1],0]</div>


<div>                Pid: 0  Local rank: 0   Node rank: 0</div><div>                State: 0        App_context: 0  Slot list: NULL</div><div><br></div><div> Data for node: Name: cut1n8            Launch id: -1   Arch: 0 State: 2</div>


<div>        Num boards: 1   Num sockets/board: 2    Num cores/socket: 4</div><div>        Daemon: [[51942,0],1]   Daemon launched: False</div><div>        Num slots: 0    Slots in use: 1</div><div>        Num slots allocated: 0  Max slots: 0</div>


<div>        Username on node: NULL</div><div>        Num procs: 1    Next node_rank: 1</div><div>        Data for proc: [[51942,1],0]</div><div>                Pid: 0  Local rank: 0   Node rank: 0</div><div>                State: 0        App_context: 0  Slot list: NULL</div>


</div><div><div>[cut1n7:19375] *** Process received signal ***</div><div>[cut1n7:19375] Signal: Segmentation fault (11)</div><div>[cut1n7:19375] Signal code: Address not mapped (1)</div><div>[cut1n7:19375] Failing at address: 0x50</div>


<div>[cut1n7:19375] [ 0] /lib64/libpthread.so.0 [0x37bda0de80]</div><div>[cut1n7:19375] [ 1] /apps/mpi/openmpi/1.4.2-gcc-4.1.2-may.12.10/lib/libopen-rte.so.0(orte_util_encode_pidmap+0xdb) [0x2aed0f93af8b]  </div><div>[cut1n7:19375] [ 2] /apps/mpi/openmpi/1.4.2-gcc-4.1.2-may.12.10/lib/libopen-rte.so.0(orte_odls_base_default_get_add_procs_data+0x655) [0x2aed0f9462f5]</div>


<div>[cut1n7:19375] [ 3] /apps/mpi/openmpi/1.4.2-gcc-4.1.2-may.12.10/lib/libopen-rte.so.0(orte_plm_base_launch_apps+0x10b) [0x2aed0f94d31b]</div><div>[cut1n7:19375] [ 4] /apps/mpi/openmpi/1.4.2-gcc-4.1.2-may.12.10/lib/openmpi/mca_plm_slurm.so [0x2aed107f6ecf]</div>


<div>[cut1n7:19375] [ 5] mpirun [0x40335a]</div><div>[cut1n7:19375] [ 6] mpirun [0x4029f3]</div><div>[cut1n7:19375] [ 7] /lib64/libc.so.6(__libc_start_main+0xf4) [0x37bce1d8b4]</div><div>[cut1n7:19375] [ 8] mpirun [0x402929]</div>


<div>[cut1n7:19375] *** End of error message ***</div><div>Segmentation fault</div></div><div>---<br><br></div><div>I&#39;ll look into a slurm version update.  Previously, SLURM 1.0.30 and Open MPI 1.3.2 working together.  Just curious what was giving me heartache here ... </div>
<div><div></div><div class="h5">

<br><div class="gmail_quote">On Mon, May 17, 2010 at 4:06 PM, Ralph Castain <span dir="ltr">&lt;<a href="mailto:rhc@open-mpi.org" target="_blank">rhc@open-mpi.org</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">


That&#39;s a pretty old version of slurm - I don&#39;t have access to anything that old to test against. You could try running it with --display-allocation --display-devel-map to see what ORTE thinks the allocation is and how it mapped the procs. It sounds like something may be having a problem there...<br>



<br><br><div class="gmail_quote"><div><div></div><div>On Mon, May 17, 2010 at 11:08 AM, Christopher Maestas <span dir="ltr">&lt;<a href="mailto:cdmaestas@gmail.com" target="_blank">cdmaestas@gmail.com</a>&gt;</span> wrote:<br>


</div></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div><div></div><div>
Hello,<div><br></div><div>I&#39;ve been having some troubles with OpenMPI 1.4.X and slurm recently.  I seem to be able to run jobs this way ok:</div><div>---</div><div><div>sh-3.1$ mpirun -np 2 mpi_hello</div><div>Hello, I am node cut1n7 with rank 0</div>





<div>Hello, I am node cut1n8 with rank 1</div></div><div>--</div><div><br></div><div>However if I try and use the -npernode option I get:</div><div>---</div><div><div>sh-3.1$ mpirun -npernode 1 mpi_hello</div><div>[cut1n7:16368] *** Process received signal ***</div>





<div>[cut1n7:16368] Signal: Segmentation fault (11)</div><div>[cut1n7:16368] Signal code: Address not mapped (1)</div><div>[cut1n7:16368] Failing at address: 0x50</div><div>[cut1n7:16368] [ 0] /lib64/libpthread.so.0 [0x37bda0de80]</div>





<div>[cut1n7:16368] [ 1] /apps/mpi/openmpi/1.4.2-gcc-4.1.2-may.12.10/lib/libopen-rte.so.0(orte_util_encode_pidmap+0xdb) [0x2b73eb84df8b]</div><div>[cut1n7:16368] [ 2] /apps/mpi/openmpi/1.4.2-gcc-4.1.2-may.12.10/lib/libopen-rte.so.0(orte_odls_base_default_get_add_procs_data+0x655) [0x2b73eb8592f5]</div>





<div>[cut1n7:16368] [ 3] /apps/mpi/openmpi/1.4.2-gcc-4.1.2-may.12.10/lib/libopen-rte.so.0(orte_plm_base_launch_apps+0x10b) [0x2b73eb86031b]</div><div>[cut1n7:16368] [ 4] /apps/mpi/openmpi/1.4.2-gcc-4.1.2-may.12.10/lib/openmpi/mca_plm_slurm.so [0x2b73ec709ecf]</div>





<div>[cut1n7:16368] [ 5] mpirun [0x40335a]</div><div>[cut1n7:16368] [ 6] mpirun [0x4029f3]</div><div>[cut1n7:16368] [ 7] /lib64/libc.so.6(__libc_start_main+0xf4) [0x37bce1d8b4]</div><div>[cut1n7:16368] [ 8] mpirun [0x402929]</div>





<div>[cut1n7:16368] *** End of error message ***</div><div>Segmentation fault</div></div><div>---</div><div><br></div><div>This is ompi 1.4.2, gcc 4.1.1 and slurm 2.0.9 ... I&#39;m sure it&#39;s a rather silly detail on my end, but figure I should start this thread for any insights and feedback I can help provide to resolve this.</div>





<div><br></div><div>Thanks,</div><div>-cdm</div>
<br></div></div><div>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@open-mpi.org" target="_blank">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br></div></blockquote></div><br>
<br>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@open-mpi.org" target="_blank">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br></blockquote></div><br></div></div></div></div>
<br>_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@open-mpi.org">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br></blockquote></div><br>