I have an application that appears to function as I expect
when compiled with openmpi-1.4.2 on FreeBSD 9.0. But, it
appears to hang during communication between nodes. What
follows is the long version.
I configure 1.4.2 with
./configure --prefix=/usr/local/openmpi-1.4.2 \
--enable-mpirun-prefix-by-default --disable-shared --enable-static
The Fortran compiler is gfortran 4.5.3. I rebuild my application
and launch the app from node10 with
% /usr/local/openmpi-1.4.2/bin/mpiexec -mca btl tcp,self -machinefile mf1 \
-np 4 sasmp sas.in
where the machine file is
% cat mf1
node10 slots=3
node11 slots=4
Using top(1) on node10 and node11, I see
node10
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
74158 kargl 1 65 0 302M 293M CPU1 1 57:06 99.12% sasmp
74160 kargl 1 65 0 306M 298M CPU0 0 57:06 99.07% sasmp
74159 kargl 1 65 0 306M 298M CPU3 3 57:06 99.02% sasmp
node11
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
13144 kargl 1 48 0 306M 297M CPU3 3 55:55 99.02% sasmp
The above is the expected process information, and more important
the application is producing the right answer.
Now, if I repeat everything above for 1.4.3, I get
./configure --prefix=/usr/local/openmpi-1.4.3 \
--enable-mpirun-prefix-by-default --disable-shared --enable-static
Rebuild my application and launch the app from node10 with
% /usr/local/openmpi-1.4.3/bin/mpiexec -mca btl tcp,self -machinefile mf1 \
-np 4 sasmp sas.in
node10
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
74460 kargl 1 66 0 302M 291M CPU2 2 3:15 99.03% sasmp
74462 kargl 1 66 0 302M 291M CPU3 3 3:15 99.03% sasmp
74461 kargl 1 66 0 14472K 4616K CPU1 1 3:15 99.03% sasmp
node11
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
13298 kargl 1 49 0 14472K 3336K CPU3 3 3:11 99.03% sasmp
I've left the application running for up to 12 minutes, and it never
reaches the ~300 MB SIZE nor 293M RES on node11 and the one process
of node10.
Now, if I reduce -np from 4 to 3, then only 3 processes are started
on node10, and I get the expected results. So, as soon as I try to
send something over tcp, the application stalls. Any idea on how
I might debug this problem?
--
Steve
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}