Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] OMPI 1.4.3 hangs in gather
From: Nysal Jan (jnysal_at_[hidden])
Date: 2011-01-13 01:29:21


Try manually specifying the collective component "-mca coll tuned"
You seem to be using the "sync" collective component, any stale mca param
files lying around ?

--Nysal

On Tue, Jan 11, 2011 at 6:28 PM, Doron Shoham <doron.ompi_at_[hidden]> wrote:

> Hi
>
> All machines on the setup are IDataPlex with Nehalem 12 cores per node,
> 24GB memory.
>
>
>
> · *Problem 1 – OMPI 1.4.3 hangs in gather:*
>
>
>
> I’m trying to run IMB and gather operation with OMPI 1.4.3 (Vanilla).
>
> It happens when np >= 64 and message size exceed 4k:
>
> mpirun -np 64 -machinefile voltairenodes -mca btl sm,self,openib
> imb/src-1.4.2/IMB-MPI1 gather –npmin 64
>
>
>
> voltairenodes consists of 64 machines.
>
>
>
> #----------------------------------------------------------------
>
> # Benchmarking Gather
>
> # #processes = 64
>
> #----------------------------------------------------------------
>
> #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
>
> 0 1000 0.02 0.02 0.02
>
> 1 331 14.02 14.16 14.09
>
> 2 331 12.87 13.08 12.93
>
> 4 331 14.29 14.43 14.34
>
> 8 331 16.03 16.20 16.11
>
> 16 331 17.54 17.74 17.64
>
> 32 331 20.49 20.62 20.53
>
> 64 331 23.57 23.84 23.70
>
> 128 331 28.02 28.35 28.18
>
> 256 331 34.78 34.88 34.80
>
> 512 331 46.34 46.91 46.60
>
> 1024 331 63.96 64.71 64.33
>
> 2048 331 460.67 465.74 463.18
>
> 4096 331 637.33 643.99 640.75
>
>
>
> This the padb output:
>
> padb –A –x –Ormgr=mpirun –tree:
>
>
>
> =~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2011.01.06 14:33:17
> =~=~=~=~=~=~=~=~=~=~=~=
>
>
>
> Warning, remote process state differs across ranks
>
> state : ranks
>
> R (running) :
> [1,3-6,8,10-13,16-20,23-28,30-32,34-42,44-45,47-49,51-53,56-59,61-63]
>
> S (sleeping) : [0,2,7,9,14-15,21-22,29,33,43,46,50,54-55,60]
>
> Stack trace(s) for thread: 1
>
> -----------------
>
> [0-63] (64 processes)
>
> -----------------
>
> main() at ?:?
>
> IMB_init_buffers_iter() at ?:?
>
> IMB_gather() at ?:?
>
> PMPI_Gather() at pgather.c:175
>
> mca_coll_sync_gather() at coll_sync_gather.c:46
>
> ompi_coll_tuned_gather_intra_dec_fixed() at
> coll_tuned_decision_fixed.c:714
>
> -----------------
>
> [0,3-63] (62 processes)
>
> -----------------
>
> ompi_coll_tuned_gather_intra_linear_sync() at
> coll_tuned_gather.c:248
>
> mca_pml_ob1_recv() at pml_ob1_irecv.c:104
>
> ompi_request_wait_completion() at
> ../../../../ompi/request/request.h:375
>
> opal_condition_wait() at
> ../../../../opal/threads/condition.h:99
>
> -----------------
>
> [1] (1 processes)
>
> -----------------
>
> ompi_coll_tuned_gather_intra_linear_sync() at
> coll_tuned_gather.c:302
>
> mca_pml_ob1_send() at pml_ob1_isend.c:125
>
> ompi_request_wait_completion() at
> ../../../../ompi/request/request.h:375
>
> opal_condition_wait() at
> ../../../../opal/threads/condition.h:99
>
> -----------------
>
> [2] (1 processes)
>
> -----------------
>
> ompi_coll_tuned_gather_intra_linear_sync() at
> coll_tuned_gather.c:315
>
> ompi_request_default_wait() at request/req_wait.c:37
>
> ompi_request_wait_completion() at
> ../ompi/request/request.h:375
>
> opal_condition_wait() at ../opal/threads/condition.h:99
>
> Stack trace(s) for thread: 2
>
> -----------------
>
> [0-63] (64 processes)
>
> -----------------
>
> start_thread() at ?:?
>
> btl_openib_async_thread() at btl_openib_async.c:344
>
> poll() at ?:?
>
> Stack trace(s) for thread: 3
>
> -----------------
>
> [0-63] (64 processes)
>
> -----------------
>
> start_thread() at ?:?
>
> service_thread_start() at btl_openib_fd.c:427
>
> select() at ?:?
>
> -bash-3.2$
>
>
>
>
>
> When running again padb after couple of minutes, I can see that the total
> number of processes remain in the same position but
>
> different processes are at different positions.
>
> For example, this is the diff between two padb outputs:
>
>
>
> Warning, remote process state differs across ranks
>
> state : ranks
>
> -R (running) : [0,2-4,6-13,16-18,20-21,28-31,33-36,38-56,58,60,62-63]
>
> -S (sleeping) : [1,5,14-15,19,22-27,32,37,57,59,61]
>
> +R (running) : [2,5-14,16-23,25,28-40,42-48,50-51,53-58,61,63]
>
> +S (sleeping) : [0-1,3-4,15,24,26-27,41,49,52,59-60,62]
>
> Stack trace(s) for thread: 1
>
> -----------------
>
> [0-63] (64 processes)
>
> @@ -13,21 +13,21 @@
>
> mca_coll_sync_gather() at coll_sync_gather.c:46
>
> ompi_coll_tuned_gather_intra_dec_fixed() at coll_tuned_decision_fixed.c:714
>
> -----------------
>
> - [0,3-63] (62 processes)
>
> + [0-5,8-63] (62 processes)
>
> -----------------
>
> ompi_coll_tuned_gather_intra_linear_sync() at coll_tuned_gather.c:248
>
> mca_pml_ob1_recv() at pml_ob1_irecv.c:104
>
> ompi_request_wait_completion() at ../../../../ompi/request/request.h:375
>
> opal_condition_wait() at ../../../../opal/threads/condition.h:99
>
> -----------------
>
> - [1] (1 processes)
>
> + [6] (1 processes)
>
> -----------------
>
> ompi_coll_tuned_gather_intra_linear_sync() at coll_tuned_gather.c:302
>
> mca_pml_ob1_send() at pml_ob1_isend.c:125
>
> ompi_request_wait_completion() at ../../../../ompi/request/request.h:375
>
> opal_condition_wait() at ../../../../opal/threads/condition.h:99
>
> -----------------
>
> - [2] (1 processes)
>
> + [7] (1 processes)
>
> -----------------
>
> ompi_coll_tuned_gather_intra_linear_sync() at coll_tuned_gather.c:315
>
> ompi_request_default_wait() at request/req_wait.c:37
>
>
>
>
>
> *Choosing different gather algorithm seems to bypass the hang.*
>
> I’ve used the following mca parameters:
>
> --mca coll_tuned_use_dynamic_rules 1
>
> --mca coll_tuned_gather_algorithm 1
>
>
>
> Actually, both dec_fixed and basic_linear works while binomial and
> linear_sync doesn’t.
>
>
>
> With OMPI 1.5 it doesn’t hangs (with all gather algorithms) and it much
> faster (the number of repetitions is much higher):
>
> #----------------------------------------------------------------
>
> # Benchmarking Gather
>
> # #processes = 64
>
> #----------------------------------------------------------------
>
> #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
>
> 0 1000 0.02 0.03 0.02
>
> 1 1000 18.50 18.55 18.53
>
> 2 1000 18.17 18.25 18.22
>
> 4 1000 19.04 19.10 19.07
>
> 8 1000 19.60 19.67 19.64
>
> 16 1000 21.39 21.47 21.43
>
> 32 1000 24.83 24.91 24.87
>
> 64 1000 27.35 27.45 27.40
>
> 128 1000 33.23 33.34 33.29
>
> 256 1000 41.24 41.39 41.32
>
> 512 1000 52.62 52.81 52.71
>
> 1024 1000 73.20 73.46 73.32
>
> 2048 1000 416.36 418.04 417.22
>
> 4096 1000 638.54 640.70 639.65
>
> 8192 1000 506.26 506.97 506.63
>
> 16384 1000 600.63 601.40 601.02
>
> 32768 1000 639.52 640.34 639.93
>
> 65536 640 914.22 916.02 915.13
>
> 131072 320 2287.37 2295.18 2291.35
>
> 262144 160 4041.36 4070.58 4056.27
>
> 524288 80 7292.35 7463.27 7397.14
>
> 1048576 40 13647.15 14107.15 13905.29
>
> 2097152 20 30625.00 32635.45 31815.36
>
> 4194304 10 63543.01 70987.49 68680.48
>
>
>
>
>
> · *Problem 2 – segmentation fault with OMPI 1.4.3/1.5 and IMB
> gather np=768:*
>
> When trying to run the same command but with np=768 I get segmentation
> fault:
>
> openmpi-1.4.3/bin/mpirun -np 768 -machinefile voltairenodes -mca btl
> sm,self,openib -mca coll_tuned_use_dynamic_rules 1 -mca
> coll_tuned_gather_algorithm 1 imb/src/IMB-MPI1 gather -npmin 768 -mem 1.6
>
>
>
> This happens in OMPI 1.4.3 and 1.5
>
>
>
> [compa163:20249] *** Process received signal ***
>
> [compa163:20249] Signal: Segmentation fault (11)
>
> [compa163:20249] Signal code: Address not mapped (1)
>
> [compa163:20249] Failing at address: 0x2aab4a204000
>
> [compa163:20249] [ 0] /lib64/libpthread.so.0 [0x366aa0e7c0]
>
> [compa163:20249] [ 1]
> /gpfs/asrc/home/voltaire/install//openmpi-1.4.3/lib/libmpi.so.0(ompi_convertor_unpack+0x15f)
> [0x2b077882282e]
>
> [compa163:20249] [ 2]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so
> [0x2b077b9e1672]
>
> [compa163:20249] [ 3]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so
> [0x2b077b9dd0b6]
>
> [compa163:20249] [ 4]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_btl_sm.so
> [0x2b077c459d87]
>
> [compa163:20249] [ 5]
> /gpfs/asrc/home/voltaire/install//openmpi-1.4.3/lib/libopen-pal.so.0(opal_progress+0xbe)
> [0x2b0778d845b8]
>
> [compa163:20249] [ 6]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so
> [0x2b077b9d6d62]
>
> [compa163:20249] [ 7]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so
> [0x2b077b9d6ba7]
>
> [compa163:20249] [ 8]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_pml_ob1.so
> [0x2b077b9d6a90]
>
> [compa163:20249] [ 9]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_coll_tuned.so
> [0x2b077d298dc5]
>
> [compa163:20249] [10]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_coll_tuned.so
> [0x2b077d2990d3]
>
> [compa163:20249] [11]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_coll_tuned.so
> [0x2b077d286e9b]
>
> [compa163:20249] [12]
> /gpfs/asrc/home/voltaire/install/openmpi-1.4.3/lib/openmpi/mca_coll_sync.so
> [0x2b077d07e96c]
>
> [compa163:20249] [13]
> /gpfs/asrc/home/voltaire/install//openmpi-1.4.3/lib/libmpi.so.0(PMPI_Gather+0x55e)
> [0x2b077883ec9a]
>
> [compa163:20249] [14] imb/src/IMB-MPI1(IMB_gather+0xe8) [0x40a088]
>
> [compa163:20249] [15] imb/src/IMB-MPI1(IMB_init_buffers_iter+0x28a)
> [0x405baa]
>
> [compa163:20249] [16] imb/src/IMB-MPI1(main+0x30f) [0x40362f]
>
> [compa163:20249] [17] /lib64/libc.so.6(__libc_start_main+0xf4)
> [0x3669e1d994]
>
> [compa163:20249] [18] imb/src/IMB-MPI1 [0x403269]
> [compa163:20249] *** End of error message ***
>
>
> Any ideas? More debuggin tips?
>
> Thanks,
> Doron
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>