Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] [OMPI users] huge VmRSS on rank 0 after MPI_Init when using "btl_openib_receive_queues" option
From: Eloi Gaudry (eg_at_[hidden])
Date: 2011-07-19 15:14:27


is there any other tests you could think of to progress on this issue ?

thanks,
éloi

On Wed, 13 Jul 2011 09:16:11 +0200, Eloi Gaudry <eg_at_[hidden]> wrote:

> Unfortunately, I could not observe such a VmRSS increase.
>
> Here is the command line used to perform this test:
> /opt/openmpi-1.4.2/bin/mpirun -np 256 --hostfile host.list --display-map
> --mca mpi_warn_on_fork 0 --mca btl_openib_want_fork_support 0 --mca btl
> openib,sm,self --mca btl_openib_receive_queues P,32768,128,96,64 --mca
> btl_openib_max_send_size 32768 --mca btl_openib_eager_limit 32768 --mca
> btl_openib_rndv_eager_limit 32768 $HOME/info_mp
>
> and the outputs from rank 0 and rank 255 (the root of the 1-byte
> MPI_Gather):
> [hbn0170: 0/256] Allgather with one byte to process with rank 255
> (VmRSS: 1343463424bytes)
> [hbn0170: 0/256] Done Allgather with one byte to process with rank 255
> (VmRSS: 1343488000bytes)
>
> [hbn0232: 255/256] Allgather with one byte to process with rank 255
> (VmRSS: 261124096bytes)
> [hbn0232: 255/256] Done Allgather with one byte to process with rank 255
> (VmRSS: 288129024bytes)
>
> note that 256*128*(96+64+32768)=1340108800 bytes and it matches the
> VmRSS observed on proc 0.
>
> éloi
>
>
>
> On Sat, 09 Jul 2011 13:13:04 +0200, Jeff Squyres <jsquyres_at_[hidden]>
> wrote:
>
>> Sorry for the delayed reply.
>>
>> I'm guessing that there's some kind of rooted collective operation
>> occurring during MPI_INIT. Try doing a 1 byte MPI_GATHER to another
>> rank in MCW (e.g., to rank 5) and see if the VmSS goes to the same size
>> as MCW rank 0.
>>
>>
>> On Jul 8, 2011, at 5:17 AM, Eloi Gaudry wrote:
>>
>>> what i cannot understand is the reason why this extra memory would be
>>> initialized on proc 0 only.
>>> as far as i know, this doesn't make sense.
>>> éloi
>>>
>>>> On 22/04/2011 08:52, Eloi Gaudry wrote:
>>>>> it varies with the receive_queues specification *and* with the
>>>>> number of mpi processes: memory_consumed = nb_mpi_process *
>>>>> nb_buffers * (buffer_size + low_buffer_count_watermark +
>>>>> credit_window_size )
>>>>>
>>>>> éloi
>>>>>
>>>>>
>>>>> On 04/22/2011 12:26 AM, Jeff Squyres wrote:
>>>>>> Does it vary exactly according to your receive_queues specification?
>>>>>>
>>>>>> On Apr 19, 2011, at 9:03 AM, Eloi Gaudry wrote:
>>>>>>
>>>>>>> hello,
>>>>>>>
>>>>>>> i would like to get your input on this:
>>>>>>> when launching a parallel computation on 128 nodes using openib
>>>>>>> and the "-mca btl_openib_receive_queues P,65536,256,192,128"
>>>>>>> option, i observe a rather large resident memory consumption (2GB:
>>>>>>> 65336*256*128) on the process with rank 0 (and only this process)
>>>>>>> just after a call to MPI_Init.
>>>>>>>
>>>>>>> i'd like to know why the other processes doesn't behave the same:
>>>>>>> - other processes located on the same nodes don't use that amount
>>>>>>> of memory
>>>>>>> - all others processes (i.e. located on any other nodes) neither
>>>>>>>
>>>>>>> i'm using OpenMPI-1.4.2, built with gcc-4.3.4 and
>>>>>>> '--enable-cxx-exceptions --with-pic --with-threads=posix' options.
>>>>>>>
>>>>>>> thanks for your help,
>>>>>>> éloi
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>
>

-- 
Eloi Gaudry
Senior Product Development Engineer
Free Field Technologies
Company Website: http://www.fft.be
Direct Phone Number: +32 10 495 147