Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] sendrecv_replace: long time to allocate/free memory
From: Pascal Deveze (Pascal.Deveze_at_[hidden])
Date: 2010-04-30 09:20:43


On Fri, 23 Apr 2010 at 11:29:53, George Bosilca wrote:

> If you use any kind of high performance network that require memory
> registration for communications, then this high cost for the
> MPI_Alloc_mem will be hidden by the communications. However, the
> MPI_Alloc_mem function seems horribly complicated to me, as we do the
> whole "find-the-right-allocator" step every time instead of caching
> it. While this might be improved, I'm pretty sure the major part of
> the overhead comes from the registration itself.
> The MPI_Alloc_mem function allocate the memory and then it register it
> with the high speed interconnect (Infiniband as an example). If you
> don't have IB, then this should not happens. You can try to force the
> mpool to nothing, or disable the pinning
> (mpi_leave_pinned=0,mpi_leave_pinned_pipeline=0) to see if this affect
> the performances.
>
>
I have an IB cluster with 32 cores nodes. A big part of my
communications is done through sm, so registering systematically buffers
with IB is killing performance for nothing.
Following your tip, I disabled the pinning (using "mpirun -mca
mpi_leave_pinned 0 -mca mpi_leave_pinned_pipeline 0)".
The cycle (MPI_Alloc_mem/MPI_Free_mem) takes now 120 us, while
(malloc/free) takes 1 us.

In all cases, a program calling MPI_Sendrecv_replace() is hardly
penalized by these calls to MPI_Alloc_mem/MPI_Free_mem.
That's why I proposed to come back to the malloc/free scheme in this
routine.

Pascal
> george.
>
> On Apr 22, 2010, at 08:50 , Pascal Deveze wrote:
>
>
>> Hi all,
>>
>> The sendrecv_replace in Open MPI seems to allocate/free memory with
>> MPI_Alloc_mem()/MPI_Free_mem()
>>
>> I measured the time to allocate/free a buffer of 1MB.
>> MPI_Alloc_mem/MPI_Free_mem take 350us while malloc/free only take 8us.
>>
>> malloc/free in ompi/mpi/c/sendrecv_replace.c was replaced by
>> MPI_Alloc_mem/MPI_Free_mem with this commit :
>>
>> user: twoodall
>> date: Thu Sep 22 16:43:17 2005 0000
>> summary: use MPI_Alloc_mem/MPI_Free_mem for internally allocated
>> buffers
>>
>> Is there a real reason to use these functions or can we move back to
>> malloc/free ?
>> Is there a problem on my configuration explaining such slow
>> performance with MPI_Alloc_mem ?
>>
>> Pascal
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>