Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Possible memory leak
From: ananda.mudar_at_[hidden]
Date: 2010-09-01 09:02:11


Hello Sylvain

Thanks for your explanation.

I tried using -mca coll basic,sync option also and I still see the same
issue ie; the process size increases at an alarming rate. As you see, I
am not allocating any memory inside my program.

Regards

Ananda

------------------- Original Message --------------------------------

Subject: Re: [OMPI devel] Possible memory leak
From: Sylvain Jeaugey (sylvain.jeaugey_at_[hidden])
Date: 2010-09-01 04:07:36

* Next message: Chris Samuel: "[OMPI devel] OMPI 1.5 twitter
notification plugin probably broken by switch to OAUTH"
<http://www.open-mpi.org/community/lists/devel/2010/09/8435.php>
* Previous message: Larry Baker: "[OMPI devel] Fwd: Fwd: 1.5rc5
has been posted"
<http://www.open-mpi.org/community/lists/devel/2010/08/8433.php>
* In reply to: ananda.mudar_at_[hidden]: "[OMPI devel] Possible
memory leak"
<http://www.open-mpi.org/community/lists/devel/2010/08/8431.php>

________________________________

Hi ananda,

I didn't try to run your program, but this seems logical to me.

The problem with calling MPI_Bcast repeatedly is that you may have an
infinite desynchronization between the sender and the receiver(s).
MPI_Bcast is an unidirectional operation. It does not necessary block
until the receiver(s) gets the message, hence causing a huge number of
messages to be buffered (and in the case of ft-enable-cr, I guess
everything is saved until an operation going the other way is done).

To "solve" this issue, the sync collective component has been created to

perform a barrier every N operations. So, running with -mca coll
basic,sync should make the problem disappear. I don't think it is really
a
memory leak, the memory used is needed (in case of fault) and should be
freed at the next operation going the other way (reduce, barrier,
recv/send).

This seems to be the classical problem of MPI_Bcast benchmarks. Real
applications usually don't suffer this kind of problems.

Sylvain

On Tue, 31 Aug 2010, ananda.mudar_at_[hidden] wrote:

> Hi
>
> When I run the attached program with the following arguments, the size
of MPI processes keep increasing alarmingly (I saw that the size grew
from 18M to 12G in under 10 minutes) making me suspect that there is a
major memory leak:
>
> mpirun -am ft-enable-cr --mca coll basic -np 2 <name of the
executable>
>
> If I run this program without checkpoint control ie; remove "-am
ft-enable-cr", the size of MPI processes stays constant.
>
> Also if I run this program without setting "--mca coll basic", the
size of the MPI processes stays constant.
>
> I set the mca parameter to "--mca coll basic" during my debugging
attempts of the problem related to checkpointing the program that has
repetitive MPI_Bcast() calls.
>
> FYI, I am using OpenMPI v1.4.2 with BLCR 0.8.2
>
> Thanks
> Ananda
>
> Ananda B Mudar, PMP
> Senior Technical Architect
> Wipro Technologies
> Ph: 972 765 8093 begin_of_the_skype_highlighting 972 765
8093 end_of_the_skype_highlighting
>
> Please do not print this email unless it is absolutely necessary.
>
> The information contained in this electronic message and any
attachments to this message are intended for the exclusive use of the
addressee(s) and may contain proprietary, confidential or privileged
information. If you are not the intended recipient, you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately and destroy all copies of this message and any attachments.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.
>
> www.wipro.com
>

Ananda B Mudar, PMP

Senior Technical Architect

Wipro Technologies

Ph: 972 765 8093

ananda.mudar_at_[hidden]

 

Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com