Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] mpirun with -am ft-enable-cr option runs slow if hyperthreading is disabled
From: Josh Hursey (jjhursey_at_[hidden])
Date: 2010-03-29 11:21:47


On Mar 22, 2010, at 4:41 PM, <ananda.mudar_at_[hidden]> <ananda.mudar_at_[hidden]
> wrote:

> Hi
>
> If the run my compute intensive openmpi based program using regular
> invocation of mpirun (ie; mpirun –host <hostnamelist> -np <number of
> cores>), it gets completed in few seconds but if I run the same
> program with “-am ft-enable-cr” option, the program takes 10x time
> to complete.
>
> If I enable hyperthreading on my cluster nodes and then call mpirun
> with “-am ft-enable-cr” option, the program gets completed with few
> additional seconds than the normal mpirun!!
>
> How can I improve the performance of mpirun with “-am ft-enable-cr”
> option when I disable hyperthreading on my cluster nodes? Any
> pointers will be really useful.
>
> FYI, I am using openmpi 1.3.4 library and BLCR 0.8.2. Cluster nodes
> are Nehalem based nodes with 8 cores.

I have not done any performance studies focused on hyperthreading, so
I can not say specifically what is happening.

The 10x slowdown is certainly unexpected (I don't see this in my
testing). There usually is a small slowdown (few microseconds) because
of the message tracking technique used to support the checkpoint
coordination protocol. I suspect that the cause of your problem is the
C/R thread which is probably too aggressive for your system. The
improvement with hyperthreading may be that this thread is able to sit
on one of the hardware threads and not completely steal the CPU from
the main application.

You can change how aggressive the thread is by adjusting the two
parameters below:
  http://osl.iu.edu/research/ft/ompi-cr/api.php#mca-opal_cr_thread_sleep_check
  http://osl.iu.edu/research/ft/ompi-cr/api.php#mca-opal_cr_thread_sleep_wait

I usually set the latter to:
  opal_cr_thread_sleep_wait=1000

Give that a try and let me know is that helps. You might also try to
upgrade to the 1.4 series, or even the upcoming v1.5.0 release and see
if the problem persists there.

-- Josh

>
> Thanks
> Anand
> Please do not print this email unless it is absolutely necessary.
>
> The information contained in this electronic message and any
> attachments to this message are intended for the exclusive use of
> the addressee(s) and may contain proprietary, confidential or
> privileged information. If you are not the intended recipient, you
> should not disseminate, distribute or copy this e-mail. Please
> notify the sender immediately and destroy all copies of this message
> and any attachments.
>
> WARNING: Computer viruses can be transmitted via email. The
> recipient should check this email and any attachments for the
> presence of viruses. The company accepts no liability for any damage
> caused by any virus transmitted by this email.
>
> www.wipro.com
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users