Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI users] How do I run OpenMPI safely on a Nehalem standalone machine?
From: Doug Reeder (dlr1_at_[hidden])
Date: 2010-05-04 22:32:58


Hello,

I have a mac with two quad core nehalem chips (8 cores). The sysctl
command shows 16 cpus (apparently w/ hyperthreading). I have a finite
element code that runs in parallel using openmpi. Running on the
single machine using openmpi -np 8 runs in about 2/3 time that running
with -np 16 does. The program is very well optimized for parallel
processing so I strongly suspect that hyperthreading is not helping.
The program fairly aggressively uses 100% of each cpu it is on so I
don't think hyperthreading gets much of a chance to split the cpu
activity. I would certainly welcome input/insight from an intel
hardware engineer. I make sure that I don't ask for more processors
than there are physical cores and that seems to work.

Doug Reeder
On May 4, 2010, at 7:06 PM, Gus Correa wrote:

> Hi Ralph
>
> Thank you so much for your help.
>
> You are right, paffinity is turned off (default):
>
> **************
> /opt/sw/openmpi/1.4.2/gnu-4.4.3-4/bin/ompi_info --param opal all |
> grep paffinity
> MCA opal: parameter "opal_paffinity_alone" (current
> value: "0", data source: default value, synonyms:
> mpi_paffinity_alone, mpi_paffinity_alone)
> **************
>
> I will try your suggestion to turn off HT tomorrow,
> and report back here.
> Douglas Guptill kindly sent a recipe to turn HT off via BIOS settings.
>
> Cheers,
> Gus Correa
> ---------------------------------------------------------------------
> Gustavo Correa
> Lamont-Doherty Earth Observatory - Columbia University
> Palisades, NY, 10964-8000 - USA
> ---------------------------------------------------------------------
>
>
> Ralph Castain wrote:
>> On May 4, 2010, at 4:51 PM, Gus Correa wrote:
>>> Hi Ralph
>>>
>>> Ralph Castain wrote:
>>>> One possibility is that the sm btl might not like that you have
>>>> hyperthreading enabled.
>>> I remember that hyperthreading was discussed months ago,
>>> in the previous incarnation of this problem/thread/discussion on
>>> "Nehalem vs. Open MPI".
>>> (It sounds like one of those supreme court cases ... )
>>>
>>> I don't really administer that machine,
>>> or any machine with hyperthreading,
>>> so I am not much familiar to the HT nitty-gritty.
>>> How do I turn off hyperthreading?
>>> Is it a BIOS or a Linux thing?
>>> I may try that.
>> I believe it can be turned off via an admin-level cmd, but I'm not
>> certain about it
>>>> Another thing to check: do you have any paffinity settings turned
>>>> on
>>> (e.g., mpi_paffinity_alone)?
>>>
>>> I didn't turn on or off any paffinity setting explicitly,
>>> either in the command line or in the mca config file.
>>> All that I did on the tests was to turn off "sm",
>>> or just use the default settings.
>>> I wonder if paffinity is on by default, is it?
>>> Should I turn it off?
>> It is off by default - I mention it because sometimes people have
>> it set in the default MCA param file and don't realize it is on.
>> Sounds okay here, though.
>>>> Our paffinity system doesn't handle hyperthreading at this time.
>>> OK, so *if* paffinity is on by default (Is it?),
>>> and hyperthreading is also on, as it is now,
>>> I must turn off one of them, maybe both, right?
>>> I may go combinatorial about this tomorrow.
>>> Can't do it today.
>>> Darn locked office door!
>> I would say don't worry about the paffinity right now - sounds like
>> it is off. You can always check, though, by running "ompi_info --
>> param opal all" and checking for the setting of the
>> opal_paffinity_alone variable
>>>> I'm just suspicious of the HT since you have a quad-core machine,
>>> and the limit where things work seems to be 4...
>>>
>>> It may be.
>>> If you tell me how to turn off HT (I'll google around for it
>>> meanwhile),
>>> I will do it tomorrow, if I get a chance to
>>> hard reboot that pesky machine now locked behind a door.
>> Yeah, I'm beginning to believe it is the HT that is causing the
>> problem...
>>> Thanks again for your help.
>>>
>>> Gus
>>>
>>>> On May 4, 2010, at 3:44 PM, Gus Correa wrote:
>>>>> Hi Jeff
>>>>>
>>>>> Sure, I will certainly try v1.4.2.
>>>>> I am downloading it right now.
>>>>> As of this morning, when I first downloaded,
>>>>> the web site still had 1.4.1.
>>>>> Maybe I should have refreshed the web page on my browser.
>>>>>
>>>>> I will tell you how it goes.
>>>>>
>>>>> Gus
>>>>>
>>>>> Jeff Squyres wrote:
>>>>>> Gus -- Can you try v1.4.2 which was just released today?
>>>>>> On May 4, 2010, at 4:18 PM, Gus Correa wrote:
>>>>>>> Hi Ralph
>>>>>>>
>>>>>>> Thank you very much.
>>>>>>> The "-mca btl ^sm" workaround seems to have solved the problem,
>>>>>>> at least for the little hello_c.c test.
>>>>>>> I just ran it fine up to 128 processes.
>>>>>>>
>>>>>>> I confess I am puzzled by this workaround.
>>>>>>> * Why should we turn off "sm" in a standalone machine,
>>>>>>> where everything is supposed to operate via shared memory?
>>>>>>> * Do I incur in a performance penalty by not using "sm"?
>>>>>>> * What other mechanism is actually used by OpenMPI for process
>>>>>>> communication in this case?
>>>>>>>
>>>>>>> It seems to be using tcp, because when I try -np 256 I get
>>>>>>> this error:
>>>>>>>
>>>>>>> [spinoza:02715] [[11518,0],0] ORTE_ERROR_LOG: The system limit
>>>>>>> on number
>>>>>>> of network connections a process can open was reached in file
>>>>>>> ../../../../../orte/mca/oob/tcp/oob_tcp.c at line 447
>>>>>>> --------------------------------------------------------------------------
>>>>>>> Error: system limit exceeded on number of network connections
>>>>>>> that can
>>>>>>> be open
>>>>>>> This can be resolved by setting the mca parameter
>>>>>>> opal_set_max_sys_limits to 1,
>>>>>>> increasing your limit descriptor setting (using limit or
>>>>>>> ulimit commands),
>>>>>>> or asking the system administrator to increase the system limit.
>>>>>>> --------------------------------------------------------------------------
>>>>>>>
>>>>>>> Anyway, no big deal, because we don't intend to oversubrcribe
>>>>>>> the
>>>>>>> processors on real jobs anyway (and the very error message
>>>>>>> suggests a
>>>>>>> workaround to increase np, if needed).
>>>>>>>
>>>>>>> Many thanks,
>>>>>>> Gus Correa
>>>>>>>
>>>>>>> Ralph Castain wrote:
>>>>>>>> I would certainly try it -mca btl ^sm and see if that solves
>>>>>>>> the problem.
>>>>>>>>
>>>>>>>> On May 4, 2010, at 2:38 PM, Eugene Loh wrote:
>>>>>>>>
>>>>>>>>> Gus Correa wrote:
>>>>>>>>>
>>>>>>>>>> Dear Open MPI experts
>>>>>>>>>>
>>>>>>>>>> I need your help to get Open MPI right on a standalone
>>>>>>>>>> machine with Nehalem processors.
>>>>>>>>>>
>>>>>>>>>> How to tweak the mca parameters to avoid problems
>>>>>>>>>> with Nehalem (and perhaps AMD processors also),
>>>>>>>>>> where MPI programs hang, was discussed here before.
>>>>>>>>>>
>>>>>>>>>> However, I lost track of the details, how to work around
>>>>>>>>>> the problem,
>>>>>>>>>> and if it was fully fixed already perhaps.
>>>>>>>>> Yes, perhaps the problem you're seeing is not what you
>>>>>>>>> remember being discussed.
>>>>>>>>>
>>>>>>>>> Perhaps you're thinking of https://svn.open-mpi.org/trac/ompi/ticket/2043
>>>>>>>>> . It's presumably fixed.
>>>>>>>>>
>>>>>>>>>> I am now facing the problem directly on a single Nehalem box.
>>>>>>>>>>
>>>>>>>>>> I installed OpenMPI 1.4.1 from source,
>>>>>>>>>> and compiled the test hello_c.c with mpicc.
>>>>>>>>>> Then I tried to run it with:
>>>>>>>>>>
>>>>>>>>>> 1) mpirun -np 4 a.out
>>>>>>>>>> It ran OK (but seemed to be slow).
>>>>>>>>>>
>>>>>>>>>> 2) mpirun -np 16 a.out
>>>>>>>>>> It hung, and brought the machine to a halt.
>>>>>>>>>>
>>>>>>>>>> Any words of wisdom are appreciated.
>>>>>>>>>>
>>>>>>>>>> More info:
>>>>>>>>>>
>>>>>>>>>> * OpenMPI 1.4.1 installed from source (tarball from your
>>>>>>>>>> site).
>>>>>>>>>> * Compilers are gcc/g++/gfortran 4.4.3-4.
>>>>>>>>>> * OS is Fedora Core 12.
>>>>>>>>>> * The machine is a Dell box with Intel Xeon 5540 (quad core)
>>>>>>>>>> processors on a two-way motherboard and 48GB of RAM.
>>>>>>>>>> * /proc/cpuinfo indicates that hyperthreading is turned on.
>>>>>>>>>> (I can see 16 "processors".)
>>>>>>>>>>
>>>>>>>>>> **
>>>>>>>>>>
>>>>>>>>>> What should I do?
>>>>>>>>>>
>>>>>>>>>> Use -mca btl ^sm ?
>>>>>>>>>> Use -mca btl -mca btl_sm_num_fifos=some_number ? (Which
>>>>>>>>>> number?)
>>>>>>>>>> Use Both?
>>>>>>>>>> Do something else?
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users