Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Busy waiting [was Re: (no subject)]
From: Danesh Daroui (Danesh.D_at_[hidden])
Date: 2008-04-24 13:24:35


I just wanted to add my last comment since this discussion seems to be
very hot! As Jeff
mentioned while a process is waiting to receive a message it doesn't
really matter if it uses
blocking or polling. What I really meant was that blocking can be useful
to use CPU cycles to
handle other calculations which is supposed to be done by this node if
OMPI is smart enough
tp decide such things. Otherwise, because HPC nodes are usually
deidicated nodes so there
will no other tasks which will be run in background and therefore be
influenced by spinning.
Nevertheless, I think that using blocking instead of busy loops should
have higher priority since
it can save CPU idle cycles at least for OMPI's internal tasks...

D.

Jeff Squyres skrev:
> What George said is what I meant by "it's a non-trivial amount of
> work." :-)
>
> In addition to when George adds these patches (allowing components to
> register for blocking progress), there's going to be some work to deal
> with shared memory (we have some ideas here, but it's a bit more than
> just allowing shmem to register to blocking progress) and other random
> issues that will arise.
>
>
> On Apr 24, 2008, at 11:17 AM, George Bosilca wrote:
>
>
>> Well, blocking or not blocking this is the question !!!
>> Unfortunately, it's more complex than this thread seems to indicate.
>> It's not that we didn't want to implement it in Open MPI, it's that
>> at one point we had to make a choice ... and we decided to always go
>> for performance first.
>>
>> However, there were some experimentations to go in blocking more at
>> least when only TCP was used. Unfortunately, this break some other
>> things in Open MPI, because of our progression model. We are
>> component based and these components are allowed to register
>> periodically called callbacks ... And here periodically means as
>> often as possible. There are at least 2 components that use this
>> mechanism for their own progression: romio (mca/io/romio) and one-
>> sided communications (mca/osc/*). Switching in blocking mode will
>> break these 2 components completely. This was the reason why we're
>> not blocking when only TCP is used.
>>
>> Anyway, there is a solution. We have to move from a poll base
>> progress for these components to an event base progress. There were
>> some discussions, and if I remember well ... everybody's waiting for
>> one of my patches :) A patch that allow a component to add a
>> completion callback to MPI requests ... I don't have a clear
>> deadline for this, and unfortunately I'm a little busy right now ...
>> but I'll work on it asap.
>>
>> george.
>>
>> On Apr 24, 2008, at 9:43 AM, Barry Rountree wrote:
>>
>>
>>> On Thu, Apr 24, 2008 at 12:56:03PM +0200, Ingo Josopait wrote:
>>>
>>>> I am using one of the nodes as a desktop computer. Therefore it is
>>>> most
>>>> important for me that the mpi program is not so greedily acquiring
>>>> cpu
>>>> time.
>>>>
>>> This is a kernel scheduling issue, not an OpenMPI issue. Busy
>>> waiting in
>>> one process should not cause noticable loss of responsiveness in
>>> another
>>> processes. Have you experimented with the "nice" command?
>>>
>>>
>>>> But I would imagine that the energy consumption is generally a big
>>>> issue, since energy is a major cost factor in a computer cluster.
>>>>
>>> Yup.
>>>
>>>
>>>> When a
>>>> cpu is idle, it uses considerably less energy. Last time I checked
>>>> my
>>>> computer used 180W when both cpu cores were working and 110W when
>>>> both
>>>> cores were idle.
>>>>
>>> What processor is this?
>>>
>>>
>>>> I just made a small hack to solve the problem. I inserted a simple
>>>> sleep
>>>> call into the function 'opal_condition_wait':
>>>>
>>>> --- orig/openmpi-1.2.6/opal/threads/condition.h
>>>> +++ openmpi-1.2.6/opal/threads/condition.h
>>>> @@ -78,6 +78,7 @@
>>>> #endif
>>>> } else {
>>>> while (c->c_signaled == 0) {
>>>> + usleep(1000);
>>>> opal_progress();
>>>> }
>>>> }
>>>>
>>>>
>>> I expect this would lead to increased execution time for all programs
>>> and increased energy consumption for most programs. Recall that
>>> energy
>>> is power multiplied by time. You're reducing the power on some nodes
>>> and increasing time on all nodes.
>>>
>>>
>>>> The usleep call will let the program sleep for about 4 ms (it won't
>>>> sleep for a shorter time because of some timer granularity). But
>>>> that is
>>>> good enough for me. The cpu usage is (almost) zero when the tasks
>>>> are
>>>> waiting for one another.
>>>>
>>> I think your mistake here is considering CPU load to be a useful
>>> metric.
>>> It isn't. Responsiveness is a useful metric, energy is a useful
>>> metric,
>>> but CPU load isn't a reliable guide to either of these.
>>>
>>>
>>>> For a proper implementation you would want to actively poll
>>>> without a
>>>> sleep call for a few milliseconds, and then use some other method
>>>> that
>>>> sleeps not for a fixed time, but until new messages arrive.
>>>>
>>> Well, it sounds like you can get to this before I can. Post your
>>> patch
>>> here and I'll test it on the NAS suite, UMT2K, Paradis, and a few
>>> synthetic benchmarks I've written. The cluster I use has multimeters
>>> hooked up so I can also let you know how much energy is being saved.
>>>
>>> Barry Rountree
>>> Ph.D. Candidate, Computer Science
>>> University of Georgia
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
>