Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] About MPI_TAG_UB
From: Sébastien Boisvert (sebastien.boisvert.3_at_[hidden])
Date: 2012-09-28 11:16:54


On 28/09/12 10:50 AM, Jeff Squyres wrote:
> On Sep 28, 2012, at 10:38 AM, Sébastien Boisvert wrote:
>
>> 1.5 us is very good. But I get 1.5 ms with shared queues (see above).
>
> Oh, I mis-read (I blame it on jet-lag...).
>
> Yes, that seems waaaay too high.
>
> You didn't do a developer build, did you? We add a bunch of extra debugging in developer builds that adds a bunch of latency. And you're not running over-subscribed, right?
>

I installed v1.6.2, not a developer build.

23 or 24 processes per node, 24 AMD Interlagos cores per node. So not oversubscribed.

>>> OTOH, that's pretty bad. :-)
>>
>> I know, all my Ray processes are doing busy waiting, if MPI was event-driven,
>> I would call my software sleepy Ray when latency is high.
>>
>>> I'm not sure why it would be so bad -- are you hammering the virtual router with small incoming messages?
>>
>> There are 24 AMD Opteron(tm) Processor 6172 cores for 1 Mellanox Technologies MT26428 on each node. That may be the cause too.
>
> That's a QDR HCA, right? (i.e., I assume it's very recent)

Yes.

02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)

>
> Try running some simple point-to-point benchmarks and see if you're getting the same latency (i.e., don't run benchmarks in your app -- get a baseline with some well-known benchmarks first).
>
>>> You might need to do a little profiling to see where the bottlenecks are.
>>
>> Well, with the very valuable information you provided about log_num_mtt and log_mtts_per_seg for the Linux kernel module mlx4_core, I think this may be the root of our problem.
>
> It is definitely a cause, but perhaps not the only cause.
>
>> We get 20-30 us on 4096 processes on Cray XE6, so it is unlikely that the bottleneck is in our software.
>
> Possibly not. But every environment is different, and the same software can perform differently in different environments.

I agree.

>
>> Yes, I agree on this, non-portable code is not portable and all with unexpected behaviors.
>
> Got it.
>
>> Ah I see. By removing the checks in my silly patch, I can now dictate things to do to OMPI. Hehe.
>
> :-)
>