RFC: sm Latency
RFC: sm Latency
WHAT: Introducing optimizations to reduce ping-pong
latencies over the sm BTL.
WHY: This is a visible benchmark of MPI performance.
We can improve shared-memory latencies from 30% (if hardware
latency is the limiting factor) to 2× or more (if MPI
software overhead is the limiting factor). At high process
counts, the improvement can be 10× or more.
WHERE: Somewhat in the sm BTL, but very importantly
also in the PML. Changes can be seen in
ssh://www.open-mpi.org/~tdd/hg/fastpath.
WHEN: Upon acceptance. In time for OMPI 1.4.
TIMEOUT: February 6, 2009.
This RFC is being submitted by
eugene.loh@sun.com.
WHY (details)
The sm BTL typically has the lowest hardware latencies
of any BTL. Therefore, any OMPI software overhead we otherwise
tolerate becomes glaringly obvious in sm latency measurements.
In particular, MPI pingpong latencies are oft-cited performance
benchmarks, popular indications of the quality of an MPI implementation.
Competitive vendor MPIs optimize this metric aggressively, both
for np=2 pingpongs and for pairwise pingpongs for high
np (like the popular HPCC performance test suite).
Performance reported by HPCC include:
- MPI_Send()/MPI_Recv() pingpong latency.
- MPI_Send()/MPI_Recv() pingpong latency
as the number of connections grows.
- MPI_Sendrecv() latency.
The slowdown of latency as the number of sm connections
grows becomes increasingly important on large SMPs and ever more
prevalent many-core nodes.
Other MPI implementations, such as Scali and Sun HPC ClusterTools 6,
introduced such optimizations years ago.
Performance measurements indicate that the speedups we can expect
in OMPI with these optimizations range from 30% (np=2
measurements where hardware is the bottleneck) to 2×
(np=2 measurements where software is the bottleneck) to
over 10× (large np).
WHAT (details)
Introduce an optimized "fast path" for "immediate" sends and receives.
Several actions are recommended here.
1. Invoke the sm BTL sendi (send-immediate) function
Each BTL is allowed to define a "send immediate" (sendi)
function. A BTL is not required to do so, however, in which case
the PML calls the standard BTL send function.
A sendi function has already been written for sm,
but it has not been used due to insufficient testing.
The function should be reviewed, commented in, tested, and used.
The changes are:
- File: ompi/mca/btl/sm/btl_sm.c
Declaration/Definition: mca_btl_sm
Comment in the mca_btl_sm_sendi symbol
instead of the NULL placeholder so that the
already existing sendi function will be discovered
and used by the PML.
Function: mca_btl_sm_sendi()
Review the existing sm sendi code.
My suggestions include:
- Drop the test against the eager limit since the PML
calls this function only when the eager limit is
respected.
- Make sure the function has no side effects in the
case where it does not complete. See
Open Issues, the final
section of this document, for further discussion
of "side effects".
Mostly, I have reviewed the code and believe it's
already suitable for use.
2. Move the sendi call up higher in the PML
Profiling pingpong tests, we find that not so much time is spent
in the sm BTL. Rather, the PML consumes a lot of time
preparing a "send request". While these complex data structures
are needed to track progress of a long message that will be sent
in multiple chunks and progressed over multiple entries to and
exits from the MPI library, managing this large data structure
for an "immediate" send (one chunk, one call) is overkill. Latency
can be reduced noticeably if one bypasses this data structure.
This means invoking the sendi function as early as
possible in the PML.
The changes are:
- File: ompi/mca/pml/ob1/pml_ob1_isend.c
Function: mca_pml_ob1_send()
As soon as we enter the PML send function, try to call
the BTL sendi function. If this fails for whatever
reason, continue with the traditional PML send code path.
If it succeeds, then exit the PML and return up to the calling
layer without having to have wrestled with the PML send-request
data structure.
For better software management, the attempt to find and
use a BTL sendi function can be organized into
a new mca_pml_ob1_sendi() function.
- File: ompi/mca/pml/ob1/pml_ob1_sendreq.c
Function: mca_pml_ob1_send_request_start_copy()
Remove this attempt to call the BTL sendi function,
since we've already tried to do so higher up in the PML.
3. Introduce a BTL recvi call
While optimizing the send side of a pingpong operation is helpful,
it is less than half the job. At least as many savings are possible
on the receive side.
Corresponding to what we've done on the send side, on the receive
side we can attempt, as soon as we've entered the PML, to call a
BTL recvi (receive-immediate) function, bypassing the
creation of a complex "receive request" data structure that is
not needed if the receive can be completed immediately.
Further, we can perform directed polling. OMPI pingpong latencies
grow significantly as the number of sm connections increases,
while competitors (Scali, in any case) show absolutely flat latencies
with increasing np. The recvi function could check
one connection for the specified receive and exit quickly if that
message if found.
A BTL is granted considerable latitude in the proposed recvi
functions. The principle requirement is that the recvi
either completes the specified receive completely or else
behaves as if the function was not called at all. (That is, one should
be able to revert to the traditional code path without having to worry
about any recvi side effects. So, for example, if the
recvi function encounters any fragments being returned to
the process, it is permitted to return those fragments to the free list.)
While those are the "hard requirements" for recvi, there
are also some loose guidelines. Mostly, it is understood that
recvi should return "quickly" (a loose term to be interpreted
by the BTL). If recvi can quickly complete the specified
receive, great! If not, it should return control to the PML, who
can then execute the traditional code path, which can handle long
messages (multiple chunks, multiple entries into the MPI library)
and execute other "progress" functions.
The changes are:
- File: ompi/mca/btl/btl.h
In this file, we add a typedef declaration
for what a generic recvi should look like:
typedef int (*mca_btl_base_module_recvi_fn_t)();
We also add a btl_recvi field so that a BTL
can register its recvi function, if any.
- File:
ompi/mca/btl/elan/btl_elan.c
ompi/mca/btl/gm/btl_gm.c
ompi/mca/btl/mx/btl_mx.c
ompi/mca/btl/ofud/btl_ofud.c
ompi/mca/btl/openib/btl_openib.c
ompi/mca/btl/portals/btl_portals.c
ompi/mca/btl/sctp/btl_sctp.c
ompi/mca/btl/self/btl_self.c
ompi/mca/btl/sm/btl_sm.c
ompi/mca/btl/tcp/btl_tcp.c
ompi/mca/btl/template/btl_template.c
ompi/mca/btl/udapl/btl_udapl.c
Each BTL must add a recvi field to its module.
In most cases, BTLs will not define a
recvi function, and the field will be set to
NULL.
- File: ompi/mca/btl/sm/btl_sm.c
Function: mca_btl_sm_recvi()
For the sm BTL, we set the
field to the name of the BTL's recvi
function: mca_btl_sm_recvi.
We also add code to define the behavior of the function.
- File: ompi/mca/btl/sm/btl_sm.h
Prototype: mca_btl_sm_recvi()
We also add a prototype for the new function.
- File: ompi/mca/pml/ob1/pml_ob1_irecv.c
Function: mca_pml_ob1_recv()
As soon as we enter the PML, we try to find and use
a BTL's recvi function. If we succeed, we
can exit the PML without having had invoked the heavy-duty
PML receive-request data structure. If we fail, we
simply revert to the traditional PML receive code path,
without having to worry about any side effects that the
failed recvi might have left.
It is helpful to contain the recvi attempt
in a new mca_pml_ob1_recvi() function, which
we add.
- File: ompi/class/ompi_fifo.h
Function: ompi_fifo_probe_tail()
We don't want recvi to leave any side effects
if it encounters a message it is not prepared to handle.
Therefore, we need to be able to see what is on a FIFO
without popping that entry off the FIFO. Therefore, we
add this new function that probes the FIFO without
disturbing it.
4. Introduce an "immediate" data convertor
One of our aims here is to reduce
latency by bypassing expensive PML send and receive request data
structures. Again, these structures are useful when we intend
to complete a message over multiple chunks and multiple MPI
library invocations, but are overkill for a message that can be
completed all at once.
The same is true of data convertors. Convertors pack user data
into shared-memory buffers or unpack them on the receive side.
Convertors allow a message to be sent in multiple chunks, over
the course of multiple unrelated MPI calls, and for noncontiguous
datatypes. These sophisticated data structures are overkill in
some important cases, such as messages that are handled in a
single chunk and in a single MPI call and consist of a single
contiguous block of data.
While data convertors are not typically too expensive, for
shared-memory latency, where all other costs have been pared back
to a minimum, convertors become noticeable -- around 10%.
Therefore, we recognize special cases where we can have barebones,
minimal, data convertors. In these cases, we initialize the
convertor structure minimally -- e.g., a buffer address, a
number of bytes to copy, and a flag indicating that all other
fields are uninitialized. If this is not possible (e.g., because
a non-contiguous user-derived datatype is being used), the
"immediate" send or receive uses data convertors normally.
The changes are:
- File: ompi/datatype/convertor.h
First, we add to the convertor flags a new flag
#define CONVERTOR_IMMEDIATE 0x10000000
to identify a data convertor that has been initialized
only minimally.
Further, we add three new functions:
- ompi_convertor_immediate():
try to form an "immediate" convertor
- ompi_convertor_immediate_pack():
use an "immediate" convertor to pack
- ompi_convertor_immediate_unpack():
use an "immediate" convertor to unpack
- File: ompi/mca/btl/sm/btl_sm.c
Function: mca_btl_sm_sendi and
mca_btl_sm_recvi
Use the "immediate" convertor routines to pack/unpack.
- File: ompi/mca/pml/ob1/pml_ob1_isend.c and
ompi/mca/pml/ob1/pml_ob1_irecv.c
Have the PML fast path try to construct an "immediate"
convertor.
5. Introduce an "immediate" MPI_Sendrecv()
The optimizations described here should be extended to
MPI_Sendrecv() operations. In particular, while
MPI_Send() and MPI_Recv() optimizations
improve HPCC "pingpong" latencies, we need MPI_Sendrecv()
optimizations to improve HPCC "ring" latencies.
One challenge is the current OMPI MPI/PML interface. Today,
the OMPI MPI layer breaks a Sendrecv call up into
Irecv/Send/Wait. This would seem
to defeat fast-path optimizations at least for the receive.
Some options include:
- allow the MPI layer to call "fast path" operations
- have the PML layer provide a Sendrecv interface
- have the MPI layer emit Isend/Recv/Wait
and see how effectively one can optimize the Isend
operation in the PML for the "immediate" case
Performance Measurements: Before Optimization
More measurements are desirable, but here is a sampling of data
that I happen to have from platforms that I happened to have access
to. This data characterizes OMPI today, without fast-path optimizations.
OMPI versus Other MPIs
Here are pingpong latencies, in µsec, measured with
the OSU latency test for 0 and 8 bytes.
0-byte 8-byte
OMPI 0.74 0.84 µsec
MPICH 0.70 0.77
We see OMPI lagging MPICH.
Scali and HP MPI are presumably considerably faster,
but I have no recent data.
Among other things, one can see that there is about a 10%
penalty for invoking data convertors.
Scaling with Process Count
Here are HPCC pingpong latencies from a different, older,
platform. Though only two processes participate in the pingpong,
the HPCC test reports that latency for different numbers of
processes in the job. We see that OMPI performance slows
dramatically as the number of processes is increased.
Scali (data not available) does not show such a slowdown.
np min avg max
2 2.688 2.719 2.750 usec
4 2.812 2.875 3.000
6 2.875 3.050 3.250
8 2.875 3.299 3.625
10 2.875 3.447 3.812
12 3.063 3.687 4.375
16 2.687 4.093 5.063
20 2.812 4.492 6.000
24 3.125 5.026 6.562
28 3.250 5.326 7.250
32 3.500 5.830 8.375
36 3.750 6.199 8.938
40 4.062 6.753 10.187
The data show large min-max variations in latency. These variations
happen to depend on sender and receiver ranks. Here are latencies
(rounded down to the nearst µsec) for the np=40 case
as a function of sender and receiver rank:
--------- rank of one process ----------->
- 9 9 9 9 9 9 9 9 9 9 9 9 9 8 8 7 7 7 7 7 6 7 8 7 7 7 7 7 6 7 7 7 6 7 7 7 7 6 7
9 - 9 9 9 9 9 9 9 9 8 8 8 8 8 8 7 7 7 7 8 7 7 7 7 7 6 7 7 7 7 7 6 7 6 7 7 7 7 7
9 9 - 9 9 9 9 9 9 9 8 9 7 7 7 8 9 7 7 7 7 7 7 7 7 7 6 7 8 6 7 7 7 7 7 7 6 7 7 6
9 910 - 9 9 9 8 8 8 7 9 7 8 7 7 7 8 8 7 7 8 7 7 6 7 7 7 7 7 6 6 7 6 7 7 7 7 7 7
9 9 9 9 - 9 9 9 8 8 8 7 7 8 7 8 8 8 7 7 7 8 8 7 6 6 7 8 7 7 6 6 7 7 6 7 7 6 7 7
9 9 9 9 9 - 9 9 9 8 7 7 8 8 8 7 8 7 7 8 8 6 7 7 6 7 7 7 7 6 6 6 7 7 7 7 6 6 6 6
9 9 9 9 9 9 - 9 9 8 9 8 8 8 7 8 8 7 8 6 7 7 7 7 7 7 6 6 7 7 6 7 6 7 6 7 7 6 7 6
9 9 9 9 9 9 9 - 9 8 8 8 8 9 8 7 8 7 8 7 7 6 7 7 7 7 7 6 7 7 7 7 7 7 7 7 6 7 7 7
9 9 8 9 9 8 8 9 - 7 9 9 9 7 7 7 8 8 8 7 7 7 6 7 7 7 6 7 6 6 6 6 7 6 7 6 6 6 7 6
9 9 9 9 7 7 8 8 8 - 8 9 8 7 7 7 8 7 7 7 7 7 7 7 7 7 6 6 7 6 7 6 7 7 6 7 7 6 6 6
9 9 9 9 9 8 9 9 7 9 - 8 7 8 7 7 6 8 7 7 7 6 7 7 7 7 7 7 6 6 6 6 7 7 7 6 6 7 7 6
| 9 8 8 9 8 7 8 8 8 8 7 - 9 7 7 8 7 7 7 7 7 7 7 6 6 6 7 6 7 6 6 6 7 7 6 6 7 6 7 5
| 8 8 9 8 9 7 7 8 8 7 7 8 - 7 8 9 8 7 7 7 6 6 7 7 7 7 7 6 7 6 7 7 7 6 7 6 6 6 6 6
| 8 8 8 8 8 9 7 8 8 7 7 7 7 - 8 8 8 8 7 7 7 6 7 7 7 6 6 6 6 7 7 7 7 6 6 6 6 6 5 6
| 6 7 9 9 9 7 7 8 7 7 8 7 8 7 - 6 8 7 7 7 8 7 7 7 7 6 6 7 7 7 6 7 6 7 7 6 6 6 4 5
| 7 7 6 8 7 8 8 8 7 7 8 7 8 9 7 - 7 7 7 8 7 7 6 7 7 7 7 6 7 6 7 6 6 6 6 6 6 5 5 5
7 9 7 8 9 7 8 7 8 8 8 7 7 7 7 7 - 7 8 7 8 7 7 7 7 7 6 7 7 6 6 7 6 6 6 4 5 5 5 5
rank 8 8 8 7 9 7 8 7 7 7 8 7 7 7 7 7 8 - 7 7 7 7 7 7 7 6 7 7 7 6 6 7 7 6 6 6 6 5 4 5
of 7 7 7 8 6 8 6 7 8 7 6 7 7 7 7 7 7 7 - 7 7 7 7 7 7 6 6 7 6 6 6 6 6 6 6 6 6 5 5 4
the 8 7 8 8 7 8 8 7 7 7 7 7 7 7 7 7 7 7 8 - 7 7 7 7 7 7 7 6 7 6 6 6 6 5 5 5 5 5 4 4
other 8 7 7 8 7 7 7 7 8 7 7 7 8 7 7 7 7 7 7 7 - 7 7 6 7 7 7 7 6 6 7 6 6 6 5 5 5 5 5 5
process 7 6 6 7 7 7 8 7 7 6 6 7 6 7 6 7 8 7 7 8 7 - 7 7 7 7 7 7 6 6 6 6 6 5 5 5 4 4 4 4
7 8 7 7 7 7 7 7 8 8 7 7 7 7 7 6 7 6 7 7 7 7 - 7 7 7 7 6 6 6 4 5 5 6 4 4 4 6 5 5
| 7 6 7 7 7 6 6 7 6 8 7 8 7 7 7 7 7 7 7 7 7 7 7 - 7 6 6 6 6 5 5 5 6 5 4 4 5 5 4 4
| 7 7 7 6 7 7 7 7 8 7 6 7 6 6 7 6 6 6 6 7 6 7 7 7 - 6 6 6 5 5 5 5 5 4 4 5 6 4 5 4
| 6 7 7 7 7 7 7 7 8 8 8 7 7 7 6 7 7 7 6 6 7 7 7 6 5 - 6 5 6 6 5 5 5 4 5 5 5 4 4 4
| 7 7 6 7 7 7 7 8 7 7 7 7 6 7 7 7 7 7 6 7 6 6 6 5 5 4 - 5 5 5 4 5 5 5 4 5 5 4 4 4
| 7 7 7 8 7 6 7 6 7 7 7 7 7 6 7 7 7 7 6 6 6 6 6 4 6 4 5 - 5 4 4 5 4 4 5 5 5 4 4 4
V 7 6 8 7 7 6 6 7 6 7 7 7 7 7 6 7 7 6 6 6 7 6 6 5 6 5 5 4 - 4 5 5 4 4 4 4 4 4 4 5
6 6 6 6 6 6 7 8 7 6 7 7 7 7 6 6 7 6 6 5 5 6 6 5 5 6 5 5 4 - 5 4 4 4 4 4 4 6 4 4
6 6 6 7 6 7 7 7 7 6 7 7 6 6 7 7 7 6 6 6 6 6 5 4 4 4 5 4 4 4 - 5 5 4 4 4 4 4 4 4
7 6 7 6 6 6 7 7 7 6 7 7 6 6 6 7 6 6 6 5 6 5 5 5 5 4 4 4 5 5 6 - 4 4 4 4 4 4 4 4
7 7 6 6 6 6 6 7 7 7 6 7 6 7 7 7 6 6 5 5 4 5 5 4 4 4 4 5 4 4 5 4 - 4 4 4 5 4 4 4
7 6 7 6 6 6 6 6 7 7 7 7 6 7 6 6 6 6 6 5 5 4 5 4 4 4 4 4 4 4 4 4 4 - 5 4 4 4 4 5
7 6 7 7 7 8 7 7 6 6 6 7 6 6 6 6 5 5 4 5 5 5 4 4 5 4 4 4 4 4 4 4 4 4 - 4 4 4 4 4
7 6 7 6 7 6 6 6 6 6 7 7 6 6 6 6 5 5 5 4 4 4 4 4 5 4 4 4 4 4 4 4 4 4 4 - 4 4 4 4
8 6 6 7 7 7 7 8 7 6 6 7 6 6 6 6 5 4 5 4 5 5 4 5 4 4 5 4 4 4 4 5 5 4 4 4 - 4 4 4
7 7 7 6 7 7 6 7 6 6 7 6 6 6 6 5 4 5 4 5 4 4 4 4 4 4 4 4 4 5 4 4 4 4 4 4 4 - 4 4
7 7 7 7 7 6 7 7 6 7 7 7 7 5 4 5 5 4 5 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 - 4
7 6 7 7 6 7 6 6 6 6 6 6 6 6 5 5 6 4 4 5 4 5 4 5 4 4 4 4 5 4 4 4 5 4 4 4 4 4 4 -
We see that there is a strong dependence on process rank.
Presumably, this is due to our polling loop. That is, even
if we receive our message, we still have to poll the higher
numbered ranks before we complete the receive operation.
Performance Measurements: After Optimization
We consider three metrics:
- HPCC "pingpong" latency
- OSU latency (0 bytes)
- OSU latency (8 bytes)
We report data for:
- OMPI "out of the box"
- after implementation of steps 1-2 (send side)
- after implementation of steps 1-3 (send and receive sides)
- after implementation of steps 1-4 (send and receive sides, plus data convertor)
The data are from machines that I just happened to have
available.
There is a bit of noise in these results, but the implications,
based on these and other measurements, are:
- There is some improvement from the send side.
- There is more improvement from the receive side.
- The data convertor improvements help a little more (a few percent)
for non-null messages.
- The degree of improvement depends on how fast the CPU is relative
to the memory -- that is, how important software overheads are
versus hardware latency.
- If the CPU is fast (and hardware latency is the bottleneck),
these improvements are less -- say, 20-30%.
- If the CPU is slow (and software costs are the bottleneck),
the improvements are more dramatic --
nearly a factor of 2 for non-null messages.
- As np is increased, latency stays flat. This can
represent a 10× or more improvement over out-of-the-box OMPI.
V20z
Here are results for a V20z (burl-ct-v20z-11):
HPCC OSU0 OSU8
out of box 838 770 850 nsec
Steps 1-2 862 770 860
Steps 1-3 670 610 670
Steps 1-4 642 580 610
F6900
Here are np=2 results from a 1.05-GHz (1.2?) UltraSPARC-IV F6900 server:
HPCC OSU0 OSU8
out of box 3430 2770 3340 nsec
Steps 1-2 2940 2660 3090
Steps 1-3 1854 1650 1880
Steps 1-4 1660 1640 1750
Here is the dependence on process count using HPCC:
OMPI
"out of the box" optimized
comm ----------------- -----------------
size min avg max min avg max
2 2688 2719 2750 1750 1781 1812 nsec
4 2812 2875 3000 1750 1802 1812
6 2875 3050 3250 1687 1777 1812
8 2875 3299 3625 1687 1773 1812
10 2875 3447 3812 1687 1789 1812
12 3063 3687 4375 1687 1796 1813
16 2687 4093 5063 1500 1784 1875
20 2812 4492 6000 1687 1788 1875
24 3125 5026 6562 1562 1776 1875
28 3250 5326 7250 1500 1764 1813
32 3500 5830 8375 1562 1755 1875
36 3750 6199 8938 1562 1755 1875
40 4062 6753 10187 1500 1742 1812
Note:
- At np=2, these optimizations lead to a 2×
improvement in shared-memory latency.
- Non-null messages incur more than a 10% penalty,
which is largely addressed by our data-convertor
optimization.
- At larger np, we maintain our fast performance
while OMPI "out of the box" keeps slowing down more and
more and more.
M9000
Here are results for a 128-core M9000. I think the system has:
- 2 hardware threads per core (but we only use one hardware thread per core)
- 4 cores per socket
- 4 sockets per board
- 4 boards per (half?)
- 2 (halves?) per system
As one separates the sender and receiver, hardware latency increases.
Here is the hierarchy:
latency (nsec) bandwidth (Mbyte/sec)
out-of-box fastpath out-of-box fastpath
(on-socket?) 810 480 2000 2000
(on-board?) 2050 1820 1900 1900
(half?) 3030 2840 1680 1680
3150 2960 1660 1660
Note:
- Latency benefits some hundreds of nsecs with fastpath.
- This latency improvement is striking when the hardware
latency is small, but less noticeable as as the hardware
latency increases.
- Bandwidth is not very sensitive to hardware latency
(due to prefetch) and not at all to fast-path optimizations.
Here are HPCC pingpong latencies for increasing process counts:
out-of-box fastpath
np ----------------- -----------------
min avg max min avg max
2 812 812 812 499 499 499
4 874 921 999 437 494 562
8 937 1847 2624 437 1249 1874
16 1062 2430 2937 437 1557 1937
32 1562 3850 5437 375 2211 2875
64 2687 8329 15874 437 2535 3062
80 3499 16854 41749 374 2647 3437
96 3812 31159 100812 374 2717 3437
128 5187 125774 335187 437 2793 3499
The improvements are tremendous:
- At low np, latencies are low since sender and
receiver can be colocated. Nevertheless, fast-path
optimizations provided a nearly 2× improvement.
- As np increases, fast-path latency also increases,
but this is due to higher hardware latencies. Indeed,
the "min" numbers even drop a little. The "max" fast-path
numbers basically only represent the increase in hardware
latency.
- As np increases, OMPI "out of the box" latency
suffers catastrophically. Not only is there the issue
of more connections to poll, but the polling behaviors
of non-participating processes wreak havoc on the performance
of measured processes.
We can separate the two sources of latency degradation by
putting the np-2 non-participating processes to sleep.
In that case, latency only rises to about 10-20 µsec.
So, polling of many connections causes a substantial rise
in latency, while the disturbance of hard-poll loops on
system performance is responsible for even more degradation.
Actually, even bandwidth benefits:
out-of-box fastpath
np -------------- -------------
min avg max min avg max
2 2015 2034 2053 2028 2039 2051
4 2002 2043 2077 1993 2032 2065
8 1888 1959 2035 1897 1969 2088
16 1863 1934 2046 1856 1937 2066
32 1626 1796 2038 1581 1798 2068
64 1557 1709 1969 1591 1729 2084
80 1439 1619 1902 1561 1706 2059
96 1281 1452 1722 1500 1689 2005
128 677 835 1276 893 1671 1906
Here, we see that even bandwidth suffers "out of the box"
as the number of hard-spinning processes increases. Note
the degradation in "out-of-box" average bandwidths as np
increases. In contrast, the "fastpath" average holds up well.
(The np=128 min fastpath number 893 Mbyte/sec is poor,
but analysis shows it to be a measurement outlier.)
MPI_Sendrecv()
We should also get these optimizations into MPI_Sendrecv() in
order to speed up the HPCC "ring" results. E.g., here are latencies
in µsecs for a performance measurement based on HPCC "ring" tests.
==================================================
np=64
natural random
"out of box" 11.7 10.9
fast path 8.3 6.2
fast path and 100 warmups 3.5 3.6
==================================================
np=128 latency
natural random
"out of box" 242.9 226.1
fast path 56.6 37.0
fast path and 100 warmups 4.2 4.1
==================================================
There happen to be two problems here:
- We need fast-path optimizations in MPI_Sendrecv()
for improved performance.
- The MPI collective operation preceding the "ring" measurement
has "ragged" exit times. So, the "ring" timing starts well
before all of the processes have entered that measurement.
This is a separate OMPI performance problem that must be
handled as well for good HPCC results.
Open Issues
Here are some open issues:
- Side effects. Should the sendi and
recvi functions leave any side effects if they
do not complete the specified operation?
To my taste, they should not.
Currently, however, the sendi function is expected
to allocate a descriptor if it can, even if it cannot
complete the entire send operation.
- recvi: BTL and match header.
An in-coming message starts
with a "match header", with such data as MPI source rank,
MPI communicator, and MPI tag for performing MPI message
matching. Presumably, the BTL knows nothing about this
header. Message matching is performed, for example, via
PML callback functions. We are aggressively trying to
optimize this code path, however, so we should consider
alternatives to that approach.
One alternative is simply for the BTL to perform a byte-by-byte
comparison between the received header and the specified
header. The PML already tells the BTL how many bytes are
in the header.
One problem with this approach is that the fast path would
not be able to support the wildcard tag MPI_ANY_TAG.
Further, it leaves open the question how one extracts information
(such as source or tag) from this header for the MPI_Status
structure.
We can imagine a variety of solutions here, but so far
we've implemented a very simple (even if architecturally
distasteful) solution: we hardwire information (previously
private to the PML) about the match header into the BTL.
That approach can be replaced with other solutions.
- MPI_Sendrecv() support. As discussed
earlier, we should support fast-path optimizations for
"immediate" send-receive operations. Again, this may
entail some movement of current OMPI architectural
boundaries.
Other optimizations that are needed for good HPCC results
include:
- reducing the degradation due to hard spin waits
- improving the performance of collective operations
(which "artificially" degrade HPCC "ring" test results)
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}