Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] RFC: sm Latency
From: Eugene Loh (Eugene.Loh_at_[hidden])
Date: 2009-01-17 01:48:06


RFC: sm Latency

RFC: sm Latency

WHAT: Introducing optimizations to reduce ping-pong latencies over the sm BTL.

WHY: This is a visible benchmark of MPI performance. We can improve shared-memory latencies from 30% (if hardware latency is the limiting factor) to 2× or more (if MPI software overhead is the limiting factor). At high process counts, the improvement can be 10× or more.

WHERE: Somewhat in the sm BTL, but very importantly also in the PML. Changes can be seen in ssh://www.open-mpi.org/~tdd/hg/fastpath.

WHEN: Upon acceptance. In time for OMPI 1.4.

TIMEOUT: February 6, 2009.


This RFC is being submitted by eugene.loh@sun.com.

WHY (details)

The sm BTL typically has the lowest hardware latencies of any BTL. Therefore, any OMPI software overhead we otherwise tolerate becomes glaringly obvious in sm latency measurements.

In particular, MPI pingpong latencies are oft-cited performance benchmarks, popular indications of the quality of an MPI implementation. Competitive vendor MPIs optimize this metric aggressively, both for np=2 pingpongs and for pairwise pingpongs for high np (like the popular HPCC performance test suite).

Performance reported by HPCC include:

  • MPI_Send()/MPI_Recv() pingpong latency.
  • MPI_Send()/MPI_Recv() pingpong latency as the number of connections grows.
  • MPI_Sendrecv() latency.

The slowdown of latency as the number of sm connections grows becomes increasingly important on large SMPs and ever more prevalent many-core nodes.

Other MPI implementations, such as Scali and Sun HPC ClusterTools 6, introduced such optimizations years ago.

Performance measurements indicate that the speedups we can expect in OMPI with these optimizations range from 30% (np=2 measurements where hardware is the bottleneck) to 2× (np=2 measurements where software is the bottleneck) to over 10× (large np).

WHAT (details)

Introduce an optimized "fast path" for "immediate" sends and receives. Several actions are recommended here.

1. Invoke the sm BTL sendi (send-immediate) function

Each BTL is allowed to define a "send immediate" (sendi) function. A BTL is not required to do so, however, in which case the PML calls the standard BTL send function.

A sendi function has already been written for sm, but it has not been used due to insufficient testing.

The function should be reviewed, commented in, tested, and used.

The changes are:

  • File: ompi/mca/btl/sm/btl_sm.c
    Declaration/Definition: mca_btl_sm

    Comment in the mca_btl_sm_sendi symbol instead of the NULL placeholder so that the already existing sendi function will be discovered and used by the PML.

    Function: mca_btl_sm_sendi()

    Review the existing sm sendi code. My suggestions include:

    • Drop the test against the eager limit since the PML calls this function only when the eager limit is respected.
    • Make sure the function has no side effects in the case where it does not complete. See Open Issues, the final section of this document, for further discussion of "side effects".

    Mostly, I have reviewed the code and believe it's already suitable for use.

2. Move the sendi call up higher in the PML

Profiling pingpong tests, we find that not so much time is spent in the sm BTL. Rather, the PML consumes a lot of time preparing a "send request". While these complex data structures are needed to track progress of a long message that will be sent in multiple chunks and progressed over multiple entries to and exits from the MPI library, managing this large data structure for an "immediate" send (one chunk, one call) is overkill. Latency can be reduced noticeably if one bypasses this data structure. This means invoking the sendi function as early as possible in the PML.

The changes are:

  • File: ompi/mca/pml/ob1/pml_ob1_isend.c
    Function: mca_pml_ob1_send()

    As soon as we enter the PML send function, try to call the BTL sendi function. If this fails for whatever reason, continue with the traditional PML send code path. If it succeeds, then exit the PML and return up to the calling layer without having to have wrestled with the PML send-request data structure.

    For better software management, the attempt to find and use a BTL sendi function can be organized into a new mca_pml_ob1_sendi() function.

  • File: ompi/mca/pml/ob1/pml_ob1_sendreq.c
    Function: mca_pml_ob1_send_request_start_copy()

    Remove this attempt to call the BTL sendi function, since we've already tried to do so higher up in the PML.

3. Introduce a BTL recvi call

While optimizing the send side of a pingpong operation is helpful, it is less than half the job. At least as many savings are possible on the receive side.

Corresponding to what we've done on the send side, on the receive side we can attempt, as soon as we've entered the PML, to call a BTL recvi (receive-immediate) function, bypassing the creation of a complex "receive request" data structure that is not needed if the receive can be completed immediately.

Further, we can perform directed polling. OMPI pingpong latencies grow significantly as the number of sm connections increases, while competitors (Scali, in any case) show absolutely flat latencies with increasing np. The recvi function could check one connection for the specified receive and exit quickly if that message if found.

A BTL is granted considerable latitude in the proposed recvi functions. The principle requirement is that the recvi either completes the specified receive completely or else behaves as if the function was not called at all. (That is, one should be able to revert to the traditional code path without having to worry about any recvi side effects. So, for example, if the recvi function encounters any fragments being returned to the process, it is permitted to return those fragments to the free list.)

While those are the "hard requirements" for recvi, there are also some loose guidelines. Mostly, it is understood that recvi should return "quickly" (a loose term to be interpreted by the BTL). If recvi can quickly complete the specified receive, great! If not, it should return control to the PML, who can then execute the traditional code path, which can handle long messages (multiple chunks, multiple entries into the MPI library) and execute other "progress" functions.

The changes are:

  • File: ompi/mca/btl/btl.h

    In this file, we add a typedef declaration for what a generic recvi should look like:

         typedef int (*mca_btl_base_module_recvi_fn_t)();
         

    We also add a btl_recvi field so that a BTL can register its recvi function, if any.

  • File:
    ompi/mca/btl/elan/btl_elan.c
    ompi/mca/btl/gm/btl_gm.c
    ompi/mca/btl/mx/btl_mx.c
    ompi/mca/btl/ofud/btl_ofud.c
    ompi/mca/btl/openib/btl_openib.c
    ompi/mca/btl/portals/btl_portals.c
    ompi/mca/btl/sctp/btl_sctp.c
    ompi/mca/btl/self/btl_self.c
    ompi/mca/btl/sm/btl_sm.c
    ompi/mca/btl/tcp/btl_tcp.c
    ompi/mca/btl/template/btl_template.c
    ompi/mca/btl/udapl/btl_udapl.c

    Each BTL must add a recvi field to its module. In most cases, BTLs will not define a recvi function, and the field will be set to NULL.

  • File: ompi/mca/btl/sm/btl_sm.c
    Function: mca_btl_sm_recvi()

    For the sm BTL, we set the field to the name of the BTL's recvi function: mca_btl_sm_recvi. We also add code to define the behavior of the function.

  • File: ompi/mca/btl/sm/btl_sm.h
    Prototype: mca_btl_sm_recvi()

    We also add a prototype for the new function.

  • File: ompi/mca/pml/ob1/pml_ob1_irecv.c
    Function: mca_pml_ob1_recv()

    As soon as we enter the PML, we try to find and use a BTL's recvi function. If we succeed, we can exit the PML without having had invoked the heavy-duty PML receive-request data structure. If we fail, we simply revert to the traditional PML receive code path, without having to worry about any side effects that the failed recvi might have left.

    It is helpful to contain the recvi attempt in a new mca_pml_ob1_recvi() function, which we add.

  • File: ompi/class/ompi_fifo.h
    Function: ompi_fifo_probe_tail()

    We don't want recvi to leave any side effects if it encounters a message it is not prepared to handle. Therefore, we need to be able to see what is on a FIFO without popping that entry off the FIFO. Therefore, we add this new function that probes the FIFO without disturbing it.

4. Introduce an "immediate" data convertor

One of our aims here is to reduce latency by bypassing expensive PML send and receive request data structures. Again, these structures are useful when we intend to complete a message over multiple chunks and multiple MPI library invocations, but are overkill for a message that can be completed all at once.

The same is true of data convertors. Convertors pack user data into shared-memory buffers or unpack them on the receive side. Convertors allow a message to be sent in multiple chunks, over the course of multiple unrelated MPI calls, and for noncontiguous datatypes. These sophisticated data structures are overkill in some important cases, such as messages that are handled in a single chunk and in a single MPI call and consist of a single contiguous block of data.

While data convertors are not typically too expensive, for shared-memory latency, where all other costs have been pared back to a minimum, convertors become noticeable -- around 10%.

Therefore, we recognize special cases where we can have barebones, minimal, data convertors. In these cases, we initialize the convertor structure minimally -- e.g., a buffer address, a number of bytes to copy, and a flag indicating that all other fields are uninitialized. If this is not possible (e.g., because a non-contiguous user-derived datatype is being used), the "immediate" send or receive uses data convertors normally.

The changes are:

  • File: ompi/datatype/convertor.h

    First, we add to the convertor flags a new flag

         #define CONVERTOR_IMMEDIATE        0x10000000
         
    to identify a data convertor that has been initialized only minimally.

    Further, we add three new functions:

    • ompi_convertor_immediate(): try to form an "immediate" convertor
    • ompi_convertor_immediate_pack(): use an "immediate" convertor to pack
    • ompi_convertor_immediate_unpack(): use an "immediate" convertor to unpack
  • File: ompi/mca/btl/sm/btl_sm.c
    Function: mca_btl_sm_sendi and mca_btl_sm_recvi

    Use the "immediate" convertor routines to pack/unpack.

  • File: ompi/mca/pml/ob1/pml_ob1_isend.c and ompi/mca/pml/ob1/pml_ob1_irecv.c

    Have the PML fast path try to construct an "immediate" convertor.

5. Introduce an "immediate" MPI_Sendrecv()

The optimizations described here should be extended to MPI_Sendrecv() operations. In particular, while MPI_Send() and MPI_Recv() optimizations improve HPCC "pingpong" latencies, we need MPI_Sendrecv() optimizations to improve HPCC "ring" latencies.

One challenge is the current OMPI MPI/PML interface. Today, the OMPI MPI layer breaks a Sendrecv call up into Irecv/Send/Wait. This would seem to defeat fast-path optimizations at least for the receive. Some options include:

  • allow the MPI layer to call "fast path" operations
  • have the PML layer provide a Sendrecv interface
  • have the MPI layer emit Isend/Recv/Wait and see how effectively one can optimize the Isend operation in the PML for the "immediate" case

Performance Measurements: Before Optimization

More measurements are desirable, but here is a sampling of data that I happen to have from platforms that I happened to have access to. This data characterizes OMPI today, without fast-path optimizations.

OMPI versus Other MPIs

Here are pingpong latencies, in µsec, measured with the OSU latency test for 0 and 8 bytes.

                       0-byte  8-byte

     OMPI              0.74    0.84  µsec
     MPICH             0.70    0.77

We see OMPI lagging MPICH.

Scali and HP MPI are presumably considerably faster, but I have no recent data.

Among other things, one can see that there is about a 10% penalty for invoking data convertors.

Scaling with Process Count

Here are HPCC pingpong latencies from a different, older, platform. Though only two processes participate in the pingpong, the HPCC test reports that latency for different numbers of processes in the job. We see that OMPI performance slows dramatically as the number of processes is increased. Scali (data not available) does not show such a slowdown.

     np    min    avg    max

      2   2.688  2.719  2.750 usec
      4   2.812  2.875  3.000
      6   2.875  3.050  3.250
      8   2.875  3.299  3.625
     10   2.875  3.447  3.812
     12   3.063  3.687  4.375
     16   2.687  4.093  5.063
     20   2.812  4.492  6.000
     24   3.125  5.026  6.562
     28   3.250  5.326  7.250
     32   3.500  5.830  8.375
     36   3.750  6.199  8.938
     40   4.062  6.753 10.187

The data show large min-max variations in latency. These variations happen to depend on sender and receiver ranks. Here are latencies (rounded down to the nearst µsec) for the np=40 case as a function of sender and receiver rank:

                                    ---------   rank of one process   ----------->

                     - 9 9 9 9 9 9 9 9 9 9 9 9 9 8 8 7 7 7 7 7 6 7 8 7 7 7 7 7 6 7 7 7 6 7 7 7 7 6 7
                     9 - 9 9 9 9 9 9 9 9 8 8 8 8 8 8 7 7 7 7 8 7 7 7 7 7 6 7 7 7 7 7 6 7 6 7 7 7 7 7
                     9 9 - 9 9 9 9 9 9 9 8 9 7 7 7 8 9 7 7 7 7 7 7 7 7 7 6 7 8 6 7 7 7 7 7 7 6 7 7 6
                     9 910 - 9 9 9 8 8 8 7 9 7 8 7 7 7 8 8 7 7 8 7 7 6 7 7 7 7 7 6 6 7 6 7 7 7 7 7 7
                     9 9 9 9 - 9 9 9 8 8 8 7 7 8 7 8 8 8 7 7 7 8 8 7 6 6 7 8 7 7 6 6 7 7 6 7 7 6 7 7
                     9 9 9 9 9 - 9 9 9 8 7 7 8 8 8 7 8 7 7 8 8 6 7 7 6 7 7 7 7 6 6 6 7 7 7 7 6 6 6 6
                     9 9 9 9 9 9 - 9 9 8 9 8 8 8 7 8 8 7 8 6 7 7 7 7 7 7 6 6 7 7 6 7 6 7 6 7 7 6 7 6
                     9 9 9 9 9 9 9 - 9 8 8 8 8 9 8 7 8 7 8 7 7 6 7 7 7 7 7 6 7 7 7 7 7 7 7 7 6 7 7 7
                     9 9 8 9 9 8 8 9 - 7 9 9 9 7 7 7 8 8 8 7 7 7 6 7 7 7 6 7 6 6 6 6 7 6 7 6 6 6 7 6
                     9 9 9 9 7 7 8 8 8 - 8 9 8 7 7 7 8 7 7 7 7 7 7 7 7 7 6 6 7 6 7 6 7 7 6 7 7 6 6 6
                     9 9 9 9 9 8 9 9 7 9 - 8 7 8 7 7 6 8 7 7 7 6 7 7 7 7 7 7 6 6 6 6 7 7 7 6 6 7 7 6
           |         9 8 8 9 8 7 8 8 8 8 7 - 9 7 7 8 7 7 7 7 7 7 7 6 6 6 7 6 7 6 6 6 7 7 6 6 7 6 7 5
           |         8 8 9 8 9 7 7 8 8 7 7 8 - 7 8 9 8 7 7 7 6 6 7 7 7 7 7 6 7 6 7 7 7 6 7 6 6 6 6 6
           |         8 8 8 8 8 9 7 8 8 7 7 7 7 - 8 8 8 8 7 7 7 6 7 7 7 6 6 6 6 7 7 7 7 6 6 6 6 6 5 6
           |         6 7 9 9 9 7 7 8 7 7 8 7 8 7 - 6 8 7 7 7 8 7 7 7 7 6 6 7 7 7 6 7 6 7 7 6 6 6 4 5
           |         7 7 6 8 7 8 8 8 7 7 8 7 8 9 7 - 7 7 7 8 7 7 6 7 7 7 7 6 7 6 7 6 6 6 6 6 6 5 5 5
                     7 9 7 8 9 7 8 7 8 8 8 7 7 7 7 7 - 7 8 7 8 7 7 7 7 7 6 7 7 6 6 7 6 6 6 4 5 5 5 5
         rank        8 8 8 7 9 7 8 7 7 7 8 7 7 7 7 7 8 - 7 7 7 7 7 7 7 6 7 7 7 6 6 7 7 6 6 6 6 5 4 5
          of         7 7 7 8 6 8 6 7 8 7 6 7 7 7 7 7 7 7 - 7 7 7 7 7 7 6 6 7 6 6 6 6 6 6 6 6 6 5 5 4
         the         8 7 8 8 7 8 8 7 7 7 7 7 7 7 7 7 7 7 8 - 7 7 7 7 7 7 7 6 7 6 6 6 6 5 5 5 5 5 4 4
        other        8 7 7 8 7 7 7 7 8 7 7 7 8 7 7 7 7 7 7 7 - 7 7 6 7 7 7 7 6 6 7 6 6 6 5 5 5 5 5 5
       process       7 6 6 7 7 7 8 7 7 6 6 7 6 7 6 7 8 7 7 8 7 - 7 7 7 7 7 7 6 6 6 6 6 5 5 5 4 4 4 4
                     7 8 7 7 7 7 7 7 8 8 7 7 7 7 7 6 7 6 7 7 7 7 - 7 7 7 7 6 6 6 4 5 5 6 4 4 4 6 5 5
           |         7 6 7 7 7 6 6 7 6 8 7 8 7 7 7 7 7 7 7 7 7 7 7 - 7 6 6 6 6 5 5 5 6 5 4 4 5 5 4 4
           |         7 7 7 6 7 7 7 7 8 7 6 7 6 6 7 6 6 6 6 7 6 7 7 7 - 6 6 6 5 5 5 5 5 4 4 5 6 4 5 4
           |         6 7 7 7 7 7 7 7 8 8 8 7 7 7 6 7 7 7 6 6 7 7 7 6 5 - 6 5 6 6 5 5 5 4 5 5 5 4 4 4
           |         7 7 6 7 7 7 7 8 7 7 7 7 6 7 7 7 7 7 6 7 6 6 6 5 5 4 - 5 5 5 4 5 5 5 4 5 5 4 4 4
           |         7 7 7 8 7 6 7 6 7 7 7 7 7 6 7 7 7 7 6 6 6 6 6 4 6 4 5 - 5 4 4 5 4 4 5 5 5 4 4 4
           V         7 6 8 7 7 6 6 7 6 7 7 7 7 7 6 7 7 6 6 6 7 6 6 5 6 5 5 4 - 4 5 5 4 4 4 4 4 4 4 5
                     6 6 6 6 6 6 7 8 7 6 7 7 7 7 6 6 7 6 6 5 5 6 6 5 5 6 5 5 4 - 5 4 4 4 4 4 4 6 4 4
                     6 6 6 7 6 7 7 7 7 6 7 7 6 6 7 7 7 6 6 6 6 6 5 4 4 4 5 4 4 4 - 5 5 4 4 4 4 4 4 4
                     7 6 7 6 6 6 7 7 7 6 7 7 6 6 6 7 6 6 6 5 6 5 5 5 5 4 4 4 5 5 6 - 4 4 4 4 4 4 4 4
                     7 7 6 6 6 6 6 7 7 7 6 7 6 7 7 7 6 6 5 5 4 5 5 4 4 4 4 5 4 4 5 4 - 4 4 4 5 4 4 4
                     7 6 7 6 6 6 6 6 7 7 7 7 6 7 6 6 6 6 6 5 5 4 5 4 4 4 4 4 4 4 4 4 4 - 5 4 4 4 4 5
                     7 6 7 7 7 8 7 7 6 6 6 7 6 6 6 6 5 5 4 5 5 5 4 4 5 4 4 4 4 4 4 4 4 4 - 4 4 4 4 4
                     7 6 7 6 7 6 6 6 6 6 7 7 6 6 6 6 5 5 5 4 4 4 4 4 5 4 4 4 4 4 4 4 4 4 4 - 4 4 4 4
                     8 6 6 7 7 7 7 8 7 6 6 7 6 6 6 6 5 4 5 4 5 5 4 5 4 4 5 4 4 4 4 5 5 4 4 4 - 4 4 4
                     7 7 7 6 7 7 6 7 6 6 7 6 6 6 6 5 4 5 4 5 4 4 4 4 4 4 4 4 4 5 4 4 4 4 4 4 4 - 4 4
                     7 7 7 7 7 6 7 7 6 7 7 7 7 5 4 5 5 4 5 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 - 4
                     7 6 7 7 6 7 6 6 6 6 6 6 6 6 5 5 6 4 4 5 4 5 4 5 4 4 4 4 5 4 4 4 5 4 4 4 4 4 4 -

We see that there is a strong dependence on process rank. Presumably, this is due to our polling loop. That is, even if we receive our message, we still have to poll the higher numbered ranks before we complete the receive operation.

Performance Measurements: After Optimization

We consider three metrics:

  • HPCC "pingpong" latency
  • OSU latency (0 bytes)
  • OSU latency (8 bytes)

We report data for:

  • OMPI "out of the box"
  • after implementation of steps 1-2 (send side)
  • after implementation of steps 1-3 (send and receive sides)
  • after implementation of steps 1-4 (send and receive sides, plus data convertor)

The data are from machines that I just happened to have available.

There is a bit of noise in these results, but the implications, based on these and other measurements, are:

  • There is some improvement from the send side.
  • There is more improvement from the receive side.
  • The data convertor improvements help a little more (a few percent) for non-null messages.
  • The degree of improvement depends on how fast the CPU is relative to the memory -- that is, how important software overheads are versus hardware latency.
    • If the CPU is fast (and hardware latency is the bottleneck), these improvements are less -- say, 20-30%.
    • If the CPU is slow (and software costs are the bottleneck), the improvements are more dramatic -- nearly a factor of 2 for non-null messages.
  • As np is increased, latency stays flat. This can represent a 10× or more improvement over out-of-the-box OMPI.

V20z

Here are results for a V20z (burl-ct-v20z-11):

                  HPCC OSU0 OSU8

   out of box      838  770  850   nsec
   Steps 1-2       862  770  860
   Steps 1-3       670  610  670
   Steps 1-4       642  580  610

F6900

Here are np=2 results from a 1.05-GHz (1.2?) UltraSPARC-IV F6900 server:

                  HPCC OSU0 OSU8

   out of box     3430 2770 3340   nsec
   Steps 1-2      2940 2660 3090
   Steps 1-3      1854 1650 1880
   Steps 1-4      1660 1640 1750

Here is the dependence on process count using HPCC:

                   OMPI
             "out of the box"            optimized
 comm       -----------------       -----------------
 size         min   avg   max         min   avg   max

    2        2688  2719  2750        1750  1781  1812   nsec
    4        2812  2875  3000        1750  1802  1812
    6        2875  3050  3250        1687  1777  1812
    8        2875  3299  3625        1687  1773  1812
   10        2875  3447  3812        1687  1789  1812
   12        3063  3687  4375        1687  1796  1813
   16        2687  4093  5063        1500  1784  1875
   20        2812  4492  6000        1687  1788  1875
   24        3125  5026  6562        1562  1776  1875
   28        3250  5326  7250        1500  1764  1813
   32        3500  5830  8375        1562  1755  1875
   36        3750  6199  8938        1562  1755  1875
   40        4062  6753 10187        1500  1742  1812

Note:

  • At np=2, these optimizations lead to a 2× improvement in shared-memory latency.
  • Non-null messages incur more than a 10% penalty, which is largely addressed by our data-convertor optimization.
  • At larger np, we maintain our fast performance while OMPI "out of the box" keeps slowing down more and more and more.

M9000

Here are results for a 128-core M9000. I think the system has:

  • 2 hardware threads per core (but we only use one hardware thread per core)
  • 4 cores per socket
  • 4 sockets per board
  • 4 boards per (half?)
  • 2 (halves?) per system

As one separates the sender and receiver, hardware latency increases. Here is the hierarchy:

                     latency (nsec)       bandwidth (Mbyte/sec)
                 out-of-box  fastpath     out-of-box  fastpath
   (on-socket?)      810        480          2000       2000
    (on-board?)     2050       1820          1900       1900
        (half?)     3030       2840          1680       1680
                    3150       2960          1660       1660

Note:

  • Latency benefits some hundreds of nsecs with fastpath.
  • This latency improvement is striking when the hardware latency is small, but less noticeable as as the hardware latency increases.
  • Bandwidth is not very sensitive to hardware latency (due to prefetch) and not at all to fast-path optimizations.

Here are HPCC pingpong latencies for increasing process counts:

             out-of-box             fastpath
     np  -----------------     -----------------
         min    avg    max     min    avg    max

      2  812    812    812     499    499    499
      4  874    921    999     437    494    562
      8  937   1847   2624     437   1249   1874
     16 1062   2430   2937     437   1557   1937
     32 1562   3850   5437     375   2211   2875
     64 2687   8329  15874     437   2535   3062
     80 3499  16854  41749     374   2647   3437
     96 3812  31159 100812     374   2717   3437
    128 5187 125774 335187     437   2793   3499

The improvements are tremendous:

  • At low np, latencies are low since sender and receiver can be colocated. Nevertheless, fast-path optimizations provided a nearly 2× improvement.
  • As np increases, fast-path latency also increases, but this is due to higher hardware latencies. Indeed, the "min" numbers even drop a little. The "max" fast-path numbers basically only represent the increase in hardware latency.
  • As np increases, OMPI "out of the box" latency suffers catastrophically. Not only is there the issue of more connections to poll, but the polling behaviors of non-participating processes wreak havoc on the performance of measured processes.

    We can separate the two sources of latency degradation by putting the np-2 non-participating processes to sleep. In that case, latency only rises to about 10-20 µsec. So, polling of many connections causes a substantial rise in latency, while the disturbance of hard-poll loops on system performance is responsible for even more degradation.

Actually, even bandwidth benefits:

            out-of-box          fastpath
     np   --------------      -------------
           min  avg  max      min  avg  max

      2   2015 2034 2053     2028 2039 2051
      4   2002 2043 2077     1993 2032 2065
      8   1888 1959 2035     1897 1969 2088
     16   1863 1934 2046     1856 1937 2066
     32   1626 1796 2038     1581 1798 2068
     64   1557 1709 1969     1591 1729 2084
     80   1439 1619 1902     1561 1706 2059
     96   1281 1452 1722     1500 1689 2005
    128    677  835 1276      893 1671 1906

Here, we see that even bandwidth suffers "out of the box" as the number of hard-spinning processes increases. Note the degradation in "out-of-box" average bandwidths as np increases. In contrast, the "fastpath" average holds up well. (The np=128 min fastpath number 893 Mbyte/sec is poor, but analysis shows it to be a measurement outlier.)

MPI_Sendrecv()

We should also get these optimizations into MPI_Sendrecv() in order to speed up the HPCC "ring" results. E.g., here are latencies in µsecs for a performance measurement based on HPCC "ring" tests.

==================================================
np=64
                                            natural random

"out of box"                                   11.7   10.9
fast path                                       8.3    6.2
fast path and 100 warmups                       3.5    3.6
==================================================
np=128 latency
                                            natural random

"out of box"                                  242.9  226.1
fast path                                      56.6   37.0
fast path and 100 warmups                       4.2    4.1
==================================================

There happen to be two problems here:

  • We need fast-path optimizations in MPI_Sendrecv() for improved performance.
  • The MPI collective operation preceding the "ring" measurement has "ragged" exit times. So, the "ring" timing starts well before all of the processes have entered that measurement. This is a separate OMPI performance problem that must be handled as well for good HPCC results.

Open Issues

Here are some open issues:

  • Side effects. Should the sendi and recvi functions leave any side effects if they do not complete the specified operation?

    To my taste, they should not.

    Currently, however, the sendi function is expected to allocate a descriptor if it can, even if it cannot complete the entire send operation.

  • recvi: BTL and match header. An in-coming message starts with a "match header", with such data as MPI source rank, MPI communicator, and MPI tag for performing MPI message matching. Presumably, the BTL knows nothing about this header. Message matching is performed, for example, via PML callback functions. We are aggressively trying to optimize this code path, however, so we should consider alternatives to that approach.

    One alternative is simply for the BTL to perform a byte-by-byte comparison between the received header and the specified header. The PML already tells the BTL how many bytes are in the header.

    One problem with this approach is that the fast path would not be able to support the wildcard tag MPI_ANY_TAG.

    Further, it leaves open the question how one extracts information (such as source or tag) from this header for the MPI_Status structure.

    We can imagine a variety of solutions here, but so far we've implemented a very simple (even if architecturally distasteful) solution: we hardwire information (previously private to the PML) about the match header into the BTL.

    That approach can be replaced with other solutions.

  • MPI_Sendrecv() support. As discussed earlier, we should support fast-path optimizations for "immediate" send-receive operations. Again, this may entail some movement of current OMPI architectural boundaries.

Other optimizations that are needed for good HPCC results include:

  • reducing the degradation due to hard spin waits
  • improving the performance of collective operations (which "artificially" degrade HPCC "ring" test results)