Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] [EXTERNAL] Re: RFC: Change ompi_proc_t endpoint data lookup
From: Barrett, Brian W (bwbarre_at_[hidden])
Date: 2013-07-19 13:29:11

On 7/19/13 10:58 AM, "George Bosilca" <bosilca_at_[hidden]> wrote:

>1. The BML endpoint structure (aka. BML proc) is well known and defined
>in the bml.h. So it is not technically opaqueŠ

It's opaque in that outside of the R2 BML, users were not supposed to poke
at what's in proc_bml without using the BML interface. Some do, although
that was easy to accommodate.

>2. When allocating an ompi_proc_t structure you will always have to
>allocate for an array large enough to contain up to the max size detected
>during configure. There is significant potential for oversized arrays in
>one of the most space critical structure.

It could, if we're not careful with our tag requests. In the prototype I
wrote up, the sizes of endpoint storage in ompi_proc_t are as follows:

  * Current trunk: 16 bytes
  * Proposed trunk, no dynamic support, no MTLs: 8 bytes
  * Proposed trunk, dynamic support, no MTLs: 16 bytes
  * Proposed trunk, dynamic support, MXM, PSM, or MX: 24 bytes
  * Proposed trunk, Portals, no dynamic support: 16 bytes
  * Proposed trunk, Portals, dynamic support: 24 bytes
  * Proposed trunk, Portals, MX, PSM, or MXM, dynamic support: 32 bytes

So, yes, you're right. But the situations where you see growth are not
normal OMPI builds (for example, Portals & MXM support). In the common
cases, we could actually shrink by 8 bytes by disabling dynamic support.
It would also (finally) allow us to run the MTLs and BTLs simultaneously,
which is something we haven't been able to do previously.

>3. I don't know at which point this really matter but with this change
>two Open MPI libraries might become binary incompatible (if the #define
>is exchanged between nodes).

The #defines are local process only. ompi_proc_ts aren't global
structures (the pointer to them is), so there's no binary incompatibility.

I hacked up a prototype in tmp-public/snl-proc-tags/ last night. It
currently lacks dynamic support (since we have no users for that), but
otherwise works.


  Brian W. Barrett
  Scalable System Software Group
  Sandia National Laboratories

  • application/pkcs7-signature attachment: smime.p7s