Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Remote key sizes
From: Kenneth Lloyd (kenneth.lloyd_at_[hidden])
Date: 2011-11-08 18:31:06

That makes sense to me.

-----Original Message-----
From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]] On
Behalf Of Nathan T. Hjelm
Sent: Tuesday, November 08, 2011 8:36 AM
To: Open MPI Developers
Subject: Re: [OMPI devel] Remote key sizes

On Tue, 8 Nov 2011 06:36:03 -0800, Rolf vandeVaart <rvandevaart_at_[hidden]>
>> george.
>>PS: Regarding the hand-copy instead of the memcpy, we tried to avoid
> using
>>memcpy in performance critical codes, especially when we know the size of
>>the data and the alignment. This relieves the compiler of adding ugly
> intrinsics,
>>allowing it to nicely pipeline to load/stores. Anyway, with both
> approaches
>>you will copy more data than needed for all BTLs except uGNI.
> I was looking at a case in a BTL I was working on where I actually need
> bytes (yes, bytes) as the remote key size as opposed to the current 16
> bytes (128 bits).
> Not sure how I can handle that yet. (I assume configure is my friend,
> even in that case, all headers will need to carry around the extra data.)

I have been thinking about this a little bit. What I think should be done
(and I am sure George will disagree) is to allow BTLs to define how long a
segment is. The PML would then just memcpy the segments into the send
buffer (instead of copying each member).

For example mca_btl_base_segment_t would become:

struct mca_btl_base_segment_t {
    size_t seg_len;

since the pml needs the segment size (it does not need anything else).

and then each btl would define its own segment like:
struct mca_btl_ugni_segment_t {
    struct mca_btl_base_segment_t base;
    gni_mem_handle_t seg_key;

and we would add:
size_t btl_segment_len;

to the mca_btl_base_module_t or the base frag so the pml knows how much it
needs to copy.

This design would address George's criticism of the length of the seg_key
and also allow BTLs to do what they need to. It would require a memcpy but
I disagree this would slow the critical path. Even if it does it would be
relatively minor (i think) and the flexibility is worth more in the long


devel mailing list
No virus found in this message.
Checked by AVG -
Version: 10.0.1411 / Virus Database: 2092/4003 - Release Date: 11/07/11