Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Brock Palen (brockp_at_[hidden])
Date: 2007-06-12 10:57:36


On Jun 12, 2007, at 10:52 AM, Brian Barrett wrote:

> On Jun 11, 2007, at 9:27 AM, Brock Palen wrote:
>
>> With openmpi-1.2.0
>>
>> i ran a: ompi_info --param btl tcp
>>
>> and i see reference to:
>>
>> MCA btl: parameter "btl_tcp_min_rdma_size" (current value: "131072")
>> MCA btl: parameter "btl_tcp_max_rdma_size" (current value:
>> "2147483647")
>>
>> Can TCP support RDMA? I thought you needed fancy hardware to get
>> such support? Light on this subject is highly appreciated.
>>
>> Also if a user using ethernet, is trying to up the limit for
>> 'greedy' messages that the btl_tcp_eager_limit? Is there a problem
>> increasing its size? We will test it with his app of-course, but
>> was wondering if there was a 'gotcha' I was going to walk into.
>
> Hi Brock -
>
> The "rdma" part of the TCP transport isn't real RDMA, but just which
> protocol is used by the upper layers to transfer data. In the send/
> receive protocol, receives always involve a copy. Using the RDMA
> protocol (which is pretty simple to fake with a send/receive
> interface), the TCP BTL header includes the remote address and no
> copy is involved. So no, we haven't discovered some hidden interface
> in TCP -- just trying to have as few special cases for various
> interconnects as possible :).
>
> Yes, increasing the btl_tcp_eager_limit is how you increase the
> "greedy" message size. It's currently 64k, and the only problem with
> increasing it is memory usage. With TCP, even if you need to send a
> 4 byte message, 64K will be used on both the sender and receiver
> during transfer and these fragments are free-listed, so you can very
> quickly cause Open MPI to use lots and lots of memory if the eager
> limit is too big. If you start seeing segfauls, bus errors, and
> failed mallocs, you might have bumped the eager limit too high and
> run yourself out of memory...
>
> Brian

Thanks thats exactly what i want to know! (and kind of assumed)
Brock
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>