Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Question about RDMA
From: Gabriele Fatigati (g.fatigati_at_[hidden])
Date: 2008-06-05 09:47:39


Hi,
i'm testing SKaMPI Benchmark on IBM Blade System over Infiniband. Current
version of OpenMPI is 1.2.6
I have tried to disable RDMA setting btl_openib_use_eager_rdma = 0. But, i
have noted that, in MPI collectives execution time, there are few difference
beetween RDMA active and none. Before tests, I expected that with RDMA off,
excecution time was more long.

So, i suppose that SKaMPI benchmark does continues reallocation of buffers
that forbid benefits of RDMA protocol. Indeed, if initial buffer address
change every time, we have to do very much registration of memory pages
afterwards decay of perfomance.

I used RDMA pipeline protocol. This protocol should makes no assumption
about the application reuse of source and target buffers. But, is it every
true?
Parameters net are explained below.

MCA btl: parameter "btl_openib_mpool" (current value: "rdma")
MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current value: "4")
MCA btl: parameter "btl_openib_use_eager_rdma" (current value: "1")
MCA btl: parameter "btl_openib_eager_rdma_threshold" (current value: "16")
MCA btl: parameter "btl_openib_max_eager_rdma" (current value: "16")
MCA btl: parameter "btl_openib_eager_rdma_num" (current value: "16")
MCA btl: parameter "btl_openib_min_rdma_size" (current value: "1048576")
MCA btl: parameter "btl_openib_max_rdma_size" (current value: "1048576")

-- 
Gabriele Fatigati
CINECA Systems & Tecnologies Department
Supercomputing Group
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati_at_[hidden]