Hi Jeff,
thanks for you reply. I did understand previous questions about RDMA. Ever with SKaMPI, i tried to run with mpi_leave_pinned = 1, as you have suggested. But also in this case, execution time is very similar to previous case.
Does it means that SKaMPI, reallocates buffer every time ? For example, with "MPI_Bcast-length" test, over 128 procs, the collective is repeated about 28 times, increasing buffer size for each step by internal formula, and finale buffer size =2097152 K.
Since there aren't advantages with leave_pinned = 1, it means that SKaMPI doesn't allocates buffer of 2097152 K initially, but it allocates small buffer and reallocates buffer every time, with more large size. Is it possible? If no, which is the cause of similar performance?
Another question: RDMA pipeline protocol for long messages, in OpenMPI 1.2.6 is setting by default?2008/6/6 Jeff Squyres <jsquyres@cisco.com>:
Note that "eager" RDMA is only used for short messages -- it's not
really relevant to whether the same user buffers are re-used or not
(the mpi_leave_pinned parameter for long messages is only useful if
long buffers are re-used). See this FAQ item:
http://www.open-mpi.org/faq/?category=openfabrics#ib-small-message-rdma
For benchmarks (like SKAMPI) that re-use long buffers, you might want
to use the mpi_leave_pinned MCA parameter:
http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned
http://www.open-mpi.org/faq/?category=tuning#running-perf-numbers
> g.fatigati@cineca.it _______________________________________________
On Jun 5, 2008, at 9:47 AM, Gabriele Fatigati wrote:
>
> Hi,
> i'm testing SKaMPI Benchmark on IBM Blade System over Infiniband.
> Current version of OpenMPI is 1.2.6
> I have tried to disable RDMA setting btl_openib_use_eager_rdma = 0.
> But, i have noted that, in MPI collectives execution time, there are
> few difference beetween RDMA active and none. Before tests, I
> expected that with RDMA off, excecution time was more long.
>
> So, i suppose that SKaMPI benchmark does continues reallocation of
> buffers that forbid benefits of RDMA protocol. Indeed, if initial
> buffer address change every time, we have to do very much
> registration of memory pages afterwards decay of perfomance.
>
> I used RDMA pipeline protocol. This protocol should makes no
> assumption about the application reuse of source and target buffers.
> But, is it every true?
> Parameters net are explained below.
>
> MCA btl: parameter "btl_openib_mpool" (current value: "rdma")
> MCA btl: parameter "btl_openib_ib_max_rdma_dst
> _ops" (current value: "4")
> MCA btl: parameter "btl_openib_use_eager_rdma" (current value: "1")
> MCA btl: parameter "btl_openib_eager_rdma_threshold" (current value:
> "16")
> MCA btl: parameter "btl_openib_max_eager_rdma" (current value: "16")
> MCA btl: parameter "btl_openib_eager_rdma_num" (current value: "16")
> MCA btl: parameter "btl_openib_min_rdma_size" (current value:
> "1048576")
> MCA btl: parameter "btl_openib_max_rdma_size" (current value:
> "1048576")
>
> --
> Gabriele Fatigati
>
> CINECA Systems & Tecnologies Department
>
> Supercomputing Group
>
> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>
> www.cineca.it Tel: +39 051 6171722
>
> users mailing list
> users@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Gabriele Fatigati
CINECA Systems & Tecnologies Department
Supercomputing Group
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati@cineca.it