On Thu, 16 Mar 2006, Jean Latour wrote:
> My questions are :
> a) Is OpenMPI doing in this case TCP/IP over IB ? (I guess so)
If the path to the mvapi library is correct then Open MPI will use mvapi
not TCP over IB. There is a simple way to check. "ompi_info --param btl
mvapi" will print all the parameters attached to the mvapi driver. If
there is no mvapi in the output, then mvapi was not correctly detected.
But I don't think it's the case, because if I remember well we have a
protection at configure time. If you specify one of the drivers and we're
not able to correctly use the libraries, we will stop the configure.
> b) Is it possible to improve significantly these values by changing the
> defaults ?
By default we are using a very conservative approach. We never leave the
memory pinned down, and that decrease the performance for a ping-pong.
There are pro and cons for that, too long to be explained here, but in
general we're seeing better performance for real-life applications with
our default approach, and that's our main goal.
Now, if you want to get better performance for the ping-pong test please
read the FAQ at http://www.open-mpi.org/faq/?category=infiniband.
These are the 3 flags that affect the mvapi performance for the ping-pong
case (add them in $(HOME)/.openmpi/mca-params.conf):
> I have used several mca btl parameters but without improving the maximum
> For example : --mca btl mvapi --mca btl_mvapi_max_send_size 8388608
It is difficult to improve the maximum bandwidth without the leave_pinned
activated. But you can improve the bandwidth for medium size messages.
Play with btl_mvapi_eager_limit to set the limit between short and
rendez-vous protocol. "ompi_info --param btl mvapi" will give you a full
list of parameters as well as their description.
> c) Is it possible that other IB hardware implementations have better
> performances with OpenMPI ?
The maximum bandwidth depend on several factors. One of the most
importants is the maximum bandwidth on your node bus. To reach 800 and
more MB/s you definitively need a PCI-X 16 ...
> d) Is it possible to use specific IB drivers for optimal performance ?
> (should reach almost 800 MB/sec)
Once the 3 options are set, you should see an improvement on the
Let me know if it does not solve your problems.
"We must accept finite disappointment, but we must never lose infinite
Martin Luther King