Hi,<br />&nbsp;&nbsp;&nbsp; The code was pretty simple. I was trying to send 8MB data from one rank to other in a loop(say 1000 iterations). And then i was taking the average of time taken and was calculating the bandwidth.<br /><br />The above logic i tried with both mpirun-with-mca-parameters and without any parameters. And to my surprise, the performance was degrading when i was trying to manipulate.<br /><br />Now I have another question in mind. Is it possible to have IB Hardware Multicast implementation in OpenMPI? I have gone through the issues/challenges for the same, but also read couple of people who have successfully done it for Ethernet/Giga-bit Ethernet and IPoIB ofcourse in experimental stage. Actually i want to contribute for it in OpenMPI and need the help for the same.<br /><br />-Neeraj<br /><br />On Thu, 11 Oct 2007 12:01:39 +0200 Open MPI Users  wrote<br /><br />  Hi Neeraj,<br /><br />  &gt;        Could anyone tell me the important tuning parameters in openmpi with<br /><br />  &gt;    IB interconnect? I tried setting eager_rdma, min_rdma_size,<br /><br />  &gt;    mpi_leave_pinned parameters from the mpirun command line on 38 nodes<br /><br />  &gt;    cluster (38*2 processors) but in vain. I found simple mpirun with no mca<br /><br />  &gt;    parameters performing better. I conducted test on P2P send/receive with<br /><br />  &gt;    data size of 8MB.<br /><br />  The performance of the BTL with different parameters depends heavily on<br /><br />  the code that you run. E.g., leave_pinned works very well with many<br /><br />  microbenchmarks (e.g., bandwidth/overlap-wise) but may not perform well<br /><br />  with real applications that use different memory regions. It's pretty<br /><br />  much the same with the other parameters. The default values are<br /><br />  considered best for many applications. Can you provide us any details<br /><br />  about the code you're runnning to test performance? <br /><br />  <br /><br />  &gt;        Similarly i patched HPL linpack code with libnbc(non blocking<br /><br />  &gt;    collectives) and found no performance benefits. I went through its patch<br /><br />  &gt;    and found that, its probably not overlapping computation with<br /><br />  &gt;    communication.<br /><br />  Ah, so there are two things. LibNBC provides overlap, most overlap is<br /><br />  achieved if memory regions are reused and leave_pinned is activated. But<br /><br />  again, this is highly application-dependent. However, the patch for the<br /><br />  Linpack code (I guess you refer to the patch from the LibNBC webpage<br /><br />  [1]) is in experimental stage (as the website says) and is not properly<br /><br />  tested for performance benefit. The original HPL provides something like<br /><br />  a broadcast start and broadcast end phase. I just replaced them with<br /><br />  non-blocking calls to NBC_Ibcast() and did not find the time to do any<br /><br />  performance/code analysis yet. Any input by HPL experts is appreciated!<br /><br />  <br /><br />  Best,<br /><br />    Torsten<br /><br />  <br /><br />  [1]: http://www.unixer.de/research/nbcoll/hpl/<br /><br />  <br /><br />  -- <br /><br />   bash$ :(){ :|:&amp;};: --------------------- http://www.unixer.de/ -----<br /><br />  "Software Engineering is that part of Computer Science which is too<br /><br />  difficult for the Computer Scientist." ~ F. L. Bauer<br /><br />  _______________________________________________<br /><br />  users mailing list<br /><br />  users@open-mpi.org<br /><br />  http://www.open-mpi.org/mailman/listinfo.cgi/users<br /><br />  
<br><Table border=0 Width=644 Height=57 cellspacing=0 cellpadding=0 style='font-family:Verdana;font-size:11px;line-height:15px;'><TR><td><a href='http://adworks.rediff.com/cgi-bin/AdWorks/click.cgi/www.rediff.com/signature-home.htm/1050715198@Middle5/1473272_1466559/1472748/1?PARTNER=3&OAS_QUERY=null target=new '><img src =http://imadworks.rediff.com/cgi-bin/AdWorks/adimage.cgi/1473272_1466559/creative_1472748.gif  alt='BM'  border=0></a></td></TR></Table>