Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] TIPC BTL code ready for review
From: Xin He (xin.i.he_at_[hidden])
Date: 2011-09-06 07:14:31


Sorry, it took me so long to reply, I was out of office these days.

The paper I sent, the author did the test using 100% CPU, but not in my
case.
But since I did not look into the code of TIPC, I am afraid I cannot
explain it. I can only
show you my result and hope you get the same or even better result in
your environment :)

I did Netpipe test on TCP, I have not found any good tools for TIPC,
maybe I will write one myself later.

/Xin

Send and receive buffers are 16384 and 87380 bytes
(A bug in Linux doubles the requested buffer sizes)
Now starting the main loop
   0: 1 bytes 1960 times --> 0.14 Mbps in 53.00 usec
   1: 2 bytes 1886 times --> 0.29 Mbps in 53.28 usec
   2: 3 bytes 1876 times --> 0.43 Mbps in 53.23 usec
   3: 4 bytes 1252 times --> 0.57 Mbps in 53.28 usec
   4: 6 bytes 1407 times --> 0.87 Mbps in 52.74 usec
   5: 8 bytes 948 times --> 1.14 Mbps in 53.35 usec
   6: 12 bytes 1171 times --> 1.71 Mbps in 53.40 usec
   7: 13 bytes 780 times --> 1.86 Mbps in 53.32 usec
   8: 16 bytes 865 times --> 2.28 Mbps in 53.43 usec
   9: 19 bytes 1052 times --> 2.71 Mbps in 53.57 usec
  10: 21 bytes 1178 times --> 2.99 Mbps in 53.57 usec
  11: 24 bytes 1244 times --> 3.41 Mbps in 53.62 usec
  12: 27 bytes 1320 times --> 3.84 Mbps in 53.60 usec
  13: 29 bytes 829 times --> 4.14 Mbps in 53.47 usec
  14: 32 bytes 902 times --> 4.57 Mbps in 53.48 usec
  15: 35 bytes 993 times --> 4.98 Mbps in 53.62 usec
  16: 45 bytes 1065 times --> 6.40 Mbps in 53.63 usec
  17: 48 bytes 1242 times --> 6.82 Mbps in 53.66 usec
  18: 51 bytes 1281 times --> 7.25 Mbps in 53.65 usec
  19: 61 bytes 730 times --> 8.68 Mbps in 53.65 usec
  20: 64 bytes 916 times --> 9.10 Mbps in 53.65 usec
  21: 67 bytes 961 times --> 9.52 Mbps in 53.69 usec
  22: 93 bytes 1000 times --> 13.17 Mbps in 53.89 usec
  23: 96 bytes 1237 times --> 13.58 Mbps in 53.94 usec
  24: 99 bytes 1255 times --> 13.99 Mbps in 54.00 usec
  25: 125 bytes 673 times --> 17.64 Mbps in 54.06 usec
  26: 128 bytes 917 times --> 18.06 Mbps in 54.08 usec
  27: 131 bytes 938 times --> 18.46 Mbps in 54.14 usec
  28: 189 bytes 958 times --> 26.47 Mbps in 54.47 usec
  29: 192 bytes 1223 times --> 26.82 Mbps in 54.62 usec
  30: 195 bytes 1230 times --> 27.22 Mbps in 54.66 usec
  31: 253 bytes 638 times --> 35.26 Mbps in 54.75 usec
  32: 256 bytes 909 times --> 35.43 Mbps in 55.12 usec
  33: 259 bytes 914 times --> 35.84 Mbps in 55.14 usec
  34: 381 bytes 924 times --> 51.58 Mbps in 56.36 usec
  35: 384 bytes 1182 times --> 51.87 Mbps in 56.49 usec
  36: 387 bytes 1184 times --> 52.21 Mbps in 56.55 usec
  37: 509 bytes 603 times --> 63.56 Mbps in 61.10 usec
  38: 512 bytes 816 times --> 62.56 Mbps in 62.44 usec
  39: 515 bytes 803 times --> 62.56 Mbps in 62.81 usec
  40: 765 bytes 803 times --> 82.00 Mbps in 71.18 usec
  41: 768 bytes 936 times --> 82.32 Mbps in 71.18 usec
  42: 771 bytes 938 times --> 82.59 Mbps in 71.22 usec
  43: 1021 bytes 473 times --> 102.64 Mbps in 75.89 usec
  44: 1024 bytes 658 times --> 102.92 Mbps in 75.91 usec
  45: 1027 bytes 659 times --> 103.06 Mbps in 76.03 usec
  46: 1533 bytes 660 times --> 134.60 Mbps in 86.89 usec
  47: 1536 bytes 767 times --> 134.90 Mbps in 86.87 usec
  48: 1539 bytes 768 times --> 135.18 Mbps in 86.86 usec
  49: 2045 bytes 385 times --> 173.37 Mbps in 89.99 usec
  50: 2048 bytes 555 times --> 173.63 Mbps in 89.99 usec
  51: 2051 bytes 556 times --> 173.22 Mbps in 90.34 usec
  52: 3069 bytes 554 times --> 225.61 Mbps in 103.78 usec
  53: 3072 bytes 642 times --> 225.97 Mbps in 103.72 usec
  54: 3075 bytes 643 times --> 226.14 Mbps in 103.74 usec
  55: 4093 bytes 322 times --> 277.99 Mbps in 112.33 usec
  56: 4096 bytes 445 times --> 277.98 Mbps in 112.42 usec
  57: 4099 bytes 444 times --> 277.72 Mbps in 112.60 usec
  58: 6141 bytes 444 times --> 400.61 Mbps in 116.95 usec
  59: 6144 bytes 570 times --> 402.32 Mbps in 116.51 usec
  60: 6147 bytes 572 times --> 400.78 Mbps in 117.02 usec
  61: 8189 bytes 285 times --> 458.70 Mbps in 136.20 usec
  62: 8192 bytes 367 times --> 460.25 Mbps in 135.80 usec
  63: 8195 bytes 368 times --> 461.14 Mbps in 135.58 usec
  64: 12285 bytes 368 times --> 497.80 Mbps in 188.28 usec
  65: 12288 bytes 354 times --> 495.96 Mbps in 189.03 usec
  66: 12291 bytes 352 times --> 498.39 Mbps in 188.15 usec
  67: 16381 bytes 177 times --> 562.50 Mbps in 222.18 usec
  68: 16384 bytes 225 times --> 563.89 Mbps in 221.68 usec
  69: 16387 bytes 225 times --> 562.61 Mbps in 222.22 usec
  70: 24573 bytes 225 times --> 629.04 Mbps in 298.04 usec
  71: 24576 bytes 223 times --> 632.04 Mbps in 296.66 usec
  72: 24579 bytes 224 times --> 628.97 Mbps in 298.14 usec
  73: 32765 bytes 111 times --> 667.51 Mbps in 374.49 usec
  74: 32768 bytes 133 times --> 668.03 Mbps in 374.24 usec
  75: 32771 bytes 133 times --> 667.54 Mbps in 374.54 usec
  76: 49149 bytes 133 times --> 706.32 Mbps in 530.89 usec
  77: 49152 bytes 125 times --> 705.28 Mbps in 531.70 usec
  78: 49155 bytes 125 times --> 706.43 Mbps in 530.87 usec
  79: 65533 bytes 62 times --> 746.28 Mbps in 669.96 usec
  80: 65536 bytes 74 times --> 750.98 Mbps in 665.80 usec
  81: 65539 bytes 75 times --> 745.64 Mbps in 670.59 usec
  82: 98301 bytes 74 times --> 786.29 Mbps in 953.81 usec
  83: 98304 bytes 69 times --> 786.03 Mbps in 954.17 usec
  84: 98307 bytes 69 times --> 785.73 Mbps in 954.56 usec
  85: 131069 bytes 34 times --> 822.93 Mbps in 1215.15 usec
  86: 131072 bytes 41 times --> 825.56 Mbps in 1211.31 usec
  87: 131075 bytes 41 times --> 822.65 Mbps in 1215.61 usec
  88: 196605 bytes 41 times --> 847.04 Mbps in 1770.85 usec
  89: 196608 bytes 37 times --> 849.10 Mbps in 1766.57 usec
  90: 196611 bytes 37 times --> 846.81 Mbps in 1771.38 usec
  91: 262141 bytes 18 times --> 853.36 Mbps in 2343.64 usec
  92: 262144 bytes 21 times --> 853.44 Mbps in 2343.45 usec
  93: 262147 bytes 21 times --> 853.69 Mbps in 2342.81 usec
  94: 393213 bytes 21 times --> 865.59 Mbps in 3465.83 usec
  95: 393216 bytes 19 times --> 865.40 Mbps in 3466.61 usec
  96: 393219 bytes 19 times --> 865.48 Mbps in 3466.31 usec
  97: 524285 bytes 9 times --> 871.99 Mbps in 4587.17 usec
  98: 524288 bytes 10 times --> 871.85 Mbps in 4587.95 usec
  99: 524291 bytes 10 times --> 872.13 Mbps in 4586.50 usec
100: 786429 bytes 10 times --> 878.77 Mbps in 6827.70 usec
101: 786432 bytes 9 times --> 879.14 Mbps in 6824.83 usec
102: 786435 bytes 9 times --> 878.82 Mbps in 6827.39 usec
103: 1048573 bytes 4 times --> 884.29 Mbps in 9046.74 usec
104: 1048576 bytes 5 times --> 884.41 Mbps in 9045.60 usec
105: 1048579 bytes 5 times --> 884.15 Mbps in 9048.29 usec
106: 1572861 bytes 5 times --> 887.90 Mbps in 13514.99 usec
107: 1572864 bytes 4 times --> 887.90 Mbps in 13515.01 usec
108: 1572867 bytes 4 times --> 887.81 Mbps in 13516.38 usec
109: 2097149 bytes 3 times --> 889.80 Mbps in 17981.51 usec
110: 2097152 bytes 3 times --> 889.91 Mbps in 17979.33 usec
111: 2097155 bytes 3 times --> 889.90 Mbps in 17979.65 usec
112: 3145725 bytes 3 times --> 892.55 Mbps in 26889.17 usec
113: 3145728 bytes 3 times --> 892.60 Mbps in 26887.83 usec
114: 3145731 bytes 3 times --> 892.57 Mbps in 26888.68 usec
115: 4194301 bytes 3 times --> 893.98 Mbps in 35795.15 usec
116: 4194304 bytes 3 times --> 893.95 Mbps in 35796.01 usec
117: 4194307 bytes 3 times --> 893.94 Mbps in 35796.66 usec
118: 6291453 bytes 3 times --> 895.36 Mbps in 53609.49 usec
119: 6291456 bytes 3 times --> 895.36 Mbps in 53609.49 usec
120: 6291459 bytes 3 times --> 895.39 Mbps in 53608.00 usec
121: 8388605 bytes 3 times --> 895.86 Mbps in 71439.65 usec
122: 8388608 bytes 3 times --> 895.87 Mbps in 71438.84 usec
123: 8388611 bytes 3 times --> 895.80 Mbps in 71444.32 usec

On 09/01/2011 04:08 PM, Jeff Squyres wrote:
> On Sep 1, 2011, at 7:05 AM, Xin He wrote:
>
>> And get the result as in appendix. It seems that TCP has better performances with smaller message while TIPC with larger message.
> Interesting. Any idea why? From the TIPC paper you sent, one of TIPC's strengths was that it was supposed to be faster than TCP for small messages.
>
> Do you know what the raw performance numbers are for TCP and TIPC on this machine without MPI?
>