I guess we have strayed a bit from the original post. My personal opinion is
that a number of codes can run in HPC-like mode over Gigabit ethernet, not
just the trivially parallelizable. The hardware components are one key;
PCI-X, low hardware latency NIC (Intel PRO 1000 is 6.6 microsecs vs about 14
for the Bcom 5721), and a non-blocking (that's the key word) switch. Then
you need a good driver and a good MPI software layer. At present MPICH is
ahead of LAM/OpenMPI/MVAPICH in its implementation of optimized collectives.
At least that's how it seems to me (let me say that quickly, before I get
flamed). MPICH got a bad rap performance wise because its TCP driver was
mediocre (compared with LAM and OpenMPI). But MPICH + GAMMA is very fast.
MPIGAMMA even beats out our Infiniband cluster running OpenMPI on the
MPI_Allreduce; the test was with 64 cpus-32 nodes on the GAMMA cluster (Dual
core P4) and 16 nodes on the Infiniband (Dual Dual-core Opterons). The IB
cluster worked out at 24MBytes/sec (vector size/time) and the GigE +
MPIGAMMA was 39MBytes/sec. On the other hand, if I use my own optimized
AllReduce (a simplified version of the one in MPICH) on the IB cluster it
gets 108MByte/sec. So the tricky thing is all the components need to be in
place to get good application performance.
GAMMA is not so easy to set up-I had considerable help from Giuseppe. It has
libraries to compile and the kernel needs to be recompiled. Once I got that
automated I can build and install a new version of GAMMA in about 5 mins.
The MPIGAMMA build is just like MPICH and MPIGAMMA works almost exactly the
same. So any application that will compile under MPICH should compile under
MPIGAMMA, just by changing the path. I have run half a dozen apps with
GAMMA. Netpipe, Netbench (my network tester-a simplified version of IMB),
Susp3D (my own code-a CFD like application), DLPOLY all compile out of the
box. Gromacs compiles but has a couple of "bugs" that crash on execution.
One is an archaic test for MPICH that prevents a clean exit-must have been a
bugfix for an earlier version of MPICH. The other seems to be an fclose of
an unassigned file pointer. It works OK in LAM but my guess is its illegal
strictly speaking. A student was supposed to check on that. VASP also
compiles out of the box if you can compile it with MPICH. But there is a
problem with the MPIGAMMA and the MPI_Alltoall function right now. It works
but it suffers from hangups and long delays. So GAMMA is not good for VASP
at this moment. You see the substantial performance improvements sometimes,
but other times its dreadfully slow. I can reproduce the problem with an
AlltoAll test code and Giuseppe is going to try to debug the problem.
So GAMMA is not a pancea. In most circumstances it is stable and
predictable; much more reproducble than MPI over TCP. But there are still
may be one or two bugs and several issues.
1) Since GAMMA is tightly entwined in the kernel a crash frequently brings
the whole system down, which is a bit annoying; also it can crash other
nodes in the same GAMMA Virtual Machine.
2) NIC's are very buggy hardware-if you look at a TCP driver there are a
large number of hardware bugfixes in them. A number of GAMMA problems can be
traced to this. It's a lot of work to reprogram all the workarounds.
3) GAMMA nodes have to be preconfigured at boot. You can run more than one
job on a GAMMA virtual machine, but it's a little iffy; there can be
interactions between nodes on the same VM even if they are running different
jobs. Different GAMMA VM's need a different VLAN. So a multiuser environment
is still problematic.
4) Giuseppe said MPIGAMMA was a very difficult code to write-so I would
guess a port to OpenMPI would not be trivial. Also I would want to see
optimized collectives in OpenMPI before I switched from MPICH
As far as I know GAMMA is the most advanced non TCP protocol. At core it
really works well, but it still needs a lot more testing and development.
Giuseppe is great to work with if anyone out there is interested. Go to the
MPIGAMMA website for more info