On May 8, 2009, at 9:01 PM, Allan Menezes wrote:
> Does openmpi version 1.3.2 for Fedora Core 10 x86_64 work with 4
> gigabit pci-express ethernet cards per node stably.
It should. I routinely test over 3 or 4 IP interfaces (to include
IPoIB and 1GB/10GB NICs).
> I tried it on six Asus P5Q-VM motherboards with 4 cards and 8GB ram
> Intel Quad core Cpus each as follows:
> eth0 - intel pro 1000 pt pci express gigabit cards.
> eth1 - TP LINK's TG-3468 realtek r8111B chipset pci express gigabit
> eth2 - realtek 8111C chipset gigabit pci express ethernet builtv in
> on mobo
> eth3 - TP LINK's TG-3468 realtek r8111B chipset pci express gigabit
> with all using mtu's of 3000 and the latest intel and realtek drivers
> from their respective websites
> and hand configured and compiled kernel 188.8.131.52
> I tried hpl-2.0 and gotoblas for checking my cluster and get approx
> GFlops if i use
> only eth0, eth1,eth3 or eth0, eth2, eth3 stably
> but i get 203 GFlops with eth0, eth1,eth2,eth3 and the hpl test fail
> after about the third test.
Are you saying that after three consecutive tests, the IP devices
start failing? If so, it sounds like a kernel / driver problem. If
you reboot, do the problems go away?
What happens if you restrict OMPI to just 1 device and run HPL 5-10
times on each device? Do you see the same degradation? E.g., can you
localize which device is causing problems?
> Any help would be very much appreciated as i would like to use 4 eth
> cards per node.
> Note: the measured performance of all cards is approximately 922
> with jumbo frames of 3000
> using Netpipe and NPtcp and with four cards between two nodes i
> with NPmpi
> compiled with openmpi approximately 3400 Mbits/s which is good! Scales
> linearly with 4 times 900 Mbits/sec
> THank you,
> Allan Menezes
> users mailing list