If they are 8 core Intel machines, I believe this is the case:
*) Each pair of cores share an L2-cache. So using two cores that share
cache will probably reduce performance.
*) Each Quad core CPU has its own memory bus (Dual independent bus),
so using more than one core on a quad CPU can reduce performance if
the bus is a bottle neck.
In your first case, both L2-cache and memory bus are shared. In your
second case, only memory bus is shared. In your third case no L2-cache
or memory bus are shared (The Linux scheduler maps processes so that
they run on different CPUs if possible).
If you want another performance-case, you can map the processes such
that they run on 4 different nodes, but share L2-cache. This can be
done by something like this mpirun -n 8 taskset -cp 0,4 LU.C.8 . Core
ID 0 and 4 share L2-cache on our system at least. I guess you are not
that interested, but it is possible! :)
In addition I don't believe there is very much communication happening
in the LU-benchmark compared to the other NAS benchmarks.
All in all, I agree with both of you. Both the L2-cache and the memory
bus are probably slowing you down.
As for the sys% time, I believe it is the NIC driver. The more inter-
node communication, the more sys%. The shared memory communication
module (BTL SM) does all its communication in user space, as you
-Torje S. Henriksen
On Sep 30, 2008, at 6:55 PM, Jeff Squyres wrote:
> Are these intel-based machines? I have seen similar effects
> mentioned earlier in this thread where having all 8 cores banging on
> memory pretty much kills performance on the UMA-style intel 8 core
> machines. I'm not a hardware expert, but I've stayed away from
> buying 8-core servers for exactly this reason. AMD's been NUMA all
> along, and Intel's newer chips are NUMA to alleviate some of this
> bus pressure.
> ~2x performance loss (between 8 and 4 cores on a single node) seems
> a bit excessive, but I guess it could happen...? (I don't have any
> hard numbers either way)
> On Sep 29, 2008, at 2:30 PM, Leonardo Fialho wrote:
>> Hi All,
>> I´m doing some probes in a multi core (8 cores per node) machine
>> with NAS benchmarks. Something that I consider strange is
>> I´m using only one NIC and paffinity:
>> -n 8
>> --hostfile ./hostfile
>> --mca mpi_paffinity_alone 1
>> --mca btl_tcp_if_include eth1
>> I have sufficient memory to run this application in only one node,
>> 1) If I use one node (8 cores) the "user" % is around 100% per
>> core. The execution time is around 430 seconds.
>> 2) If I use 2 nodes (4 cores in each node) the "user" % is around
>> 95% per core and the "sys" % is 5%. The execution time is around
>> 220 seconds.
>> 3) If I use 4 nodes (1 cores in each node) the "user" % is around
>> %85 per core and the "sys" % is 15%. The execution time is around
>> 200 seconds.
>> Well... the questions are:
>> A) The execution time in case "1" should be smaller (only sm
>> communication, no?) than case "2" and "3", no? Cache problems?
>> B) Why the "sys" time while using communication inter nodes? NIC
>> driver? Why this time increase when I balance the load across the
>> Leonardo Fialho
>> Computer Architecture and Operating Systems Department - CAOS
>> Universidad Autonoma de Barcelona - UAB
>> ETSE, Edifcio Q, QC/3088
>> Phone: +34-93-581-2888
>> Fax: +34-93-581-2478
>> users mailing list
> Jeff Squyres
> Cisco Systems
> users mailing list