Miguel Figueiredo Mascarenhas Sousa Filipe wrote:
>>> the MPI model assumes you don't have a "shared memory" system..
>>> therefore it is "message passing" oriented, and not designed to
>>> perform optimally on shared memory systems (like SMPs, or numa-CCs).
>> For many programs with both MPI and shared memory implementations, the
>> MPI version runs faster on SMPs and numa-CCs. Why? See the previous
> Of course it does..its faster to copy data in main memory than it is
> to do it thought any kind of network interface. You can optimize you
> message passing implementation to a couple of memory to memory copies
> when ranks are on the same node. In the worst case, even if using
> local IP addresses to communicate between peers/ranks (in the same
> node), the operating system doesn't even touch the interface.. it
> will just copy data from a tcp sender buffer to a tcp receiver
> buffer.. in the end - that's always faster than going through a
> phisical network link.
There are a lot of papers about the relative merits of a mixed shared
MPI model - OpenMP on-node and MPI inter-node, for example. Generally they
seem to show that MPI is at least as good.
It isn't true that copying data in main memory (with a CPU) is always
the NIC. Several effects arise: when you copy between address spaces,
then the OS overhead
of making a shared memory region can be high, the CPUs may not be as
good at copying
as the NIC, having the CPU do it can trash the caches, and leave the
copied data in the
wrong cache altogether, and having the CPU do it loses the parallelism
you can get
by letting a separate processor (the NIC) do the work.