Torsten Hoefler wrote:
> Hi Joachim,
>> Is there a formula to calculate the number of socket connections per
>> node (S) that an OpenMPI application needs for running (via sockets) on
>> N nodes with P processes each? I guess something like
>> S = P * (N-1)*P
> I would say that S=(N-1)*P (why do you assume that S=O(P^2)?) is an
For N nodes running P processes each, each single process sees (N-1)*P
remote processes. I'd say this results in (N-1)*P^2 connections *per node*.
> upper bound for the MPI layer (each rank connects to all other ranks
> besides the ones on its node (connect via sm) and there is no connection
> 'aggregation' on a single node yet). However, the connection
> establishment is lazy (each connection is established with the first
> send - see btl_tcp_endpoint.c:495). So you'll have the fully-connected
> case after an MPI_Alltoall with a sufficient size or a similar send/recv
> pattern :).
O.k. But what happens if a connection can not be established due to
resource limits? Are existing connections closed? O
> You could try to use the hierarch collective component to minimize the
> number of active connections (open sockets) per node.
Yes, this also can help to increase throughput for small data size.
Joachim Worringen, Software Architect, Dolphin Interconnect Solutions
phone ++49/(0)228/324 08 17 - http://www.dolphinics.com