> >> Is there a formula to calculate the number of socket connections per
> >> node (S) that an OpenMPI application needs for running (via sockets) on
> >> N nodes with P processes each? I guess something like
> >> S = P * (N-1)*P
> > I would say that S=(N-1)*P (why do you assume that S=O(P^2)?) is an
> For N nodes running P processes each, each single process sees (N-1)*P
> remote processes. I'd say this results in (N-1)*P^2 connections *per node*.
you're right, I didn't read the *per node* :).
> > upper bound for the MPI layer (each rank connects to all other ranks
> > besides the ones on its node (connect via sm) and there is no connection
> > 'aggregation' on a single node yet). However, the connection
> > establishment is lazy (each connection is established with the first
> > send - see btl_tcp_endpoint.c:495). So you'll have the fully-connected
> > case after an MPI_Alltoall with a sufficient size or a similar send/recv
> > pattern :).
> O.k. But what happens if a connection can not be established due to
> resource limits? Are existing connections closed? O
No, connections are never closed afaik (there is also the problem to
determine which connection should be closed :). If the connection can
not be established (e.g. socket() or connect() returns values < 0), the
peer is marked as not reachable and the send fails (ompi exits with an
> > You could try to use the hierarch collective component to minimize the
> > number of active connections (open sockets) per node.
> Yes, this also can help to increase throughput for small data size.
sure, and it could avoid potential flow-control problems for large data.
"It is easier to write an incorrect program than understand a correct
one." Alan J. Perlis.