Nathan, could you please advise what is expected startup time for OpenMPI
job at such scale (128K ranks)? I'm interesting in
1) time from mpirun start to completion of MPI_Init()
2) time from MPI_Init() start to completion of MPI_Init()
>From my experience for 52800 rank job
1) took around 20 min
2) took around 12 min
that actually looks like a hung.
Any advice how to improve startup times of large scale jobs would be very
From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On
Behalf Of Nathan Hjelm
Sent: Tuesday, April 23, 2013 2:47 AM
To: Open MPI Users
Subject: Re: [OMPI users] OpenMPI at scale on Cray XK7
On Mon, Apr 22, 2013 at 03:17:16PM -0700, Mike Clark wrote:
> I am trying to run OpenMPI on the Cray XK7 system at Oak Ridge National
Lab (Titan), and am running in an issue whereby MPI_Init seems to hang
indefinitely, but this issue only arises at large scale, e.g., when
running on 18560 compute nodes (with two MPI processes per node). The
application runs successfully on 4600 nodes, and we are currently trying
to test a 9000 node job to see if this fails or runs.
> We are launching our job using something like the following
> # mpirun command
> mpicmd="$OMP_DIR/bin/mpirun --prefix $OMP_DIR -np 37120 --npernode 2
--bind-to core --bind-to numa $app $args"
> # Print and Run the Command
> echo $mpicmd
> $mpicmd >& $output
> Are there any issues that I should be aware of when running OpenMPI on
37120 processes or when running on the Cray Gemini Interconnect?
We have only tested Open MPI up to 131072 ranks on 8192 nodes. Have you
tried running DDT on the process to see where it is hung up?
I have a Titan account so I can help with debugging. I would like to get
this issue fixed in 1.7.2.
users mailing list