Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Open-MPI build of NAMD launched from srun over 20% slowed than with mpirun
From: Christopher Samuel (samuel_at_[hidden])
Date: 2013-09-04 22:15:38


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/09/13 18:33, George Bosilca wrote:

> You can confirm that the slowdown happen during the MPI
> initialization stages by profiling the application (especially the
> MPI_Init call).

NAMD helpfully prints benchmark and timing numbers during the initial
part of the simulation, so here's what they say. For both seconds
per step and days per nanosecond of simulation less is better.

I've included the benchmark numbers (every 100 steps or so from the
start) and the final timing number after 25000 steps. It looks like
to me (as a sysadmin and not an MD person) that the final timing
number includes CPU time in seconds per step and wallclock time in
seconds per step.

64 cores over 10 nodes:

OMPI 1.7.3a1r29103 mpirun

Info: Benchmark time: 64 CPUs 0.410424 s/step 2.37514 days/ns 909.57 MB memory
Info: Benchmark time: 64 CPUs 0.392106 s/step 2.26913 days/ns 909.57 MB memory
Info: Benchmark time: 64 CPUs 0.313136 s/step 1.81213 days/ns 909.57 MB memory
Info: Benchmark time: 64 CPUs 0.316792 s/step 1.83329 days/ns 909.57 MB memory
Info: Benchmark time: 64 CPUs 0.313867 s/step 1.81636 days/ns 909.57 MB memory

TIMING: 25000 CPU: 8247.2, 0.330157/step Wall: 8247.2, 0.330157/step, 0.0229276 hours remaining, 921.894531 MB of memory in use.

OMPI 1.7.3a1r29103 srun

Info: Benchmark time: 64 CPUs 0.341967 s/step 1.97897 days/ns 903.883 MB memory
Info: Benchmark time: 64 CPUs 0.339644 s/step 1.96553 days/ns 903.883 MB memory
Info: Benchmark time: 64 CPUs 0.284424 s/step 1.64597 days/ns 903.883 MB memory
Info: Benchmark time: 64 CPUs 0.28115 s/step 1.62702 days/ns 903.883 MB memory
Info: Benchmark time: 64 CPUs 0.279536 s/step 1.61769 days/ns 903.883 MB memory

TIMING: 25000 CPU: 7390.15, 0.296/step Wall: 7390.15, 0.296/step, 0.0205555 hours remaining, 915.746094 MB of memory in use.

64 cores over 18 nodes:

OMPI 1.6.5 mpirun

Info: Benchmark time: 64 CPUs 0.366327 s/step 2.11995 days/ns 939.527 MB memory
Info: Benchmark time: 64 CPUs 0.359805 s/step 2.0822 days/ns 939.527 MB memory
Info: Benchmark time: 64 CPUs 0.292342 s/step 1.69179 days/ns 939.527 MB memory
Info: Benchmark time: 64 CPUs 0.293499 s/step 1.69849 days/ns 939.527 MB memory
Info: Benchmark time: 64 CPUs 0.292355 s/step 1.69187 days/ns 939.527 MB memory

TIMING: 25000 CPU: 7754.17, 0.312071/step Wall: 7754.17, 0.312071/step, 0.0216716 hours remaining, 950.929688 MB of memory in use.

OMPI 1.7.3a1r29103 srun

Info: Benchmark time: 64 CPUs 0.347864 s/step 2.0131 days/ns 904.91 MB memory
Info: Benchmark time: 64 CPUs 0.346367 s/step 2.00444 days/ns 904.91 MB memory
Info: Benchmark time: 64 CPUs 0.29007 s/step 1.67865 days/ns 904.91 MB memory
Info: Benchmark time: 64 CPUs 0.279447 s/step 1.61717 days/ns 904.91 MB memory
Info: Benchmark time: 64 CPUs 0.280824 s/step 1.62514 days/ns 904.91 MB memory

TIMING: 25000 CPU: 7420.91, 0.296029/step Wall: 7420.91, 0.296029/step, 0.0205575 hours remaining, 916.312500 MB of memory in use.

Hope this is useful!

All the best,
Chris
- --
 Christopher Samuel Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel_at_[hidden] Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/ http://twitter.com/vlsci

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlIn6UoACgkQO2KABBYQAh9GWgCghcYKSj1i9rDDQospURAeusD5
E+EAn2beqUlYZWHxi1Dgj8ZEpiai4zH1
=k5Uz
-----END PGP SIGNATURE-----