Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Roadrunner blasts past the petaflop mark with Open MPI
From: Brad Benton (bradford.benton_at_[hidden])
Date: 2008-06-17 10:18:36


On Mon, Jun 16, 2008 at 9:46 PM, Brock Palen <brockp_at_[hidden]> wrote:

> Brad just curious.
> Did you tweak any other values for starting and running a job on such a
> large system? You say unmodified, but OpenMPI lets you tweak many values
> at runtime.
>

Ahh...sorry for the confusion. By "unmodified", I meant at the source code
level. The version of Open MPI used was from a straight checkout of the
trunk and did not have any special code modifications for Roadrunner. We
certainly did change some of the default run-time settings.

> I would be curious to expand what I know from what you discovered.
>

The main run-time parameters used are as follows:

- For scalable process launch
  routed = binomial

- To reduce the overhead for IB connections:
  btl_openib_receive_queues =
P,128,64,32,32,32:S,2048,1024,128,32:S,12288,1024,128,32:S,65536,1024,128,32

- Bind processes to computational cores:
  opal_paffinity_alone = 1

- Leave buffers pinned and registered
  mpi_leave_pinned = 1

--brad

>
> Brock Palen
> www.umich.edu/~brockp <http://www.umich.edu/%7Ebrockp>
> Center for Advanced Computing
> brockp_at_[hidden]
> (734)936-1985
>
>
>
> On Jun 16, 2008, at 10:12 PM, Brad Benton wrote:
>
> Greetings Open MPI users; we thought you'd be interested in the
> following announcement...
>
> A new supercomputer, powered by Open MPI, has broken the petaflop
> barrier to become the world's fastest supercomputer. The
> "Roadrunner" system was jointly developed by Los Alamos National
> Laboratories and IBM. Roadrunner's design uses a cluster of AMD
> dual-core processors coupled with computational accelerators based
> on the IBM Cell Broadband Engine. The cluster consists of 3,060
> nodes, each of which has 2 dual-core AMD processors and associated
> Cell accelerators. The AMD nodes are connected with 4x DDR
> InfiniBand links.
>
> Open MPI was used as the communications library for the 12,240
> processes comprising the Linpack run which broke the Petaflop
> barrier at 1.026 Petaflop/s. The version of Open MPI used in the
> run-for-record was a pre-release version of the upcoming 1.3
> release. Enhancements in this release include modifications for
> efficient, scalable process launch. As such, Open MPI was run
> unmodified from a snapshot of the pre-1.3 source base (meaning:
> there are no Roadrunner-specific enhancements that are unportable to
> other environments -- all Open MPI users benefit from the
> scalability and performance improvements contributed by the
> Roadrunner project).
>
> --Brad Benton
> Open MPI/Roadrunner Team
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>