On Mon, Jun 16, 2008 at 9:46 PM, Brock Palen <brockp_at_[hidden]> wrote:
> Brad just curious.
> Did you tweak any other values for starting and running a job on such a
> large system? You say unmodified, but OpenMPI lets you tweak many values
> at runtime.
Ahh...sorry for the confusion. By "unmodified", I meant at the source code
level. The version of Open MPI used was from a straight checkout of the
trunk and did not have any special code modifications for Roadrunner. We
certainly did change some of the default run-time settings.
> I would be curious to expand what I know from what you discovered.
The main run-time parameters used are as follows:
- For scalable process launch
routed = binomial
- To reduce the overhead for IB connections:
- Bind processes to computational cores:
opal_paffinity_alone = 1
- Leave buffers pinned and registered
mpi_leave_pinned = 1
> Brock Palen
> www.umich.edu/~brockp <http://www.umich.edu/%7Ebrockp>
> Center for Advanced Computing
> On Jun 16, 2008, at 10:12 PM, Brad Benton wrote:
> Greetings Open MPI users; we thought you'd be interested in the
> following announcement...
> A new supercomputer, powered by Open MPI, has broken the petaflop
> barrier to become the world's fastest supercomputer. The
> "Roadrunner" system was jointly developed by Los Alamos National
> Laboratories and IBM. Roadrunner's design uses a cluster of AMD
> dual-core processors coupled with computational accelerators based
> on the IBM Cell Broadband Engine. The cluster consists of 3,060
> nodes, each of which has 2 dual-core AMD processors and associated
> Cell accelerators. The AMD nodes are connected with 4x DDR
> InfiniBand links.
> Open MPI was used as the communications library for the 12,240
> processes comprising the Linpack run which broke the Petaflop
> barrier at 1.026 Petaflop/s. The version of Open MPI used in the
> run-for-record was a pre-release version of the upcoming 1.3
> release. Enhancements in this release include modifications for
> efficient, scalable process launch. As such, Open MPI was run
> unmodified from a snapshot of the pre-1.3 source base (meaning:
> there are no Roadrunner-specific enhancements that are unportable to
> other environments -- all Open MPI users benefit from the
> scalability and performance improvements contributed by the
> Roadrunner project).
> --Brad Benton
> Open MPI/Roadrunner Team
> users mailing list
> users mailing list