On Mon, Jun 16, 2008 at 9:46 PM, Brock Palen <
brockp@umich.edu> wrote:
Brad just curious.
Did you tweak any other values for starting and running a job on such a large system? You say unmodified, but OpenMPI lets you tweak many values at runtime.
Ahh...sorry for the confusion. By "unmodified", I meant at the source code level. The version of Open MPI used was from a straight checkout of the trunk and did not have any special code modifications for Roadrunner. We certainly did change some of the default run-time settings.
I would be curious to expand what I know from what you discovered.
The main run-time parameters used are as follows:
- For scalable process launch
routed = binomial
- To reduce the overhead for IB connections:
btl_openib_receive_queues = P,128,64,32,32,32:S,2048,1024,128,32:S,12288,1024,128,32:S,65536,1024,128,32
- Bind processes to computational cores:
opal_paffinity_alone = 1
- Leave buffers pinned and registered
mpi_leave_pinned = 1
--brad
Brock Palen
Center for Advanced Computing
(734)936-1985
On Jun 16, 2008, at 10:12 PM, Brad Benton wrote:
Greetings Open MPI users; we thought you'd be interested in the
following announcement...
A new supercomputer, powered by Open MPI, has broken the petaflop
barrier to become the world's fastest supercomputer. The
"Roadrunner" system was jointly developed by Los Alamos National
Laboratories and IBM. Roadrunner's design uses a cluster of AMD
dual-core processors coupled with computational accelerators based
on the IBM Cell Broadband Engine. The cluster consists of 3,060
nodes, each of which has 2 dual-core AMD processors and associated
Cell accelerators. The AMD nodes are connected with 4x DDR
InfiniBand links.
Open MPI was used as the communications library for the 12,240
processes comprising the Linpack run which broke the Petaflop
barrier at 1.026 Petaflop/s. The version of Open MPI used in the
run-for-record was a pre-release version of the upcoming 1.3
release. Enhancements in this release include modifications for
efficient, scalable process launch. As such, Open MPI was run
unmodified from a snapshot of the pre-1.3 source base (meaning:
there are no Roadrunner-specific enhancements that are unportable to
other environments -- all Open MPI users benefit from the
scalability and performance improvements contributed by the
Roadrunner project).
--Brad Benton
Open MPI/Roadrunner Team
_______________________________________________
users mailing list
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users