Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] "Open MPI"-based MPI library used by K computer
From: Kawashima (t-kawashima_at_[hidden])
Date: 2011-06-29 03:57:28


Hi Jeff, Ralph, and all,

Thank you for your reply.
RIKEN and Fujitsu will work toword 10Pflops with Open MPI continuously.

Here we can explain some parts of our MPI:

As page 13 of Koh Hotta's presentation shows, we extended OMPI
communication layers.

> http://www.fujitsu.com/downloads/TC/sc10/programming-on-k-computer.pdf
# Sorry, this figure is somewhat broken. Arrows point incorrect layers.

First, we created a new BTL component, 'tofu BTL'. It's not so special
one but dedicated to our Tofu interconnect. But its latency was not
enough for us.

So we created a new framework, 'LLP', and its component, 'tofu LLP'.
It bypasses request object creation in PML and BML/BTL, and sends
a message immediately if possible.

Also, we modified tuned COLL to implement interconnect-and-topology-
specific bcast/allgather/alltoall/allreduce algorithm. These algorithm
implementations also bypass PML/BML/BTL to eliminate protocol and software
overhead.

To achieve above, we created 'tofu COMMON', like sm (ompi/mca/common/sm/).

Is there interesting one?

Though our BTL and COLL are quite interconnect-specific, LLP may be
contributed in the future.

Regards,

Takahiro Kawashima,
MPI development team,
Fujitsu

> I echo what Ralph said -- congratulations!
>
> Let us know when you'll be ready to contribute back what you can.
>
> Thanks!
>
>
> On Jun 27, 2011, at 9:58 PM, Takahiro Kawashima wrote:
>
> > Dear Open MPI community,
> >
> > I'm a member of MPI library development team in Fujitsu. Shinji
> > Sumimoto, whose name appears in Jeff's blog, is one of our bosses.
> >
> > As Rayson and Jeff noted, K computer, world's most powerful HPC system
> > developed by RIKEN and Fujitsu, utilizes Open MPI as a base of its MPI
> > library. We, Fujitsu, are pleased to announce that, and also have special
> > thanks to Open MPI community.
> > We are sorry to be late announce!
> >
> > Our MPI library is based on Open MPI 1.4 series, and has a new point-
> > to-point component (BTL) and new topology-aware collective communication
> > algorithms (COLL). Also, it is adapted to our runtime environment (ESS,
> > PLM, GRPCOMM etc).
> >
> > K computer connects 68,544 nodes by our custom interconnect.
> > Its runtime environment is our proprietary one. So we don't use orted.
> > We cannot tell start-up time yet because of disclosure restriction, sorry.
> >
> > We are surprised by the extensibility of Open MPI, and have proved that
> > Open MPI is scalable to 68,000 processes level! We feel pleasure to
> > utilize such a great open-source software.
> >
> > We cannot tell detail of our technology yet because of our contract
> > with RIKEN AICS, however, we will plan to feedback of our improvements
> > and bug fixes. We can contribute some bug fixes soon, however, for
> > contribution of our improvements will be next year with Open MPI
> > agreement.
> >
> > Best regards,
> >
> > MPI development team,
> > Fujitsu
> >
> >
> >> I got more information:
> >>
> >> http://blogs.cisco.com/performance/open-mpi-powers-8-petaflops/
> >>
> >> Short version: yes, Open MPI is used on K and was used to power the 8PF runs.
> >>
> >> w00t!
> >>
> >>
> >>
> >> On Jun 24, 2011, at 7:16 PM, Jeff Squyres wrote:
> >>
> >>> w00t!
> >>>
> >>> OMPI powers 8 petaflops!
> >>> (at least I'm guessing that -- does anyone know if that's true?)
> >>>
> >>>
> >>> On Jun 24, 2011, at 7:03 PM, Rayson Ho wrote:
> >>>
> >>>> Interesting... page 11:
> >>>>
> >>>> http://www.fujitsu.com/downloads/TC/sc10/programming-on-k-computer.pdf
> >>>>
> >>>> Open MPI based:
> >>>>
> >>>> * Open Standard, Open Source, Multi-Platform including PC Cluster.
> >>>> * Adding extension to Open MPI for "Tofu" interconnect
> >>>>
> >>>> Rayson
> > _______________________________________________
> > devel mailing list
> > devel_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/