On Jun 30, 2010, at 12:36 AM, Paul Hilscher wrote:
> >Open MPI currently has very limited cartesian support -- it actually doesn't remap anything.
> I see, OpenMPI doesn't remap anything; this explains probably why my runtime of my simulation varies sometimes between 30% for the same setup.
If you're relying on remapping and your environment would benefit from remapping, then yes, that could be a factor here.
> >Would you have any interest in writing a partitioning algorithm for your needs within the context of a plugin? I'd be happy to walk >you through the process; it's not too complicated (although we should probably move the discussion off to the Open MPI devel >mailing list).
> I guess after using for more than a decade Open Source Software, it's time to give something back :). ... so yes, I am willing to do that !!
> Because I am not yet experienced with OpenMPI internals, I would really appreciate your advice, if you could tell me where exactly I have to dig into.. I guess it should be around ompi_topo_create function, but how to write MPI_Cart_Create as a plugin, I will rely on you information. And do you know if MPICH, LAM etc. have an efficient implementation of MPI_Cart_Create ? so I can borrow some ideas from them....
Heh. I used to be the LAM/MPI guy; we stole some of LAM's topology stuff (including the unity mapping) for Open MPI. :-)
What you really want to do is create a new "topo" plugin. Topo plugins are automatically called behind the scenes for MPI_Cart_create and friends -- you don't need to muck about in ompi_topo_create() at all.
If you look in ompi/mca/topo, that's the top-level directory for the topo framework (meaning: all of our topo "glue" code is in there in the "base/" subdirectory, and all topo plugins are subdirs in there as well). There's only 1 plugin at the moment: it's called "unity" -- because it doesn't do any transforms.
Everything that the unity component does is contained in ompi/mca/topo/unity.
Before going any further, let's talk about some logistics.
1. Where to do this work? Possibilities are:
a. The v1.4 branch
b. The v1.5 branch
c. Our Subversion trunk
a. is not a good choice; it's in it's "stable" phase, meaning that new features are not supposed to be added, and the v1.5.0 release it potentially within the next week or two. b. is also not a good choice; we tend to develop on the SVN trunk and then migrate to the v1.5 branch as needed. c. is probably a reasonable choice, but I'd actually prefer d; here's why...
In MPI-2.2, some new topology functions were added. I set about adding these functions to Open MPI, which meant expanding the topo framework a bit (and the unity plugin). I got about halfway through this work before being called off to tend other fires. As with a lot of Open MPI development, I did this work in a Mercurial branch off our subversion trunk.
Your offer inspires me to finally get off my butt and finish the MPI-2.2 topology functions in that Mercurial branch. Collaborating in the Mercurial branch is probably a good place to do this work -- several of us OMPI developers use bitbucket.org for Mercurial hosting, which offers free accounts for collaborators:
The above URL is where my Mercurial branch is for this work. If you sign up for an account on bitbucket (it's free), I'll give you write access.
This work will eventually be committed back to the SVN trunk and then move in to the v1.5 branch, meaning that it would be included in future v1.5.x releases.
2. For any contribution of "non-trivial" size, we need a signed copy of the Open MPI 3rd party contribution agreement. I apologize about this, but it's what we need to do to maintain cleanliness of the intellectual property in Open MPI and guarantee that we can continue to distribute it under the BSD license. See:
Note that the form is pretty much the Apache contribution agreement; we essentially s/Apache/Open MPI/g.
If it'll take a little time to get this done, that's no problem -- we can collaborate in the bitbucket and just hold off bringing any of your work into SVN until the agreement is signed.
3. In the bitbucket repo, I created an "example" topo plugin in addition to the unity plugin. It'll never be used at run-time; it's sole purpose in life is to be an example on how to write new topo plugins.
I'll need to re-evaluate the state of that repo and see where it is, what is left to be done, etc. But I think it's at least in a compilable state and should at least nominally work (the new MPI-2.2 stuff may not work yet).
4. Some references for you:
- Read https://svn.open-mpi.org/trac/ompi/browser/trunk/HACKING for how to build OMPI from an SVN or Mercurial checkout.
- Read https://svn.open-mpi.org/trac/ompi/wiki/devel/CreateComponent for a description of how to add a new plugin ("component") to Open MPI
- Read https://svn.open-mpi.org/trac/ompi/wiki/devel/Autogen to understand what autogen.sh does in the build process.
For corporate legal information go to: