>> > think we all agree that STCI and OMPI have different objectives and
>> requirements. OMPI is facing the need to launch and operate at extreme scales
>> by next summer, has received a lot of interest in having it report errors
>> into various systems, etc. We don't have all the answers as to what will be
>> necessary to meet these requirements, but indications so far are that tighter
>> integration, not deeper abstraction, between the various layers will be
>> needed. By that, I don't mean we will violate abstraction layers, but rather
>> that the various layers need to work more as a tightly tuned instrument, with
>> each layer operating based on a clear knowledge of how the other layers are
OMPI and STCI are two different things together, and I have vested interest
in both, and have no desire
to have either go south. You have a set of requirement at LANL which are
important, and we also have a set of requirement at ORNL, and as such we
need to compromise on these
in the code base. We have MPI level goals, which will be accomplished in
the OMPI code base, and
tools and other related goals that will be accomplished in other code bases.
We both have the need to function well at the high end, so have the same set
of goals there.
> For example, for modex-less operations, the MPI/BTLs have to know that the
RTE/OS will be providing certain information. This means that they don't have to
go out and discover it themselves every time. Yes, we will leave that as the
default behavior so that small and/or unmanaged clusters can operate, but we
have to also introduce logic that can detect when we are utilizing this
alternative capability and exploit it. While we are trying our best to avoid
introducing RTE-like calls into the code, the fact is that we may well have to
do so (we have already identified one btl that will definitely need to). It is
simply too early to make the decision to cut that off now - we don't know what
the long-term impacts of such a decision will be.
This is where discussions will need to go both ways. Your changes also can
impact us, and we need to agree
to those changes, just as much as you need to agree with the changes we are
proposing. This is not a code
base focused on a single institution's requirements, and we all do our best
(and I believe tend to
succeed) at helping meet all of our needs.
> Finally, although I don't do much on the MPI layer, I am concerned about
performance. I would tend to oppose any additional abstraction until we can
measure the performance impact. Thus, I would like to see the BTL move done on a
tmp branch (technology to branch up to the implementer - I don't care) so we can
verify that it isn't hurting us in some unforeseeable manner.
Agreed - at least for the last phase of what we are suggesting, but we can
talk about this. I am a bit
confused about how the location of the source code has anything to do with
how it performs at run-time.
At this stage we have said nothing about changing the way the btl works,
just cosmetic things. When it
comes to enabling the use of stci with ompi, then these issues will come up,
and need to be addressed
very carefully. To be honest, since we don't want to change the btl's
(aside from add some attributes)
I don't expect this to be an issue, UNLESS we end up needing to change some
data structures for abstraction
purposes. This is where we need to be very careful. If you look at what
has happened with the btl's
(actually first the PTL's) historically, I have been one of the ones pushing
hard for improved performance -
why would this change now ?
> So I guess my concerns really boil down to dealing with conflicting schedules
and requirements, how to support multiple possibly competing groups that want to
share one or more parts of our code base, and retaining an OMPI-first philosophy
when it comes to what changes get made. My proposed solution is:
This is the problem we face all the time, and on a regular basis we as a
community do our best to help
each other out. This is one of the reasons 1.3 is as late as it is, and
this is a good thing that will
continue as long as this is a community project.
> 1. shift our repository to a technical solution that supports broader code
> 2. have the non-OMPI groups access our code base via that technology. They can
"pull" changes at will, subject to the licensing agreement. It is true that they
may have to do some local editing if the change hits a spot where they have
local mods to support their system, but both Hg and GIT are very good at
handling this - much better than svn ever has been.
> 3. if there are minor mods required to make the BTL code area easier to share
via the above methods, then we should explore and implement them. Certainly,
renaming #define values would seem a no-brainer. I suspect there are other
similar things that could be done. Removing orte/opal dependencies is more
controversial and would need to thoroughly be examined.
> 4. OMPI decides what changes get made to its code base. We are polite about it
and talk to the other groups to try and minimize impact, but ultimately we do
what is best for OMPI, and send out notifications (perhaps a new mailing list
specifically for that purpose) when changes occur. Note that this would have
helped the Eclipse group enormously as otherwise they drown in the devel list
trying to spot the changes.
I don't see that anything else is being proposed. The emerging STCI
community and the OMPI community are
not two non-overlapping groups, and run-time support we want to bring into
OMPI is to support new
functionality. The main point is that this is not STCI vs. OMPI at all.
> My $0.0002 - hope it helps
> On Dec 4, 2008, at 6:00 PM, Richard Graham wrote:
> Let me start the e-mail conversation, and see how far we get.
> Goal: The goal several of us have is to be able to use the btlâÃÃ´s outside of
the MPI layer in Open MPI. The layer itself is generic, w/o specific knowledge
of Upper Level Protocols, so is well suited for this sort of use.
> Technical Approach: What we have suggested is to start the process with the
Open MPI code base, and make it independent of the mpi-layer (which it is now),
and the run-time layer.
> Before we get into any specific technical details,
> the first question I have is are people totally opposed to the notion of
making the btlâÃÃ´s independent of MPI and the run-time ?
> This does not mean that it canâÃÃ´t be used by it, but that there are well
defined abstraction layers, i.e., are people against the goal in the first place
> What are alternative suggestions to the technical approach ?
> One suggestion has been to branch and patch. To me this is a long-term
> What are peoples thoughts here ?
> devel mailing list
> devel mailing list