WHAT: make mpi_leave_pinned=1 by default when a BTL is used that would
benefit from it (when possible; 0 when not, obviously)
WHY: Several reasons:
- we continually get beat up because of "lower performance" on
benchmarks by default (I get beat up, at least ;-) )
- ptmalloc is no longer compiled in user apps by default, but mallopt
may be available
- ptmalloc has been linked in on many platforms by default for a long
- our ptmalloc settings were such that memory was rarely returned to
the OS -- quite similar to mallopt
- very few people have complained about the above policy
- therefore, it may be ok to mallopt by default if there is a device
in the run that would benefit from it
WHERE: openib BTL, MPI runtime directory
WHEN: before v1.3 ships
TIMEOUT: Fri, July 11, 2008
I'm assuming that this topic will generate a fair amount of
I'm basically getting tired of people complaining that OMPI has lower
default benchmark performance on OpenFabrics networks. I don't mind
explaining the mpi_leave_pinned flag; what I do mind is that customers
and users who refuse to use it (which is at least sort of
understandable). I also mind that other MPI implementations
(sometimes knowingly) compare Open MPI without leave_pinned to their
implementations with leave_pinned. Explaining it after the fact is
never quite as compelling when there is a big poster on a show floor
showing MPI XYZ with great ping pong performance and OMPI with lousy
ping pong performance.
- OMPI is the only MPI that doesn't do the "leave pinned" trick by
default on OpenFabrics networks
- I know that pingpong benchmarks are meaningless. Customers and
users don't care. We cannot move this mountain.
- I know that leave_pinned is frequently meaningless to real apps
(although Torsten likes to argue otherwise -- and he's got at least
some real-world data points that show otherwise :-) ).
- I know that it's only OpenFabrics networks that require this setting
and that many people think OpenFabrics is broken because of this.
Let's leave such religious arguments at the door; I'm not happy we
have to do it either, but that's not the issue here.
So my proposal is to enable mpi_leave_pinned by default:
- if there's a BTL in the app that would benefit (i.e., openib). This
would likely entail adding some clever callback from the openib BTL
init, or somesuch (I have not thought this through yet).
- mallopt or ptmalloc is available