Hi Brad,
WHAT: Adds Cross Memory Attach support to the sm btl
WHY: For faster intranode communication
WHERE: ompi/mca/btl/sm/
WHEN: Open MPI trunk
TIMEOUT: 13/2/2012
Cross Memory Attach (CMA) is a pair of new syscalls (process_vm_readv
and process_vm_writev) which allow for fast intranode
communication. It has added to the Linux 3.2 kernel. There is a man page
for the new system calls here:
http://ozlabs.org/~cyeoh/cma/process_vm_readv.txt
Attached is a patch for the OMPI trunk tree which augments the sm btl
to use these calls.
- CMA is quite similar in many respects to KNEM and what I've done is to
pretty much copy what KNEM does in many cases.
- Both KNEM and CMA can be compiled in at the same time, though if
both are enabled at runtime, only KNEM runs. To enable CMA use
--mca btl_sm_use_cma 1
- To enable CMA at compile time, add --with-cma=yes to the configure
command line. Support for the syscalls is in the git glibc archive,
but its not yet out in the distros so as an interim
workaround I have added some arch/os specific wrappers which are used
if the syscalls are not found at configure time. The syscalls numbers
won't change as 3.2 is out.
- I'm far from sure that the way I have used CMA in OMPI is the best
way to do it, so any feedback is very welcome.
Regards,
Chris
--
cyeoh_at_[hidden]
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}