Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] RFC: Support Cross Memory Attach in sm btl
From: Christopher Yeoh (cyeoh_at_[hidden])
Date: 2012-01-12 04:34:06

Hi Brad,

WHAT: Adds Cross Memory Attach support to the sm btl

WHY: For faster intranode communication

WHERE: ompi/mca/btl/sm/

WHEN: Open MPI trunk

TIMEOUT: 13/2/2012

Cross Memory Attach (CMA) is a pair of new syscalls (process_vm_readv
and process_vm_writev) which allow for fast intranode
communication. It has added to the Linux 3.2 kernel. There is a man page
for the new system calls here:

Attached is a patch for the OMPI trunk tree which augments the sm btl
to use these calls.

- CMA is quite similar in many respects to KNEM and what I've done is to
  pretty much copy what KNEM does in many cases.

- Both KNEM and CMA can be compiled in at the same time, though if
  both are enabled at runtime, only KNEM runs. To enable CMA use
  --mca btl_sm_use_cma 1

- To enable CMA at compile time, add --with-cma=yes to the configure
  command line. Support for the syscalls is in the git glibc archive,
  but its not yet out in the distros so as an interim
  workaround I have added some arch/os specific wrappers which are used
  if the syscalls are not found at configure time. The syscalls numbers
  won't change as 3.2 is out.

- I'm far from sure that the way I have used CMA in OMPI is the best
  way to do it, so any feedback is very welcome.