Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] SM BTL NUMA awareness patches
From: Gleb Natapov (glebn_at_[hidden])
Date: 2008-05-28 09:17:27


Attached two patches implement NUMA awareness in SM BTL. The first one
adds two new functions to maffinity framework required by the second
patch. The functions are:

 opal_maffinity_base_node_name_to_id() - gets a string that represents a
                                         memory node name and translates
                                         it to memory node id.
 opal_maffinity_base_bind() - binds an address range to specific
                                         memory node.

The bind() function cannot be implemented by all maffinity components.
(There is no way first_use maffinity component can implement such
functionality). In this case this function can be set to NULL.

The second one adds NUMA awareness support to SM BTL and SM MPOOL. Each
process determines what CPU it is running on and exchange this info with
other local processes. Each process creates separate MPOOL for every
memory node available and use them to allocate memory on specific memory
nodes if needed. For instance circular buffer memory is always allocated
on memory node local to receiver process.

To use this on a Linux machine carto file with HW topology description should
be provided. Processes should be bound to specific CPU (by specifying
rank file for instance) and session directory should be created on tmpfs
file system (otherwise Linux ignores memory binding commands) by
setting orte_tmpdir_base parameter to point to tmpfs mount point.

Questions and suggestion are alway welcome.