Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Shared memory optimizations in OMPI
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2011-11-22 09:04:02


On Nov 22, 2011, at 1:09 AM, Shamik Ganguly wrote:

> I want to trace when the MPI library prevents an MPI_Send from going to the socket and makes it access shared memory because the target node is on the same chip (CMP). I want to use PIN to trace this. Can you please give me some pointers about which functions are taking this decision so that I can instrument the appropriate library calls in PIN?

What's PIN?

The decision is made in the ob1 PML plugin. Way back during MPI_INIT, each MPI process creates lists of BTLs to use to contact each MPI process peer.

When a process is on the same *node* (e.g., a single server) -- not just the same processor socket/chip -- the shared memory BTL is given preference to all other BTLs by use of a priority mechanism. Hence, the "sm" BTL is put at the front of the BML lists (BML = BTL multiplexing layer -- it's essentially just list management for BTLs).

Later, when MPI_SEND comes through, it uses the already-setup BML lists to determine which BTL(s) to use to send a message.

That's the 50,000 foot view.

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/