Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] RFC: Calling ibv_fork_init() in the openib BTL
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2013-12-06 10:15:02


To those who care about the openib BTL...

SHORT VERSION
-------------

Do you really want to call ibv_fork_init() in the openib BTL by default?

MORE DETAIL
-----------

Rolf V. pointed out to me yesterday that we're calling ibv_fork_init() in the openib BTL. He asked if we did the same in the usnic BTL. We don't, and here's why:

1. it adds a slight performance penalty for ibv_reg_mr/ibv_dereg_mr
2. the only thing ibv_fork_init() protects against is the child sending from memory that it thinks should already be registered:

-----
MPI_Init(...)
if (0 == fork()) {
    ibv_post_send(some_previously_pinned_buffer, ...);
    // ^^ this can't work because the buffer is *not* pinned in the child
    // (for lack of a longer explanation here)
}
-----

3. ibv_fork_init() is not intended to protect against a child invoking an MPI function (if they do that; they get what they deserve!).

Note that #2 can't happen, because MPI doesn't expose its protection domains, queue pairs, or registrations (or any of its verbs constructs) at all.

Hence, all ibv_fork_init() does is a) impose a performance penalty, and b) make memory physically unavailable in a child process, such that:

----
ibv_fork_init();
a = malloc(...);
a[0] = 17;
ibv_reg_mr(a, ...);
if (0 == fork()) {
    printf("this is a[0]: %d\n", a[0]);
    // ^^ This will segv
}
-----
But the registered memory may actually be useful in the child.
So I just thought I'd pass this along, and ask the openib-caring people of the world if you really still want to be calling ibv_fork_init() by default in the openib BTL.
-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/