Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] RFC: Change default behavior of calling ibv_fork_init
From: Rolf vandeVaart (rvandevaart_at_[hidden])
Date: 2014-07-31 10:01:08


WHAT: Change default behavior in openib to not call ibv_fork_init() even if available.
WHY: There are some strange interactions with ummunotify that cause errors. In addition, see the additional points below.
WHEN: After next weekly meeting, August 5, 2014
DETAILS: This change will just be a couple of lines. Current default behavior is to call ibv_fork_init() if support exists. New default behavior is to call it only if asked for.
Essentially, default setting of btl_openib_want_fork_support will change from -1 (use it if available) to 0 (do not use unless asked for)

Here are details from an earlier post last year. http://www.open-mpi.org/community/lists/devel/2013/12/13395.php
Subject: [OMPI devel] RFC: Calling ibv_fork_init() in the openib BTL
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2013-12-06 10:15:02
To those who care about the openib BTL...
SHORT VERSION
-------------
Do you really want to call ibv_fork_init() in the openib BTL by default?
MORE DETAIL
-----------
Rolf V. pointed out to me yesterday that we're calling ibv_fork_init() in the openib BTL. He asked if we did the same in the usnic BTL. We don't, and here's why:
1. it adds a slight performance penalty for ibv_reg_mr/ibv_dereg_mr
2. the only thing ibv_fork_init() protects against is the child sending from memory that it thinks should already be registered:
-----
MPI_Init(...)
if (0 == fork()) {
    ibv_post_send(some_previously_pinned_buffer, ...);
    // ^^ this can't work because the buffer is *not* pinned in the child
    // (for lack of a longer explanation here)
}
-----
3. ibv_fork_init() is not intended to protect against a child invoking an MPI function (if they do that; they get what they deserve!).
Note that #2 can't happen, because MPI doesn't expose its protection domains, queue pairs, or registrations (or any of its verbs constructs) at all.
Hence, all ibv_fork_init() does is a) impose a performance penalty, and b) make memory physically unavailable in a child process, such that:

----
ibv_fork_init();
a = malloc(...);
a[0] = 17;
ibv_reg_mr(a, ...);
if (0 == fork()) {
    printf("this is a[0]: %d\n", a[0]);
    // ^^ This will segv
}
-----
But the registered memory may actually be useful in the child.
So I just thought I'd pass this along, and ask the openib-caring people of the world if you really still want to be calling ibv_fork_init() by default in the openib BTL.
--
Jeff Squyres
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------