Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] importing to MPI data already in memory from another process
From: Dave Goodell (dgoodell) (dgoodell_at_[hidden])
Date: 2014-06-27 12:11:45


On Jun 27, 2014, at 8:53 AM, Brock Palen <brockp_at_[hidden]> wrote:

> Is there a way to import/map memory from a process (data acquisition) such that an MPI program could 'take' or see that memory?
>
> We have a need to do data acquisition at the rate of .7TB/s and need todo some shuffles/computation on these data, some of the nodes are directly connected to the device, and some will do processing.
>
> Here is the proposed flow:
>
> * Data collector nodes runs process collecting data from device
> * Those nodes somehow pass the data to an MPI job running on these nodes and a number of other nodes (cpu need for filterting is greater than what the 16 data nodes can provide).

For a non-MPI solution for intranode data transfer in this case, take a look at vmsplice(2):

http://man7.org/linux/man-pages/man2/vmsplice.2.html

Pay particular attention to the SPLICE_F_GIFT flag, which will allow you to simply give memory pages away to the MPI process, avoiding unnecessary data copies. You would just need a pipe shared between the data collector process and the MPI process (and to be a bit careful with your memory allocation/management, since any page you gift away should probably come from mmap(2) directly).

Otherwise, as George mentioned, I would investigate converting your current data collector processes to also be MPI processes so that they can simply communicate the data to the rest of the cluster.

-Dave