Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Gleb Natapov (glebn_at_[hidden])
Date: 2006-11-02 15:39:59


On Thu, Nov 02, 2006 at 11:57:16AM -0800, Brian Budge wrote:
> Thanks for the help guys.
>
> In my case the memory will be allocated and pinned by my other device
> driver. Is it safe to simply use that memory? My pages won't be unpinned
> as a result?
>
If your driver plays nicely with openib driver everything should be OK.
If by pinned you mean mlcok() then your are save since openib doesn't
use mlock().

> As far as registration, I am sure that OpenMPI will do a better job of that
> than I could, so I won't even attempt to futz with that.
>
> Thanks,
> Brian
>
> On 11/2/06, Brian W Barrett <bbarrett_at_[hidden]> wrote:
> >
> >Locking a page with mlock() is not all that is required for RDMA
> >using InfiniBand (or Myrinet, for that matter). You have to call
> >that device's registration function first. In Open MPI, that can be
> >done implicitly with the mpi_leave_pinned option, which will pin
> >memory as needed and then leave it pinned for the life of the
> >buffer. Or it can be done ahead of time by calling MPI_ALLOC_MEM.
> >
> >Because the amount of memory a NIC can have pinned at any time may
> >not directly match the total amount of memory that can be mlock()ed
> >at any given time, it's also not a safe assumption that a buffer
> >allocated with MPI_ALLOC_MEM or used with an RDMA transfer from MPI
> >is going to be mlock()ed as a side effect of NIC registration. Open
> >MPI internally might unregister that memory with the NIC in order to
> >register a different memory segment for another memory transfer.
> >
> >Brian
> >
> >
> >On Nov 2, 2006, at 12:22 PM, Brian Budge wrote:
> >
> >> Thanks for the pointer, it was a very interesting read.
> >>
> >> It seems that by default OpenMPI uses the nifty pipelining trick
> >> with pinning pages while transfer is happening. Also the pinning
> >> can be (somewhat) perminant and the state is cached so that next
> >> usage requires no registration. I guess it is possible to use pre-
> >> pinned memory, but do I need to do anything special to do so? I
> >> will already have some buffers pinned to allow DMAs to devices
> >> across PCI-Express, so it makes sense to use one pinned buffer so
> >> that I can avoid memcpys.
> >>
> >> Are there any HOWTO tutorials or anything? I've searched around,
> >> but it's possible I just used the wrong search terms.
> >>
> >> Thanks,
> >> Brian
> >>
> >>
> >>
> >> On 11/2/06, Jeff Squyres <jsquyres_at_[hidden]> wrote: This paper
> >> explains it pretty well:
> >>
> >> http://www.open-mpi.org/papers/euro-pvmmpi-2006-hpc-protocols/
> >>
> >>
> >>
> >> On Nov 2, 2006, at 1:37 PM, Brian Budge wrote:
> >>
> >> > Hi all -
> >> >
> >> > I'm wondering how DMA is handled in OpenMPI when using the
> >> > infiniband protocol. In particular, will I get a speed gain if my
> >> > read/write buffers are already pinned via mlock?
> >> >
> >> > Thanks,
> >> > Brian
> >> > _______________________________________________
> >> > users mailing list
> >> > users_at_[hidden]
> >> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >>
> >> --
> >> Jeff Squyres
> >> Server Virtualization Business Unit
> >> Cisco Systems
> >>
> >> _______________________________________________
> >> users mailing list
> >> users_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >> _______________________________________________
> >> users mailing list
> >> users_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> >_______________________________________________
> >users mailing list
> >users_at_[hidden]
> >http://www.open-mpi.org/mailman/listinfo.cgi/users
> >

> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

--
			Gleb.