Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Regarding Connection establishment in OpenMPI (Jeff Squyres)
From: Bhargava Ramu Kavati (ramu.kavati_at_[hidden])
Date: 2011-09-06 13:24:26


Hi Jeff,
As per our last discussion, MPI_INIT(..) uses TCP socket to exchange its
service-id/lid with other MPI processes. I assume this applies irrespective
of underlying library used to establish connection i.e libibcm or
librdmacm. Please correct me if I am wrong.

Message: 1
> Date: Wed, 24 Aug 2011 12:06:30 -0400
> From: Jeff Squyres <jsquyres_at_[hidden]>
> Subject: Re: [OMPI devel] Regarding Connection establishment in
> OpenMPI (Jeff Squyres)
> To: Open MPI Developers <devel_at_[hidden]>
> Message-ID: <C730E008-DEE5-4444-B2F2-5AA4FDD9A2C1_at_[hidden]>
> Content-Type: text/plain; charset=us-ascii
>
> At the moment, our only "OOB" (out of band) module uses TCP sockets. This
> can use traditional ethernet or an emulated IP layer, such as IPoIB.
>
>
> On Aug 24, 2011, at 11:58 AM, Bhargava Ramu Kavati wrote:
>
> > Hi Jeff,
> > Thank you for your prompt response. I have a query related to MPI_INIT
> here. What is the underlying transport mechanism does OpenMPI uses to
> exchange service-id/lid via MPI_INIT, is it TCP/IP socket ?
> >
> > Thanks & Regards,
> > Ramu
> >
> > Message: 2
> > Date: Mon, 22 Aug 2011 17:33:19 -0400
> > From: Jeff Squyres <jsquyres_at_[hidden]>
> > Subject: Re: [OMPI devel] Regarding Connection establishment in
> > OpenMPI
> > To: Open MPI Developers <devel_at_[hidden]>
> > Message-ID: <2399C470-7F91-49D4-A463-A8994691ABA7_at_[hidden]>
> > Content-Type: text/plain; charset=us-ascii
> >
> > On Aug 22, 2011, at 9:35 AM, Bhargava Ramu Kavati wrote:
> >
> > > I am trying to explore the details of connection establishment in
> OpenMPI using libibcm/librdmacm.
> >
> > Note that the IB community has given up on ibcm. Our support of it is
> incomplete; I wouldn't look at it as an example.
> >
> > > In the code, I could not find how OpenMPI app is getting service-id/lid
> of remote node to which it wants to connect.
> >
> > In the normal case, we pass that information during MPI_INIT. It's a
> global gather / broadcast operation that we refer to as the "modex" (module
> exchange). I.e., each openib BTL module instance publishes its address
> information in the modex and sends it. Near the end of MPI_INIT, each MPI
> process receives the modex broadcast and caches it.
> >
> > During connection establishment, an MPI process will look in its modex
> cache to find the connection information for the peer process that it wants
> to connect to.
> >
> > > Also, I did not see any query in the code related to service_record_get
> from SA. Can you please desribe what is happening OR Am I missing something
> here ?
> >
> > IIRC, we don't currently use the SA because of its serialization and
> other resource bottlenecks (this is a hand-waving answer; I don't remember
> the exact reasons for not using the SA, but there were many discussions
> between the MPI and OpenFabrics communities a long time ago. The SA issues
> were not resolved to the MPI community's liking, IIRC, but this was a long
> time ago, and I don't even work for an IB vendor any more, so I might not be
> remembering this correctly...).
> >
> > --
> > Jeff Squyres
> > jsquyres_at_[hidden]
> > For corporate legal information go to:
> > http://www.cisco.com/web/about/doing_business/legal/cri/
> >
> >
> > _______________________________________________
> > devel mailing list
> > devel_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
>
>
> ------------------------------
>
>