Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] [visit-developers] /usr/bin/ld: cannot find -lrdmacm on 9184
From: tom fogal (tfogal_at_[hidden])
Date: 2009-12-17 15:14:40


Simon Su <newsgroup4ssu_at_[hidden]> writes:
> Hi Tom,
>
> "hello world" MPI program also won't compile if
> librdmacm-devel-1.0.8-5.el5 is not installed. I have asked the person
> who maintain the openmpi package on how they were compiled. My guess
> is librdmacm-devel-1.0.8-5.el5 may need to be added as dependency
> package for openmpi010208-gcc-devel-1.2.8-8.cses.5.PU_IAS.5 package
> (where I got my openmpi installation) to solve the problem and to
> verify that we have the correct openmpi compilation.

Yes, I agree, from this message && your last (to OMPI: Simon mentioned
that the issue goes away if he installs librdmacm-devel), it sounds
like librdmacm-devel is a "build-dep", but the openmpi-devel package
needs it as a "dep".

Anyway, I'll leave it up to you to forward the error/conclusion to
whomever your upstream is. Thanks for digging into this,

-tom

> On Wed, Dec 16, 2009 at 5:45 PM, tom fogal <tfogal_at_[hidden]> wrote:
>
> > Hi,
> >
> > Jeff sent this reply to our inquiry yesterday.
> >
> > Simon -- can you give it a read? In particular, validating you've got
> > the right mpic++ sounds like a good idea. We're also curious if a
> > simple "hello world" MPI program can link using gcc + the flags from
> > mpic++ -show.
> >
> > -tom
> >
> > ------- Forwarded Message
> >
> > From: Jeff Squyres <jsquyres_at_[hidden]>
> > In-Reply-To: <auto-000021432141_at_[hidden]>
> > Date: Wed, 16 Dec 2009 17:20:35 -0500
> > To: "Open MPI Users" <users_at_[hidden]>
> > Cc: VisIt Developers <visit-developers_at_[hidden]>
> > Subject: Re: [OMPI users] [visit-developers] /usr/bin/ld: cannot
> > find-lrdmacm on 9184
> >
> > It depends on how Open MPI was built.
> >
> > If Open MPI was built without plugins (i.e., all the plugins are slurped up
> > into libmpi and friends), then yes, applications need to link against
> > librdmacm to use the RDMA CM mode of OpenFabrics transport.
> >
> > If Open MPI was built with plugins (which is the default), then apps don't
> > need to link against librdmacm because the only use of rdmacm is in an Open
> > MPI plugin, and that plugin was linked against librdmacm.
> >
> > Make sense?
> >
> > That being said, the output from mpic++ --showme should give you something
> > that is directly compile-/link-able. So it is odd if mpic++ is showing you
> > something that can't (or shouldn't?) be done. Did a -L argument get lost
> > somewhere, perchance?
> >
> > Does linking MPI applications with mpic++ work properly, or does it result
> > in the same error? If it results in the same error, then perhaps something
> > has changed since Open MPI was installed...?
> >
> > All this being said, two other random points:
> >
> > 1. Ensure that you're using the "right" mpic++. I.e., make sure it matches
> > the version/installation of Open MPI that you're trying to use.
> >
> > 2. If you don't link with the librdmacm, you're probably not losing any
> > important functionality unless you have an iWarp-based cluster (that's the
> > only transport that *needs* librdmacm). IB-based networks can use
> > librdmacm, but don't *need* it (it's only used for making initial
> > connections, so using librdmacm or not has no implications on overall MPI
> > performance). It's still odd that mpic++ wants it and it can't be found,
> > though...
> >
> > Does that helps?
> >
> >
> > On Dec 15, 2009, at 11:11 PM, tom fogal wrote:
> >
> > > Simon Su <newsgroup4ssu_at_[hidden]> writes:
> > > > Hi Tom,
> > > >
> > > > I am using the standard openmpi package that run on all the cluster
> > > > machines here at Princeton. So, maybe I shouldn't touch openmpi. But,
> > > > removing -lrdmacm from the MPI_LIBS line in the machinename.conf file
> > > > worked. Any implication from doing this?
> > >
> > > The only thing it could possibly do is disable RDMA for you. However,
> > > since removing it did not produce any undefined symbol errors, my guess
> > > is that your OpenMPI isn't using RDMA anyway.
> > >
> > > There might be an OpenMPI bug here, though. I've cc'd the OpenMPI
> > > community to see if they have any input. As a summary for them: Simon
> > > is trying to build our MPI-enabled application. A script which tries
> > > to automate this adds the output of "mpic++ -show". His build then
> > > failed because it attempted to link against librdmacm, which does not
> > > exist in his normal search paths (or maybe at all). Is it possible
> > > that `mpic++ -show' includes/adds "-lrdmacm" even when OpenMPI is not
> > > itself using the library?
> > >
> > > Thanks,
> > >
> > > -tom
> > >
> > > > On Tue, Dec 15, 2009 at 8:46 PM, tom fogal <tfogal_at_[hidden]>
> > wrote:
> > > >
> > > > > Simon Su <newsgroup4ssu_at_[hidden]> writes:
> > > > > > I am getting this error message while building 9184.
> > > > > [snip]
> > > > > > -lz -lm -ldl -lpthread -L/usr/local/openmpi/1.3.3/gcc/x86_64/lib64
> > > > > > -lmpi_cxx -lmpi -lopen-rte -lopen-pal -lrdmacm -libverbs -lnuma
> > -ldl
> > > > > -lnsl
> > > > > > -lutil -lm -lcognomen \
> > > > > > -L/usr/local/openmpi/1.3.3/gcc/x86_64/lib64 -lmpi_cxx -lmpi
> > > > > > -lopen-rte -lopen-pal -lrdmacm -libverbs -lnuma -ldl -lnsl -lutil
> > -lm
> > > > > > -lcognomen
> > > > > > /usr/bin/ld: cannot find -lrdmacm
> > > > > > collect2: ld returned 1 exit status
> > > > >
> > > > > Your OpenMPI install (incorrectly?) thinks it has librdmacm
> > available,
> > > > > but the library isn't in your search path.
> > > > >
> > > > > It apparently defaults to enabled in 1.3.3. That seems rather
> > > > > silly, since I imagine the library requires RDMA hardware, which
> > > > > is of course not ubiquitous. Anyway, try configuring OpenMPI with
> > > > > --disable-openib-rdmacm and then rerunning build_visit.
> > > > >
> > > > > Of course, if you actually have an RDMA cluster, you'll want to delve
> > > > > deeper.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > -tom
> > > _______________________________________________
> > > users mailing list
> > > users_at_[hidden]
> > > http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >
> >
> >
> > - --
> > Jeff Squyres
> > jsquyres_at_[hidden]
> >
> >
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> > ------- End of Forwarded Message
> >
> >
>
> --001636e0b6b9dccd4d047af22bb9
> Content-Type: text/html; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
>
> Hi Tom,<br><br>&quot;hello world&quot; MPI program also won&#39;t compile i=
> f librdmacm-devel-1.0.8-5.el5 is not installed. I have asked the person who=
> maintain the openmpi package on how they were compiled. My guess is librdm=
> acm-devel-1.0.8-5.el5 may need to be added as dependency package for openmp=
> i010208-gcc-devel-1.2.8-8.cses.5.PU_IAS.5 package (where I got my openmpi i=
> nstallation) to solve the problem and to verify that we have the correct op=
> enmpi compilation.<br>
> <br>Thanks for your help<br>Simon<br><br><div class=3D"gmail_quote">On Wed,=
> Dec 16, 2009 at 5:45 PM, tom fogal <span dir=3D"ltr">&lt;<a href=3D"mailto=
> :tfogal_at_[hidden]">tfogal_at_[hidden]</a>&gt;</span> wrote:<br><blockqu=
> ote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, 204, 204=
> ); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> Hi,<br>
> <br>
> Jeff sent this reply to our inquiry yesterday.<br>
> <br>
> Simon -- can you give it a read? =A0In particular, validating you&#39;ve go=
> t<br>
> the right mpic++ sounds like a good idea. =A0We&#39;re also curious if a<br=
> >
> simple &quot;hello world&quot; MPI program can link using gcc + the flags f=
> rom<br>
> mpic++ -show.<br>
> <br>
> -tom<br>
> <br>
> ------- Forwarded Message<br>
> <br>
> From: Jeff Squyres &lt;<a href=3D"mailto:jsquyres_at_[hidden]">jsquyres_at_cisco=
> .com</a>&gt;<br>
> In-Reply-To: &lt;<a href=3D"mailto:auto-000021432141_at_[hidden]">auto-000=
> 021432141_at_[hidden]</a>&gt;<br>
> Date: Wed, 16 Dec 2009 17:20:35 -0500<br>
> To: &quot;Open MPI Users&quot; &lt;<a href=3D"mailto:users_at_[hidden]">us=
> ers_at_[hidden]</a>&gt;<br>
> Cc: VisIt Developers &lt;<a href=3D"mailto:visit-developers_at_[hidden]"=
> >visit-developers_at_[hidden]</a>&gt;<br>
> Subject: Re: [OMPI users] [visit-developers] /usr/bin/ld: cannot<br>
> =A0 =A0 =A0 =A0find-lrdmacm on 9184<br>
> <br>
> It depends on how Open MPI was built.<br>
> <br>
> If Open MPI was built without plugins (i.e., all the plugins are slurped up=
> into libmpi and friends), then yes, applications need to link against libr=
> dmacm to use the RDMA CM mode of OpenFabrics transport.<br>
> <br>
> If Open MPI was built with plugins (which is the default), then apps don&#3=
> 9;t need to link against librdmacm because the only use of rdmacm is in an =
> Open MPI plugin, and that plugin was linked against librdmacm.<br>
> <br>
> Make sense?<br>
> <br>
> That being said, the output from mpic++ --showme should give you something =
> that is directly compile-/link-able. =A0So it is odd if mpic++ is showing y=
> ou something that can&#39;t (or shouldn&#39;t?) be done. =A0Did a -L argume=
> nt get lost somewhere, perchance?<br>
>
> <br>
> Does linking MPI applications with mpic++ work properly, or does it result =
> in the same error? =A0If it results in the same error, then perhaps somethi=
> ng has changed since Open MPI was installed...?<br>
> <br>
> All this being said, two other random points:<br>
> <br>
> 1. Ensure that you&#39;re using the &quot;right&quot; mpic++. =A0I.e., make=
> sure it matches the version/installation of Open MPI that you&#39;re tryin=
> g to use.<br>
> <br>
> 2. If you don&#39;t link with the librdmacm, you&#39;re probably not losing=
> any important functionality unless you have an iWarp-based cluster (that&#=
> 39;s the only transport that *needs* librdmacm). =A0IB-based networks can u=
> se librdmacm, but don&#39;t *need* it (it&#39;s only used for making initia=
> l connections, so using librdmacm or not has no implications on overall MPI=
> performance). =A0It&#39;s still odd that mpic++ wants it and it can&#39;t =
> be found, though...<br>
>
> <br>
> Does that helps?<br>
> <div><div></div><div class=3D"h5"><br>
> <br>
> On Dec 15, 2009, at 11:11 PM, tom fogal wrote:<br>
> <br>
> &gt; Simon Su &lt;<a href=3D"mailto:newsgroup4ssu_at_[hidden]">newsgroup4ssu@=
> gmail.com</a>&gt; writes:<br>
> &gt; &gt; Hi Tom,<br>
> &gt; &gt;<br>
> &gt; &gt; I am using the standard openmpi package that run on all the clust=
> er<br>
> &gt; &gt; machines here at Princeton. So, maybe I shouldn&#39;t touch openm=
> pi. But,<br>
> &gt; &gt; removing -lrdmacm from the MPI_LIBS line in the machinename.conf =
> file<br>
> &gt; &gt; worked. =A0Any implication from doing this?<br>
> &gt;<br>
> &gt; The only thing it could possibly do is disable RDMA for you. =A0Howeve=
> r,<br>
> &gt; since removing it did not produce any undefined symbol errors, my gues=
> s<br>
> &gt; is that your OpenMPI isn&#39;t using RDMA anyway.<br>
> &gt;<br>
> &gt; There might be an OpenMPI bug here, though. =A0I&#39;ve cc&#39;d the O=
> penMPI<br>
> &gt; community to see if they have any input. =A0As a summary for them: Sim=
> on<br>
> &gt; is trying to build our MPI-enabled application. =A0A script which trie=
> s<br>
> &gt; to automate this adds the output of &quot;mpic++ -show&quot;. =A0His b=
> uild then<br>
> &gt; failed because it attempted to link against librdmacm, which does not<=
> br>
> &gt; exist in his normal search paths (or maybe at all). =A0Is it possible<=
> br>
> &gt; that `mpic++ -show&#39; includes/adds &quot;-lrdmacm&quot; even when O=
> penMPI is not<br>
> &gt; itself using the library?<br>
> &gt;<br>
> &gt; Thanks,<br>
> &gt;<br>
> &gt; -tom<br>
> &gt;<br>
> &gt; &gt; On Tue, Dec 15, 2009 at 8:46 PM, tom fogal &lt;<a href=3D"mailto:=
> tfogal_at_[hidden]">tfogal_at_[hidden]</a>&gt; wrote:<br>
> &gt; &gt;<br>
> &gt; &gt; &gt; Simon Su &lt;<a href=3D"mailto:newsgroup4ssu_at_[hidden]">news=
> group4ssu_at_[hidden]</a>&gt; writes:<br>
> &gt; &gt; &gt; &gt; I am getting this error message while building 9184.<br=
> >
> &gt; &gt; &gt; [snip]<br>
> &gt; &gt; &gt; &gt; -lz -lm -ldl =A0-lpthread -L/usr/local/openmpi/1.3.3/gc=
> c/x86_64/lib64<br>
> &gt; &gt; &gt; &gt; -lmpi_cxx -lmpi -lopen-rte -lopen-pal -lrdmacm -libverb=
> s -lnuma -ldl<br>
> &gt; &gt; &gt; -lnsl<br>
> &gt; &gt; &gt; &gt; -lutil -lm =A0-lcognomen \<br>
> &gt; &gt; &gt; &gt; =A0 =A0 =A0 =A0 -L/usr/local/openmpi/1.3.3/gcc/x86_64/l=
> ib64 -lmpi_cxx -lmpi<br>
> &gt; &gt; &gt; &gt; -lopen-rte -lopen-pal -lrdmacm -libverbs -lnuma -ldl -l=
> nsl -lutil -lm<br>
> &gt; &gt; &gt; &gt; -lcognomen<br>
> &gt; &gt; &gt; &gt; /usr/bin/ld: cannot find -lrdmacm<br>
> &gt; &gt; &gt; &gt; collect2: ld returned 1 exit status<br>
> &gt; &gt; &gt;<br>
> &gt; &gt; &gt; Your OpenMPI install (incorrectly?) thinks it has librdmacm =
> available,<br>
> &gt; &gt; &gt; but the library isn&#39;t in your search path.<br>
> &gt; &gt; &gt;<br>
> &gt; &gt; &gt; It apparently defaults to enabled in 1.3.3. =A0That seems ra=
> ther<br>
> &gt; &gt; &gt; silly, since I imagine the library requires RDMA hardware, w=
> hich<br>
> &gt; &gt; &gt; is of course not ubiquitous. =A0Anyway, try configuring Open=
> MPI with<br>
> &gt; &gt; &gt; --disable-openib-rdmacm and then rerunning build_visit.<br>
> &gt; &gt; &gt;<br>
> &gt; &gt; &gt; Of course, if you actually have an RDMA cluster, you&#39;ll =
> want to delve<br>
> &gt; &gt; &gt; deeper.<br>
> &gt; &gt; &gt;<br>
> &gt; &gt; &gt; Cheers,<br>
> &gt; &gt; &gt;<br>
> &gt; &gt; &gt; -tom<br>
> </div></div>&gt; _______________________________________________<br>
> &gt; users mailing list<br>
> &gt; <a href=3D"mailto:users_at_[hidden]">users_at_[hidden]</a><br>
> &gt; <a href=3D"http://www.open-mpi.org/mailman/listinfo.cgi/users" target=
> =3D"_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users><br>
> &gt;<br>
> <br>
> <br>
> - --<br>
> Jeff Squyres<br>
> <a href=3D"mailto:jsquyres_at_[hidden]">jsquyres_at_[hidden]</a><br>
> <br>
> <br>
> _______________________________________________<br>
> users mailing list<br>
> <a href=3D"mailto:users_at_[hidden]">users_at_[hidden]</a><br>
> <a href=3D"
http://www.open-mpi.org/mailman/listinfo.cgi/users" target=3D"_b=
> lank">http://www.open-mpi.org/mailman/listinfo.cgi/users><br>
> <br>
> ------- End of Forwarded Message<br>
> <br>
> </blockquote></div><br>
>
> --001636e0b6b9dccd4d047af22bb9--