Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Configure issue with/without HWLOC when PGI used and CUDA support enabled
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2014-02-14 12:58:30


To avoid a few back-n-forths in email, you might want to send all the diagnostic info here:

    http://www.open-mpi.org/community/help/

On Feb 14, 2014, at 12:15 PM, Rolf vandeVaart <rvandevaart_at_[hidden]> wrote:

> I assume your first issue is happening because you configured hwloc with cuda support which creates a dependency on libcudart.so. Not sure why that would mess up Open MPI. Can you send me how you configured hwloc?
>
> I am not sure I understand the second issue. Open MPI puts everything in lib even though you may be building for 64 bits. So all of these are fine.
> -I/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC/lib
> -Wl,-rpath -Wl,/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC/lib
> -L/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC/lib
>
> Rolf
>
>> -----Original Message-----
>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Filippo Spiga
>> Sent: Friday, February 14, 2014 9:44 AM
>> To: users_at_[hidden]
>> Subject: [OMPI users] Configure issue with/without HWLOC when PGI used
>> and CUDA support enabled
>>
>> Dear Open MPI developers,
>>
>> I just want to point to a weird behavior of the configure procedure I
>> discovered. I am compiling Open MPI 1.7.4 with CUDA support (CUDA 6.0 RC)
>> and PGI 14.1
>>
>> If I explicitly compile against a self-compiled version of HWLOC (1.8.1) using
>> this configure line ../configure CC=pgcc CXX=pgCC FC=pgf90 F90=pgf90 --
>> prefix=/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC --
>> enable-mpirun-prefix-by-default --with-fca=$FCA_DIR --with-
>> mxm=$MXM_DIR --with-knem=$KNEM_DIR --with-hwloc=/usr/local/Cluster-
>> Users/fs395/hwlock-1.8.1/gcc-4.4.7_cuda-6.0RC --with-
>> slurm=/usr/local/Cluster-Apps/slurm --with-cuda=/usr/local/Cluster-
>> Users/fs395/cuda/6.0-RC
>>
>> make fails telling me that it cannot find "-lcudart".
>>
>>
>> If I compile without HWLOC using this configure line:
>> ../configure CC=pgcc CXX=pgCC FC=pgf90 F90=pgf90 --
>> prefix=/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-6.0RC --
>> enable-mpirun-prefix-by-default --with-fca=$FCA_DIR --with-
>> mxm=$MXM_DIR --with-knem=$KNEM_DIR --with-slurm=/usr/local/Cluster-
>> Apps/slurm --with-cuda=/usr/local/Cluster-Users/fs395/cuda/6.0-RC
>>
>> make succeeds and I have Open MPI compiled properly.
>>
>> $ mpif90 -show
>> pgf90 -I/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-
>> 6.0RC/include -I/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-14.1_cuda-
>> 6.0RC/lib -Wl,-rpath -Wl,/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-
>> 14.1_cuda-6.0RC/lib -L/usr/local/Cluster-Users/fs395/openmpi-1.7.4/pgi-
>> 14.1_cuda-6.0RC/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -
>> lmpi $ ompi_info --all | grep btl_openib_have_cuda_gdr
>> MCA btl: informational "btl_openib_have_cuda_gdr" (current value:
>> "true", data source: default, level: 5 tuner/detail, type: bool)
>>
>> I wonder why the configure picks up lib instead of lib64. I will test the build
>> using real codes.
>>
>> Cheers,
>> Filippo
>>
>> --
>> Mr. Filippo SPIGA, M.Sc.
>> http://www.linkedin.com/in/filippospiga ~ skype: filippo.spiga
>>
>> <Nobody will drive us out of Cantor's paradise.> ~ David Hilbert
>>
>> *****
>> Disclaimer: "Please note this message and any attachments are
>> CONFIDENTIAL and may be privileged or otherwise protected from disclosure.
>> The contents are not to be disclosed to anyone other than the addressee.
>> Unauthorized recipients are requested to preserve this confidentiality and to
>> advise the sender immediately of any error in transmission."
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> -----------------------------------------------------------------------------------
> This email message is for the sole use of the intended recipient(s) and may contain
> confidential information. Any unauthorized review, use, disclosure or distribution
> is prohibited. If you are not the intended recipient, please contact the sender by
> reply email and destroy all copies of the original message.
> -----------------------------------------------------------------------------------
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/