Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2
From: Tim Carlson (tim.carlson_at_[hidden])
Date: 2013-07-08 18:54:32


On Mon, 8 Jul 2013, Elken, Tom wrote:

My mistake on the OFED bits. The host I was installing on did not have all
of the MPSS software installed (my cluster admin node and not one of the
compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with
compiling the btl:openib bits with both the GNU and Intel compilers using
the cross-compiler route (-mmic on the Intel side)

Still working on getting the resulting mpicc wrapper working on the MIC
side. When I get a working example I'll post the results.

Thanks!

Tim

>
>  
>
> Hi Tim,
>
>  
>
>  
>
> Well, in general and not on MIC I usually build the MPI stacks using the
> Intel compiler set. Have you ran into s/w that requires GCC instead of Intel
> compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to produce
> MIC native code (the OpenMPI stack for that matter)? 
>
> [Tom]
>
> Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to
> build MIC code.
>
>  
>
> Tim wrote: “My first pass at doing a cross-compile with the GNU compilers
> failed to produce something with OFED support (not surprising)
>
> export PATH=/usr/linux-k1om-4.7/bin:$PATH
> ./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
> --disable-mpi-f77
>
> checking if MCA component btl:openib can compile... no
>
>  
>
> Regarding a Gnu cross compiler, this worked for one of our engineers
> building for True Scale HCAs and PSM/infinipath.  But it may give useful
> tips for building for btl:openib as well:
>
>  
>
> 3. How to configure/compile OpenMPI:
>
>        a). untar the openmpi tarball.
>
>        b). edit configure in top directory, add '-linfinipath' after
> '-lpsm_infinipath "
>
>            but not necessary for messages, only for command lines.
>
>  
>
>        c). run the following script,
>
> #!/bin/sh
>
>  
>
> ./configure \
>
> --host=x86_64-k1om-linux \
>
> --enable-mpi-f77=no --enable-mpi-f90=no \
>
> --with-psm=/…/psm-7.6 \
>
> --prefix=/…/openmpi \
>
> CC=x86_64-k1om-linux-gcc  CXX=x86_64-k1om-linux-g++ \
>
> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib
>
>  
>
>        d). run 'make' and 'make install'
>
>  
>
> OK, I see that they did not configure for mpi-f77 & mpif90, but perhaps this
> is still helpful, if the AR and RANLIB flags are important.
>
> -Tom
>
>  
>
>  
>
>  
>
> regards
>
> Michael
>
>  
>
> On Mon, Jul 8, 2013 at 4:30 PM, Tim Carlson <tim.carlson_at_[hidden]> wrote:
>
> On Mon, 8 Jul 2013, Elken, Tom wrote:
>
> It isn't quite so easy.
>
> Out of the box, there is no gcc on the Phi card. You can use the cross
> compiler on the host, but you don't get gcc on the Phi by default.
>
> See this post http://software.intel.com/en-us/forums/topic/382057
>
> I really think you would need to build and install gcc on the Phi first.
>
> My first pass at doing a cross-compile with the GNU compilers failed to
> produce something with OFED support (not surprising)
>
> export PATH=/usr/linux-k1om-4.7/bin:$PATH
> ./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
> --disable-mpi-f77
>
> checking if MCA component btl:openib can compile... no
>
>
> Tim
>
>  
>
>
>  
>
> Thanks Tom, that sounds good. I will give it a try as soon as
> our Phi host
> here host gets installed. 
>
>  
>
> I assume that all the prerequisite libs and bins on the Phi side
> are
> available when we download the Phi s/w stack from Intel's site,
> right ?
>
> [Tom]
>
> Right.  When you install Intel’s MPSS (Manycore Platform
> Software Stack),
> including following the section on “OFED Support” in the readme
> file, you
> should have all the prerequisite libs and bins.  Note that I
> have not built
> Open MPI for Xeon Phi for your interconnect, but it seems to me
> that it
> should work.
>
>  
>
> -Tom
>
>  
>
> Cheers
>
> Michael
>
>  
>
>  
>
>  
>
> On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom
> <tom.elken_at_[hidden]> wrote:
>
> Do you guys have any plan to support Intel Phi in the future?
> That is,
> running MPI code on the Phi cards or across the multicore and
> Phi, as Intel
> MPI does?
>
> [Tom]
>
> Hi Michael,
>
> Because a Xeon Phi card acts a lot like a Linux host with an x86
> architecture, you can build your own Open MPI libraries to serve
> this
> purpose.
>
> Our team has used existing (an older 1.4.3 version of) Open MPI
> source to
> build an Open MPI for running MPI code on Intel Xeon Phi cards
> over Intel’s
> (formerly QLogic’s) True Scale InfiniBand fabric, and it works
> quite well. 
> We have not released a pre-built Open MPI as part of any Intel
> software
> release.   But I think if you have a compiler for Xeon Phi
> (Intel Compiler
> or GCC) and an interconnect for it, you should be able to build
> an Open MPI
> that works on Xeon Phi. 
>
> Cheers,
> Tom Elken
>
> thanks...
>
> Michael
>
>  
>
> On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain <rhc_at_[hidden]>
> wrote:
>
> Rolf will have to answer the question on level of support. The
> CUDA code is
> not in the 1.6 series as it was developed after that series went
> "stable".
> It is in the 1.7 series, although the level of support will
> likely be
> incrementally increasing as that "feature" series continues to
> evolve.
>
>
>
> On Jul 6, 2013, at 12:06 PM, Michael Thomadakis
> <drmichaelt7777_at_[hidden]>
> wrote:
>
> > Hello OpenMPI,
> >
> > I am wondering what level of support is there for CUDA and
> GPUdirect on
> OpenMPI 1.6.5 and 1.7.2.
> >
> > I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ.
> However, it
> seems that with configure v1.6.5 it was ignored.
> >
> > Can you identify GPU memory and send messages from it directly
> without
> copying to host memory first?
> >
> >
> > Or in general, what level of CUDA support is there on 1.6.5
> and 1.7.2 ? Do
> you support SDK 5.0 and above?
> >
> > Cheers ...
> > Michael
>
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>  
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>  
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>  
>
>
>