Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] 50% performance reduction due to OpenMPI v 1.3.2 forcing all MPI traffic over Ethernet instead of using Infiniband
From: Jim Kress ORG (jimkress_58_at_[hidden])
Date: 2009-06-23 17:49:44


For v 1.3.2:

Here is the ompi_info -config output and I've attached a copy of the
config.log file which seems to clearly indicate it found the infiniband
libraries.

[root_at_master ~]# ompi_info -config
           Configured by: root
           Configured on: Sun Jun 21 22:02:59 EDT 2009
          Configure host: master.org
                Built by: root
                Built on: Sun Jun 21 22:10:07 EDT 2009
              Built host: master.org
              C bindings: yes
            C++ bindings: yes
      Fortran77 bindings: yes (all)
      Fortran90 bindings: yes
 Fortran90 bindings size: small
              C compiler: gcc
     C compiler absolute: /usr/bin/gcc
             C char size: 1
             C bool size: 1
            C short size: 2
              C int size: 4
             C long size: 8
            C float size: 4
           C double size: 8
          C pointer size: 8
            C char align: 1
            C bool align: 1
             C int align: 4
           C float align: 4
          C double align: 8
            C++ compiler: g++
   C++ compiler absolute: /usr/bin/g++
      Fortran77 compiler: gfortran
  Fortran77 compiler abs: /usr/bin/gfortran
      Fortran90 compiler: gfortran
  Fortran90 compiler abs: /usr/bin/gfortran
       Fort integer size: 4
       Fort logical size: 4
 Fort logical value true: 1
      Fort have integer1: yes
      Fort have integer2: yes
      Fort have integer4: yes
      Fort have integer8: yes
     Fort have integer16: no
         Fort have real4: yes
         Fort have real8: yes
        Fort have real16: no
      Fort have complex8: yes
     Fort have complex16: yes
     Fort have complex32: no
      Fort integer1 size: 1
      Fort integer2 size: 2
      Fort integer4 size: 4
      Fort integer8 size: 8
     Fort integer16 size: -1
          Fort real size: 4
         Fort real4 size: 4
         Fort real8 size: 8
        Fort real16 size: -1
      Fort dbl prec size: 4
          Fort cplx size: 4
      Fort dbl cplx size: 4
         Fort cplx8 size: 8
        Fort cplx16 size: 16
        Fort cplx32 size: -1
      Fort integer align: 4
     Fort integer1 align: 1
     Fort integer2 align: 2
     Fort integer4 align: 4
     Fort integer8 align: 8
    Fort integer16 align: -1
         Fort real align: 4
        Fort real4 align: 4
        Fort real8 align: 8
       Fort real16 align: -1
     Fort dbl prec align: 4
         Fort cplx align: 4
     Fort dbl cplx align: 4
        Fort cplx8 align: 4
       Fort cplx16 align: 8
       Fort cplx32 align: -1
             C profiling: yes
           C++ profiling: yes
     Fortran77 profiling: yes
     Fortran90 profiling: yes
          C++ exceptions: no
          Thread support: posix (mpi: no, progress: no)
           Sparse Groups: no
            Build CFLAGS: -O3 -DNDEBUG -finline-functions
-fno-strict-aliasing
                          -pthread -fvisibility=hidden
          Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
            Build FFLAGS:
           Build FCFLAGS:
           Build LDFLAGS: -export-dynamic
              Build LIBS: -lnsl -lutil -lm
    Wrapper extra CFLAGS: -pthread
  Wrapper extra CXXFLAGS: -pthread
    Wrapper extra FFLAGS: -pthread
   Wrapper extra FCFLAGS: -pthread
   Wrapper extra LDFLAGS:
      Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm
-ldl
  Internal debug support: no
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
         libltdl support: yes
   Heterogeneous support: no
 mpirun default --prefix: no
         MPI I/O support: yes
       MPI_WTIME support: gettimeofday
Symbol visibility support: yes
   FT Checkpoint support: no (checkpoint thread: no)
[root_at_master ~]#

On Tue, 2009-06-23 at 15:20 -0600, Ralph Castain wrote:
> Hmmm...just to be clear - did you run this against OMPI 1.3.2, or
> 1.2.8? I see a 1.2.8 in your app name, hence the question.
>
> This option only works with 1.3.2, I'm afraid - it was a new feature.
>
> Ralph
>
> On Jun 23, 2009, at 2:31 PM, Jim Kress ORG wrote:
>
> > Ralph,
> >
> > I did the following:
> >
> > export OMPI_MCA_mpi_show_mca_params="file,env"
> >
> > then I checked and found it via the set command as
> >
> > OMPI_MCA_mpi_show_mca_params=file,env
> >
> > I then ran my application
> >
> > ./orca hexatriene_TDDFT_get_asa_input_parallel_1.inp >
> > 1.2.8_test_crafted_input_file.out
> >
> > and got the expected ORCA output in the .out file but nothing at the
> > command line or in the .out file about mca_params
> >
> > What did I do wrong?
> >
> > Jim
> >
> > On Mon, 2009-06-22 at 19:40 -0600, Ralph Castain wrote:
> >> Sounds very strange, indeed. You might want to check that your app is
> >> actually getting the MCA param that you think it is. Try adding:
> >>
> >> -mca mpi_show_mca_params file,env
> >>
> >> to your cmd line. This will cause rank=0 to output the MCA params it
> >> thinks were set via the default files and/or environment (including
> >> cmd line).
> >>
> >> Ralph
> >>
> >> On Jun 22, 2009, at 6:14 PM, Jim Kress ORG wrote:
> >>
> >>> For the app I am using, ORCA (a Quantum Chemistry program), when it
> >>> was
> >>> compiled using openMPI 1.2.8 and run under 1.2.8 with the
> >>> following in
> >>> the openmpi-mca-params.conf file:
> >>>
> >>> btl=self,openib
> >>>
> >>> the app ran fine with no traffic over my Ethernet network and all
> >>> traffic over my Infiniband network.
> >>>
> >>> However, now that ORCA has been recompiled with openMPI v1.3.2 and
> >>> run
> >>> under 1.3.2 (using the same openmpi-mca-params.conf file), the
> >>> performance has been reduced by 50% and all the MPI traffic is going
> >>> over the Ethernet network.
> >>>
> >>> As a matter of fact, the openMPI v1.3.2 performance now looks
> >>> exactly
> >>> like the performance I get if I use MPICH 1.2.7.
> >>>
> >>> Anyone have any ideas:
> >>>
> >>> 1) How could this have happened?
> >>>
> >>> 2) How can I fix it?
> >>>
> >>> a 50% reduction in performance is just not acceptable. Ideas/
> >>> suggestions would be appreciated.
> >>>
> >>> Jim
> >>>
> >>> _______________________________________________
> >>> users mailing list
> >>> users_at_[hidden]
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >> _______________________________________________
> >> users mailing list
> >> users_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users