Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] -Re Post- MPI_SUM is not defined on the MPI_INTEGER datatype
From: Hayato KUNIIE (kuni255_at_[hidden])
Date: 2013-06-14 22:18:03


Hello Following problem is solved by recompiling and reinstall Open MPI
for each nodes.

Thank you for your coorpolation.

-------------------------

I build bewulf type PC Cluster (Cent OS release 6.4). And I studing
about MPI.(Open MPI Ver.1.6.4) I tried following sample which using
MPI_REDUCE (FORTRAN).

Then, following Error occured.
--------------------------------------
[bwslv01:30793] *** An error occurred in MPI_Reduce: the reduction
operation MPI_SUM is not defined on the MPI_INTEGER datatype
[bwslv01:30793] *** on communicator MPI_COMM_WORLD
[bwslv01:30793] *** MPI_ERR_OP: invalid reduce operation
[bwslv01:30793] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
-------------------------------------- All informaion is showd in
attached file err.log. and source file is attached as main.f

This cluster system consist of one head node and 2 slave nodes.
And sharing home directory in head node by NFS. so Open MPI is installed
each nodes.

When I test this program on only head node, program is run correctly.
and output result.
But When I test this program on only slave node, same error occured.

Please tell me, good idea or advise.

Other information is included attached file.
Following is construction of directories .
ompiReport
├── head
│ ├── config.log // Item 3 on help page
│ ├── ifocnfig.txt // Item 8 on help page
│ ├── lstopo.txt // Item 5 on help page
   | |---- PATH.txt // Item 7 on help page
   | |---- LD_LIBRARY_PATH.txt // Item 7 on help page
│ └── ompi_info_all.txt // Item 4 on help page
├── ompi_info_full.txt // Item 6 on help page
|---- main.f // source file
|---- err.log // error message
├── slv01
│ ├── config.log // Item 3 on help page
│ ├── ifconfig.txt // Item 8 on help page
│ ├── lstopo.txt // Item 5 on help page
| |---- PATH.txt // Item 7 on help page
| |---- LD_LIBRARY_PATH.txt // Item 7 on help page
│ └── ompi_info_all.txt // Item 4 on help page
└── slv02
├── config.log // Item 3 on help page
├── ifconfig.txt // Item 8 on help page
├── lstopo.txt // Item 5 on help page
|---- PATH.txt // Item 7 on help page
|---- LD_LIBRARY_PATH.txt // Item 7 on help page
└── ompi_info_all.txt // Item 4 on help page

3 directories, 13 files

Best regards

(2013/05/16 23:24), Jeff Squyres (jsquyres) wrote:
> (OFF LIST)
>
> Let's figure this out off-list and post the final resolution back to the list.
>
> This is quite odd.
>
> You launched this mpirun from a single node, right? I'm trying to make sure that you're doing non-interactive logins on the remote nodes to find the ompi_info's, because sometimes there's a difference between paths that are set for interactive and non-interactive logins.
>
> Can you send all the information listed here:
>
> http://www.open-mpi.org/community/help/
>
>
>
> On May 16, 2013, at 9:53 AM, Hayato KUNIIE <kuni255_at_[hidden]> wrote:
>
>> Following is result of mpirun ompi_info on three_nodes.
>>
>> three nodes version is same.
>>
>> Package: Open MPI root_at_bwhead.clnet Distribution Open MPI root_at_bwslv01 Distribution Open MPI root_at_bwslv02 Distribution
>> Open MPI: 1.6.4 1.6.4 1.6.4
>> Open MPI SVN revision: r28081 r28081 r28081
>> Open MPI release date: Feb 19, 2013 Feb 19, 2013 Feb 19, 2013
>> Open RTE: 1.6.4 1.6.4 1.6.4
>> Open RTE SVN revision: r28081 r28081 r28081
>> Open RTE release date: Feb 19, 2013 Feb 19, 2013 Feb 19, 2013
>> OPAL: 1.6.4 1.6.4 1.6.4
>> OPAL SVN revision: r28081 r28081 r28081
>> OPAL release date: Feb 19, 2013 Feb 19, 2013 Feb 19, 2013
>> MPI API: 2.1 2.1 2.1
>> Ident string: 1.6.4 1.6.4 1.6.4
>> Prefix: /usr/local /usr/local /usr/local
>> Configured architecture: x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu
>> Configure host: bwhead.clnet bwslv01 bwslv02
>> Configured by: root root root
>> Configured on: Wed May 8 20:38:14 JST 2013 45 JST 2013 29 JST 2013
>> Configure host: bwhead.clnet bwslv01 bwslv02
>> Built by: root root root
>> Built on: Wed May 8 20:48:44 JST 2013 43 JST 2013 38 JST 2013
>> Built host: bwhead.clnet bwslv01 bwslv02
>> C bindings: yes yes yes
>> C++ bindings: yes yes yes
>> Fortran77 bindings: yes (all) yes (all) yes (all)
>> Fortran90 bindings: yes yes yes
>> Fortran90 bindings size: small small small
>> C compiler: gcc gcc gcc
>> C compiler absolute: /usr/bin/gcc /usr/bin/gcc /usr/bin/gcc
>> C compiler family name: GNU GNU GNU
>> C compiler version: 4.4.7 4.4.7 4.4.7
>> C++ compiler: g++ g++ g++
>> C++ compiler absolute: /usr/bin/g++ /usr/bin/g++ /usr/bin/g++
>> Fortran77 compiler: gfortran gfortran gfortran
>> Fortran77 compiler abs: /usr/bin/gfortran /usr/bin/gfortran /usr/bin/gfortran
>> Fortran90 compiler: gfortran gfortran gfortran
>> Fortran90 compiler abs: /usr/bin/gfortran /usr/bin/gfortran /usr/bin/gfortran
>> C profiling: yes yes yes
>> C++ profiling: yes yes yes
>> Fortran77 profiling: yes yes yes
>> Fortran90 profiling: yes yes yes
>> C++ exceptions: no no no
>> Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no) no) no)
>> Sparse Groups: no no no
>> Internal debug support: no no no
>> MPI interface warnings: no no no
>> MPI parameter check: runtime runtime runtime
>> Memory profiling support: no no no
>> Memory debugging support: no no no
>> libltdl support: yes yes yes
>> Heterogeneous support: no no no
>> mpirun default --prefix: no no no
>> MPI I/O support: yes yes yes
>> MPI_WTIME support: gettimeofday gettimeofday gettimeofday
>> Symbol vis. support: yes yes yes
>> Host topology support: yes yes yes
>> MPI extensions: affinity example affinity example affinity example
>> FT Checkpoint support: no (checkpoint thread: no) no) no)
>> VampirTrace support: yes yes yes
>> MPI_MAX_PROCESSOR_NAME: 256 256 256
>> MPI_MAX_ERROR_STRING: 256 256 256
>> MPI_MAX_OBJECT_NAME: 64 64 64
>> MPI_MAX_INFO_KEY: 36 36 36
>> MPI_MAX_INFO_VAL: 256 256 256
>> MPI_MAX_PORT_NAME: 1024 1024 1024
>> MPI_MAX_DATAREP_STRING: 128 128 128
>> Package: Open MPI root_at_bwslv01 Distribution execinfo (MCA v2.0, API v2.0, Component v1.6.4) execinfo (MCA v2.0, API v2.0, Component v1.6.4)
>> Open MPI: 1.6.4 linux (MCA v2.0, API v2.0, Component v1.6.4) linux (MCA v2.0, API v2.0, Component v1.6.4)
>> Open MPI SVN revision: r28081 hwloc (MCA v2.0, API v2.0, Component v1.6.4) hwloc (MCA v2.0, API v2.0, Component v1.6.4)
>> Open MPI release date: Feb 19, 2013 auto_detect (MCA v2.0, API v2.0, Component v1.6.4) auto_detect (MCA v2.0, API v2.0, Component v1.6.4)
>> Open RTE: 1.6.4 file (MCA v2.0, API v2.0, Component v1.6.4) file (MCA v2.0, API v2.0, Component v1.6.4)
>> Open RTE SVN revision: r28081 mmap (MCA v2.0, API v2.0, Component v1.6.4) mmap (MCA v2.0, API v2.0, Component v1.6.4)
>> Open RTE release date: Feb 19, 2013 posix (MCA v2.0, API v2.0, Component v1.6.4) posix (MCA v2.0, API v2.0, Component v1.6.4)
>> OPAL: 1.6.4 sysv (MCA v2.0, API v2.0, Component v1.6.4) sysv (MCA v2.0, API v2.0, Component v1.6.4)
>> OPAL SVN revision: r28081 first_use (MCA v2.0, API v2.0, Component v1.6.4) first_use (MCA v2.0, API v2.0, Component v1.6.4)
>> OPAL release date: Feb 19, 2013 hwloc (MCA v2.0, API v2.0, Component v1.6.4) hwloc (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI API: 2.1 linux (MCA v2.0, API v2.0, Component v1.6.4) linux (MCA v2.0, API v2.0, Component v1.6.4)
>> Ident string: 1.6.4 env (MCA v2.0, API v2.0, Component v1.6.4) env (MCA v2.0, API v2.0, Component v1.6.4)
>> Prefix: /usr/local config (MCA v2.0, API v2.0, Component v1.6.4) config (MCA v2.0, API v2.0, Component v1.6.4)
>> Configured architecture: x86_64-unknown-linux-gnu linux (MCA v2.0, API v2.0, Component v1.6.4) linux (MCA v2.0, API v2.0, Component v1.6.4)
>> Configure host: bwslv01 hwloc132 (MCA v2.0, API v2.0, Component v1.6.4) hwloc132 (MCA v2.0, API v2.0, Component v1.6.4)
>> Configured by: root orte (MCA v2.0, API v2.0, Component v1.6.4) orte (MCA v2.0, API v2.0, Component v1.6.4)
>> Configured on: Wed May 8 20:56:45 JST 2013 orte (MCA v2.0, API v2.0, Component v1.6.4) orte (MCA v2.0, API v2.0, Component v1.6.4)
>> Configure host: bwslv01 basic (MCA v2.0, API v2.0, Component v1.6.4) basic (MCA v2.0, API v2.0, Component v1.6.4)
>> Built by: root bucket (MCA v2.0, API v2.0, Component v1.6.4) bucket (MCA v2.0, API v2.0, Component v1.6.4)
>> Built on: Wed May 8 21:05:43 JST 2013 basic (MCA v2.0, API v2.0, Component v1.6.4) basic (MCA v2.0, API v2.0, Component v1.6.4)
>> Built host: bwslv01 hierarch (MCA v2.0, API v2.0, Component v1.6.4) hierarch (MCA v2.0, API v2.0, Component v1.6.4)
>> C bindings: yes inter (MCA v2.0, API v2.0, Component v1.6.4) inter (MCA v2.0, API v2.0, Component v1.6.4)
>> C++ bindings: yes self (MCA v2.0, API v2.0, Component v1.6.4) self (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran77 bindings: yes (all) sm (MCA v2.0, API v2.0, Component v1.6.4) sm (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran90 bindings: yes sync (MCA v2.0, API v2.0, Component v1.6.4) sync (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran90 bindings size: small tuned (MCA v2.0, API v2.0, Component v1.6.4) tuned (MCA v2.0, API v2.0, Component v1.6.4)
>> C compiler: gcc romio (MCA v2.0, API v2.0, Component v1.6.4) romio (MCA v2.0, API v2.0, Component v1.6.4)
>> C compiler absolute: /usr/bin/gcc fake (MCA v2.0, API v2.0, Component v1.6.4) fake (MCA v2.0, API v2.0, Component v1.6.4)
>> C compiler family name: GNU rdma (MCA v2.0, API v2.0, Component v1.6.4) rdma (MCA v2.0, API v2.0, Component v1.6.4)
>> C compiler version: 4.4.7 sm (MCA v2.0, API v2.0, Component v1.6.4) sm (MCA v2.0, API v2.0, Component v1.6.4)
>> C++ compiler: g++ bfo (MCA v2.0, API v2.0, Component v1.6.4) bfo (MCA v2.0, API v2.0, Component v1.6.4)
>> C++ compiler absolute: /usr/bin/g++ csum (MCA v2.0, API v2.0, Component v1.6.4) csum (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran77 compiler: gfortran ob1 (MCA v2.0, API v2.0, Component v1.6.4) ob1 (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran77 compiler abs: /usr/bin/gfortran v (MCA v2.0, API v2.0, Component v1.6.4) v (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran90 compiler: gfortran r2 (MCA v2.0, API v2.0, Component v1.6.4) r2 (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran90 compiler abs: /usr/bin/gfortran vma (MCA v2.0, API v2.0, Component v1.6.4) vma (MCA v2.0, API v2.0, Component v1.6.4)
>> C profiling: yes self (MCA v2.0, API v2.0, Component v1.6.4) self (MCA v2.0, API v2.0, Component v1.6.4)
>> C++ profiling: yes sm (MCA v2.0, API v2.0, Component v1.6.4) sm (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran77 profiling: yes tcp (MCA v2.0, API v2.0, Component v1.6.4) tcp (MCA v2.0, API v2.0, Component v1.6.4)
>> Fortran90 profiling: yes unity (MCA v2.0, API v2.0, Component v1.6.4) unity (MCA v2.0, API v2.0, Component v1.6.4)
>> C++ exceptions: no pt2pt (MCA v2.0, API v2.0, Component v1.6.4) pt2pt (MCA v2.0, API v2.0, Component v1.6.4)
>> Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no) rdma (MCA v2.0, API v2.0, Component v1.6.4) rdma (MCA v2.0, API v2.0, Component v1.6.4)
>> Sparse Groups: no hnp (MCA v2.0, API v2.0, Component v1.6.4) hnp (MCA v2.0, API v2.0, Component v1.6.4)
>> Internal debug support: no orted (MCA v2.0, API v2.0, Component v1.6.4) orted (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI interface warnings: no tool (MCA v2.0, API v2.0, Component v1.6.4) tool (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI parameter check: runtime tcp (MCA v2.0, API v2.0, Component v1.6.4) tcp (MCA v2.0, API v2.0, Component v1.6.4)
>> Memory profiling support: no default (MCA v2.0, API v2.0, Component v1.6.4) default (MCA v2.0, API v2.0, Component v1.6.4)
>> Memory debugging support: no cm (MCA v2.0, API v2.0, Component v1.6.4) cm (MCA v2.0, API v2.0, Component v1.6.4)
>> libltdl support: yes loadleveler (MCA v2.0, API v2.0, Component v1.6.4) loadleveler (MCA v2.0, API v2.0, Component v1.6.4)
>> Heterogeneous support: no slurm (MCA v2.0, API v2.0, Component v1.6.4) slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> mpirun default --prefix: no load_balance (MCA v2.0, API v2.0, Component v1.6.4) load_balance (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI I/O support: yes rank_file (MCA v2.0, API v2.0, Component v1.6.4) rank_file (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI_WTIME support: gettimeofday resilient (MCA v2.0, API v2.0, Component v1.6.4) resilient (MCA v2.0, API v2.0, Component v1.6.4)
>> Symbol vis. support: yes round_robin (MCA v2.0, API v2.0, Component v1.6.4) round_robin (MCA v2.0, API v2.0, Component v1.6.4)
>> Host topology support: yes seq (MCA v2.0, API v2.0, Component v1.6.4) seq (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI extensions: affinity example topo (MCA v2.0, API v2.0, Component v1.6.4) topo (MCA v2.0, API v2.0, Component v1.6.4)
>> FT Checkpoint support: no (checkpoint thread: no) oob (MCA v2.0, API v2.0, Component v1.6.4) oob (MCA v2.0, API v2.0, Component v1.6.4)
>> VampirTrace support: yes binomial (MCA v2.0, API v2.0, Component v1.6.4) binomial (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI_MAX_PROCESSOR_NAME: 256 cm (MCA v2.0, API v2.0, Component v1.6.4) cm (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI_MAX_ERROR_STRING: 256 direct (MCA v2.0, API v2.0, Component v1.6.4) direct (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI_MAX_OBJECT_NAME: 64 linear (MCA v2.0, API v2.0, Component v1.6.4) linear (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI_MAX_INFO_KEY: 36 radix (MCA v2.0, API v2.0, Component v1.6.4) radix (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI_MAX_INFO_VAL: 256 slave (MCA v2.0, API v2.0, Component v1.6.4) slave (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI_MAX_PORT_NAME: 1024 rsh (MCA v2.0, API v2.0, Component v1.6.4) rsh (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI_MAX_DATAREP_STRING: 128 slurm (MCA v2.0, API v2.0, Component v1.6.4) slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> Package: Open MPI root_at_bwslv02 Distribution rsh (MCA v2.0, API v2.0, Component v1.6.4) rsh (MCA v2.0, API v2.0, Component v1.6.4)
>> Open MPI: 1.6.4 default (MCA v2.0, API v2.0, Component v1.6.4) default (MCA v2.0, API v2.0, Component v1.6.4)
>> Open MPI SVN revision: r28081 env (MCA v2.0, API v2.0, Component v1.6.4) env (MCA v2.0, API v2.0, Component v1.6.4)
>> Open MPI release date: Feb 19, 2013 hnp (MCA v2.0, API v2.0, Component v1.6.4) hnp (MCA v2.0, API v2.0, Component v1.6.4)
>> Open RTE: 1.6.4 singleton (MCA v2.0, API v2.0, Component v1.6.4) singleton (MCA v2.0, API v2.0, Component v1.6.4)
>> Open RTE SVN revision: r28081 slave (MCA v2.0, API v2.0, Component v1.6.4) slave (MCA v2.0, API v2.0, Component v1.6.4)
>> Open RTE release date: Feb 19, 2013 slurm (MCA v2.0, API v2.0, Component v1.6.4) slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> OPAL: 1.6.4 slurmd (MCA v2.0, API v2.0, Component v1.6.4) slurmd (MCA v2.0, API v2.0, Component v1.6.4)
>> OPAL SVN revision: r28081 tool (MCA v2.0, API v2.0, Component v1.6.4) tool (MCA v2.0, API v2.0, Component v1.6.4)
>> OPAL release date: Feb 19, 2013 bad (MCA v2.0, API v2.0, Component v1.6.4) bad (MCA v2.0, API v2.0, Component v1.6.4)
>> MPI API: 2.1 basic (MCA v2.0, API v2.0, Component v1.6.4) basic (MCA v2.0, API v2.0, Component v1.6.4)
>> Ident string: 1.6.4 hier (MCA v2.0, API v2.0, Component v1.6.4) hier (MCA v2.0, API v2.0, Component v1.6.4)
>> Prefix: /usr/local command (MCA v2.0, API v1.0, Component v1.6.4) command (MCA v2.0, API v1.0, Component v1.6.4)
>> Configured architecture: x86_64-unknown-linux-gnu syslog (MCA v2.0, API v1.0, Component v1.6.4) syslog (MCA v2.0, API v1.0, Component v1.6.4)
>> Configure host: bwslv02
>> Configured by: root
>> Configured on: Wed May 8 20:56:29 JST 2013
>> Configure host: bwslv02
>> Built by: root
>> Built on: Wed May 8 21:05:38 JST 2013
>> Built host: bwslv02
>> C bindings: yes
>> C++ bindings: yes
>> Fortran77 bindings: yes (all)
>> Fortran90 bindings: yes
>> Fortran90 bindings size: small
>> C compiler: gcc
>> C compiler absolute: /usr/bin/gcc
>> C compiler family name: GNU
>> C compiler version: 4.4.7
>> C++ compiler: g++
>> C++ compiler absolute: /usr/bin/g++
>> Fortran77 compiler: gfortran
>> Fortran77 compiler abs: /usr/bin/gfortran
>> Fortran90 compiler: gfortran
>> Fortran90 compiler abs: /usr/bin/gfortran
>> C profiling: yes
>> C++ profiling: yes
>> Fortran77 profiling: yes
>> Fortran90 profiling: yes
>> C++ exceptions: no
>> Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no)
>> Sparse Groups: no
>> Internal debug support: no
>> MPI interface warnings: no
>> MPI parameter check: runtime
>> Memory profiling support: no
>> Memory debugging support: no
>> libltdl support: yes
>> Heterogeneous support: no
>> mpirun default --prefix: no
>> MPI I/O support: yes
>> MPI_WTIME support: gettimeofday
>> Symbol vis. support: yes
>> Host topology support: yes
>> MPI extensions: affinity example
>> FT Checkpoint support: no (checkpoint thread: no)
>> VampirTrace support: yes
>> MPI_MAX_PROCESSOR_NAME: 256
>> MPI_MAX_ERROR_STRING: 256
>> MPI_MAX_OBJECT_NAME: 64
>> MPI_MAX_INFO_KEY: 36
>> MPI_MAX_INFO_VAL: 256
>> MPI_MAX_PORT_NAME: 1024
>> MPI_MAX_DATAREP_STRING: 128
>> MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA memory: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA carto: file (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: mmap (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: posix (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: sysv (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA maffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA timer: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA sysinfo: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA hwloc: hwloc132 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA dpm: orte (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA allocator: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: inter (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: self (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: sync (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: tuned (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA io: romio (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: fake (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: bfo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: csum (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: v (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA bml: r2 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rcache: vma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: self (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: tcp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA topo: unity (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA osc: rdma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: hnp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: orted (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: tool (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA oob: tcp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA odls: default (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: cm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: loadleveler (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rml: oob (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: binomial (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: cm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: direct (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: linear (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: radix (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: slave (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA plm: rsh (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA filem: rsh (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA errmgr: default (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: env (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: hnp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: singleton (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slave (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: tool (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA notifier: command (MCA v2.0, API v1.0, Component v1.6.4)
>> MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.6.4)
>> MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA memory: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA carto: file (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: mmap (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: posix (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: sysv (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA maffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA timer: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA sysinfo: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA hwloc: hwloc132 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA dpm: orte (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA allocator: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: inter (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: self (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: sync (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: tuned (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA io: romio (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: fake (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: bfo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: csum (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: v (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA bml: r2 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rcache: vma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: self (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: tcp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA topo: unity (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA osc: rdma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: hnp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: orted (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: tool (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA oob: tcp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA odls: default (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: cm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: loadleveler (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rml: oob (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: binomial (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: cm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: direct (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: linear (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: radix (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: slave (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA plm: rsh (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA filem: rsh (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA errmgr: default (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: env (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: hnp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: singleton (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slave (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: tool (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA notifier: command (MCA v2.0, API v1.0, Component v1.6.4)
>> MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.6.4)
>> MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA memory: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA carto: file (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: mmap (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: posix (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA shmem: sysv (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA maffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA timer: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA sysinfo: linux (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA hwloc: hwloc132 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA dpm: orte (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA allocator: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: inter (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: self (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: sync (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA coll: tuned (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA io: romio (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: fake (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA mpool: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: bfo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: csum (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA pml: v (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA bml: r2 (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rcache: vma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: self (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: sm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA btl: tcp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA topo: unity (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA osc: rdma (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: hnp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: orted (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA iof: tool (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA oob: tcp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA odls: default (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: cm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: loadleveler (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA rml: oob (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: binomial (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: cm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: direct (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: linear (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: radix (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA routed: slave (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA plm: rsh (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA filem: rsh (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA errmgr: default (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: env (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: hnp (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: singleton (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slave (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA ess: tool (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.6.4)
>> MCA notifier: command (MCA v2.0, API v1.0, Component v1.6.4)
>> MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.6.4)
>>
>>
>>
>>
>> (2013/05/16 9:12), Jeff Squyres (jsquyres) wrote:
>>> I am unable to replicate your error -- 1.6.4 has MPI_REDUCE defined on MPI_SUM properly.
>>>
>>> Are you absolutely sure you're using OMPI 1.6.4 on all nodes?
>>>
>>> Try this:
>>>
>>> mpirun ... ompi_info
>>>
>>> (insert whatever hostfile and -np value you're using for your fortran test) and see what is reported as the OMPI version from other nodes.
>>>
>>>
>>> On May 15, 2013, at 7:46 AM, Hayato KUNIIE <kuni255_at_[hidden]> wrote:
>>>
>>>> I using Ver, 1.6.4 in all nodes.
>>>>
>>>> (2013/05/15 7:10), Jeff Squyres (jsquyres) wrote:
>>>>> Are you sure that you have exactly the same version of Open MPI on all your nodes?
>>>>>
>>>>>
>>>>> On May 14, 2013, at 11:39 AM, Hayato KUNIIE <kuni255_at_[hidden]> wrote:
>>>>>
>>>>>> Hello I'm kuni255
>>>>>>
>>>>>> I build bewulf type PC Cluster (Cent OS release 6.4). And I studing
>>>>>> about MPI.(Open MPI Ver.1.6.4) I tried following sample which using
>>>>>> MPI_REDUCE.
>>>>>>
>>>>>> Then, Error occured.
>>>>>>
>>>>>> This cluster system consist of one head node and 2 slave nodes.
>>>>>> And sharing home directory in head node by NFS. so Open MPI is installed
>>>>>> each nodes.
>>>>>>
>>>>>> When I test this program on only head node, program is run correctly.
>>>>>> and output result.
>>>>>> But When I test this program on only slave node, same error occured.
>>>>>>
>>>>>> Please tell me, good idea : )
>>>>>>
>>>>>> Error message
>>>>>> [bwslv01:30793] *** An error occurred in MPI_Reduce: the reduction
>>>>>> operation MPI_SUM is not defined on the MPI_INTEGER datatype
>>>>>> [bwslv01:30793] *** on communicator MPI_COMM_WORLD
>>>>>> [bwslv01:30793] *** MPI_ERR_OP: invalid reduce operation
>>>>>> [bwslv01:30793] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
>>>>>> --------------------------------------------------------------------------
>>>>>> mpirun has exited due to process rank 1 with PID 30793 on
>>>>>> node bwslv01 exiting improperly. There are two reasons this could occur:
>>>>>>
>>>>>> 1. this process did not call "init" before exiting, but others in
>>>>>> the job did. This can cause a job to hang indefinitely while it waits
>>>>>> for all processes to call "init". By rule, if one process calls "init",
>>>>>> then ALL processes must call "init" prior to termination.
>>>>>>
>>>>>> 2. this process called "init", but exited without calling "finalize".
>>>>>> By rule, all processes that call "init" MUST call "finalize" prior to
>>>>>> exiting or it will be considered an "abnormal termination"
>>>>>>
>>>>>> This may have caused other processes in the application to be
>>>>>> terminated by signals sent by mpirun (as reported here).
>>>>>> --------------------------------------------------------------------------
>>>>>> [bwhead.clnet:02147] 1 more process has sent help message
>>>>>> help-mpi-errors.txt / mpi_errors_are_fatal
>>>>>> [bwhead.clnet:02147] Set MCA parameter "orte_base_help_aggregate" to 0
>>>>>> to see all help / error messages
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Fortran90 source code
>>>>>> include 'mpif.h'
>>>>>> parameter(nmax=12)
>>>>>> integer n(nmax)
>>>>>>
>>>>>> call mpi_init(ierr)
>>>>>> call mpi_comm_size(MPI_COMM_WORLD, isize, ierr)
>>>>>> call mpi_comm_rank(MPI_COMM_WORLD, irank, ierr)
>>>>>> ista=irank*(nmax/isize) + 1
>>>>>> iend=ista+(nmax/isize-1)
>>>>>> isum=0
>>>>>> do i=1,nmax
>>>>>> n(i) = i
>>>>>> isum = isum + n(i)
>>>>>> end do
>>>>>> call mpi_reduce(isum, itmp, 1, MPI_INTEGER, MPI_SUM,
>>>>>> & 0, MPI_COMM_WORLD, ierr)
>>>>>>
>>>>>> if (irank == 0) then
>>>>>> isum=itmp
>>>>>> WRITE(*,*) isum
>>>>>> endif
>>>>>> call mpi_finalize(ierr)
>>>>>> end
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>