Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] example program "ring" hangs when running across multiple hardware nodes
From: Jed O. Kaplan (jedokaplan_at_[hidden])
Date: 2013-07-04 18:37:00


Dear openmpi gurus,

I am running openmpi 1.7.2 on a homogenous cluster of Apple XServes
running OS X 10.6.8. My hardware nodes are connected through four
gigabit ethernet connections; I have no infiniband or other high-speed
interconnect. The problem I describe below is the same if I use openmpi
1.6.5. My openmpi installation is compiled with Intel icc and ifort. See
the attached result of ompi_info --all for more details on my
installation and runtime parameters, and other diagnostic information
below

My problem is that I noticed that inter-hardware communication hangs in
one of my own programs; I thought this was the fault of my own bad
programming, so I tried some of the example programs that are
distributed with the openmpi source code. In the program "ring_*" using
whichever of the APIs (c, cxx, fortran etc.), I have the same faulty
behavior that I noticed in my own program: if I run the program on a
single hardware node (with multiple processes) it works fine. As soon as
I run the program across hardware nodes, it hangs. Below you will find
an example of the program output and other diagnostic information.

This problem has really frustrated me. Unfortunately I am not
experienced enough with openmpi to get more into the debugging.

Thank you in advance for any help you can give me!

Jed Kaplan

--- DETAILS OF MY PROBLEM ---

-- this run works because it is only on one hardware node --

jkaplan_at_grkapsrv2:~/openmpi_examples > mpirun --prefix /usr/local
--hostfile arvehosts.txt -np 3 ring_c
Process 0 sending 10 to 1, tag 201 (3 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting
Process 2 exiting

-- this run hangs when running over two hardware nodes --

jkaplan_at_grkapsrv2:~/openmpi_examples > mpirun --prefix /usr/local
--hostfile arvehosts.txt -np 4 ring_c
Process 0 sending 10 to 1, tag 201 (4 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
... hangs forever ...
^CKilled by signal 2.

-- here is what my hostfile looks like --

jkaplan_at_grkapsrv2:~/openmpi_examples > cat arvehosts.txt
#host file for ARVE group mac servers

10.0.0.21 slots=3
10.0.0.31 slots=8
10.0.0.41 slots=8
10.0.0.51 slots=8
10.0.0.61 slots=8
10.0.0.71 slots=8

-- results of ifconfig - this looks pretty much the same on all of my
servers, with different ip addresses of course --

jkaplan_at_grkapsrv2:~/openmpi_examples > ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:24:36:f3:dc:fc
        inet6 fe80::224:36ff:fef3:dcfc%en0 prefixlen 64 scopeid 0x4
        inet 128.178.107.85 netmask 0xffffff00 broadcast 128.178.107.255
        media: autoselect (1000baseT <full-duplex>)
        status: active
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:24:36:f3:dc:fa
        inet6 fe80::224:36ff:fef3:dcfa%en1 prefixlen 64 scopeid 0x5
        inet 10.0.0.2 netmask 0xff000000 broadcast 10.255.255.255
        media: autoselect (1000baseT <full-duplex,flow-control>)
        status: active
en2: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:24:36:f5:ba:4e
        inet6 fe80::224:36ff:fef5:ba4e%en2 prefixlen 64 scopeid 0x6
        inet 10.0.0.21 netmask 0xff000000 broadcast 10.255.255.255
        media: autoselect (1000baseT <full-duplex,flow-control>)
        status: active
en3: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:24:36:f5:ba:4f
        inet6 fe80::224:36ff:fef5:ba4f%en3 prefixlen 64 scopeid 0x7
        inet 10.0.0.22 netmask 0xff000000 broadcast 10.255.255.255
        media: autoselect (1000baseT <full-duplex,flow-control>)
        status: active
fw0: flags=8822<BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 4078
        lladdr 04:1e:64:ff:fe:f8:aa:d2
        media: autoselect <full-duplex>
        status: inactive

jkaplan_at_grkapsrv2:~/science_projects/mpitest/openmpi_examples > ompi_info --all
                  Prefix: /usr/local
             Exec_prefix: /usr/local
                  Bindir: /usr/local/bin
                 Sbindir: /usr/local/sbin
                  Libdir: /usr/local/lib
                  Incdir: /usr/local/include
                  Mandir: /usr/local/share/man
               Pkglibdir: /usr/local/lib/openmpi
              Libexecdir: /usr/local/libexec
             Datarootdir: /usr/local/share
                 Datadir: /usr/local/share
              Sysconfdir: /usr/local/etc
          Sharedstatedir: /usr/local/com
           Localstatedir: /usr/local/var
                 Infodir: /usr/local/share/info
              Pkgdatadir: /usr/local/share/openmpi
               Pkglibdir: /usr/local/lib/openmpi
           Pkgincludedir: /usr/local/include/openmpi
 Configured architecture: x86_64-apple-darwin10.8.0
          Configure host: grkapsrv7.epfl.ch
           Configured by: admin
           Configured on: Thu Jul 4 22:48:09 CEST 2013
          Configure host: grkapsrv7.epfl.ch
                Built by: admin
                Built on: Thu Jul 4 23:01:39 CEST 2013
              Built host: grkapsrv7.epfl.ch
              C bindings: yes
            C++ bindings: yes
             Fort mpif.h: yes (single underscore)
            Fort use mpi: yes (full: ignore TKR)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to limitations in the ifort compiler, does not support the following: array subsections, direct passthru (where possible) to underlying Open
                          MPI's C functionality
  Fort mpi_f08 subarrays: no
              C compiler: icc
     C compiler absolute: /Volumes/arve/opt/intel/composer_xe_2011_sp1.11.344/bin/intel64/icc
  C compiler family name: INTEL
      C compiler version: 1210.20120612
             C char size: 1
             C bool size: 1
            C short size: 2
              C int size: 4
             C long size: 8
            C float size: 4
           C double size: 8
          C pointer size: 8
            C char align: 1
            C bool align: 1
             C int align: 4
           C float align: 4
          C double align: 8
            C++ compiler: icpc
   C++ compiler absolute: /Volumes/arve/opt/intel/composer_xe_2011_sp1.11.344/bin/intel64/icpc
           Fort compiler: ifort
       Fort compiler abs: /Volumes/arve/opt/intel/composer_xe_2011_sp1.11.344/bin/intel64/ifort
         Fort ignore TKR: yes (!DEC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: no
      Fort optional args: yes
            Fort BIND(C): yes
            Fort PRIVATE: yes
           Fort ABSTRACT: yes
       Fort ASYNCHRONOUS: yes
          Fort PROCEDURE: yes
 Fort f08 using wrappers: yes
       Fort integer size: 4
       Fort logical size: 4
 Fort logical value true: -1
      Fort have integer1: yes
      Fort have integer2: yes
      Fort have integer4: yes
      Fort have integer8: yes
     Fort have integer16: no
         Fort have real4: yes
         Fort have real8: yes
        Fort have real16: yes
      Fort have complex8: yes
     Fort have complex16: yes
     Fort have complex32: yes
      Fort integer1 size: 1
      Fort integer2 size: 2
      Fort integer4 size: 4
      Fort integer8 size: 8
     Fort integer16 size: -1
          Fort real size: 4
         Fort real4 size: 4
         Fort real8 size: 8
        Fort real16 size: 16
      Fort dbl prec size: 8
          Fort cplx size: 8
      Fort dbl cplx size: 16
         Fort cplx8 size: 8
        Fort cplx16 size: 16
        Fort cplx32 size: 32
      Fort integer align: 1
     Fort integer1 align: 1
     Fort integer2 align: 1
     Fort integer4 align: 1
     Fort integer8 align: 1
    Fort integer16 align: -1
         Fort real align: 1
        Fort real4 align: 1
        Fort real8 align: 1
       Fort real16 align: 1
     Fort dbl prec align: 1
         Fort cplx align: 1
     Fort dbl cplx align: 1
        Fort cplx8 align: 1
       Fort cplx16 align: 1
       Fort cplx32 align: 1
             C profiling: yes
           C++ profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
          C++ exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no)
           Sparse Groups: no
            Build CFLAGS: -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -restrict -Qoption,cpp,--extended_float_types
          Build CXXFLAGS: -O3 -DNDEBUG -finline-functions
           Build FCFLAGS:
           Build LDFLAGS: -export-dynamic
              Build LIBS:
    Wrapper extra CFLAGS:
  Wrapper extra CXXFLAGS:
   Wrapper extra FCFLAGS: -Wl,-commons,use_dylibs
   Wrapper extra LDFLAGS:
      Wrapper extra LIBS: -lm
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
         libltdl support: yes
   Heterogeneous support: no
 mpirun default --prefix: no
         MPI I/O support: yes
       MPI_WTIME support: gettimeofday
     Symbol vis. support: yes
   Host topology support: yes
          MPI extensions:
   FT Checkpoint support: no (checkpoint thread: no)
   C/R Enabled Debugging: no
     VampirTrace support: yes
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
           MCA backtrace: parameter "backtrace" (current value: <none>, data source: default value)
                          Default selection set of components for the backtrace framework (<none> means use all components that can be found)
           MCA backtrace: parameter "backtrace_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the backtrace framework (0 = no verbosity)
           MCA backtrace: parameter "backtrace_execinfo_priority" (current value: <0>, data source: default value)
               MCA event: parameter "event_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level of the event framework
               MCA event: parameter "event" (current value: <none>, data source: default value)
                          Default selection set of components for the event framework (<none> means use all components that can be found)
               MCA event: parameter "event_libevent2019_priority" (current value: <0>, data source: default value)
               MCA hwloc: parameter "hwloc_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level of the hwloc framework
               MCA hwloc: parameter "hwloc_base_mem_alloc_policy" (current value: <none>, data source: default value)
                          General memory allocations placement policy (this is not memory binding). "none" means that no memory policy is applied. "local_only" means that a process' memory allocations will be
                          restricted to its local NUMA node. If using direct launch, this policy will not be in effect until after MPI_INIT. Note that operating system paging policies are unaffected by this
                          setting. For example, if "local_only" is used and local NUMA node memory is exhausted, a new memory allocation may cause paging.
               MCA hwloc: parameter "hwloc_base_mem_bind_failure_action" (current value: <warn>, data source: default value)
                          What Open MPI will do if it explicitly tries to bind memory to a specific NUMA location, and fails. Note that this is a different case than the general allocation policy described by
                          hwloc_base_alloc_policy. A value of "silent" means that Open MPI will proceed without comment. A value of "warn" means that Open MPI will warn the first time this happens, but allow
                          the job to continue (possibly with degraded performance). A value of "error" means that Open MPI will abort the job if this happens.
               MCA hwloc: parameter "hwloc_base_binding_policy" (current value: <none>, data source: default value)
                          Policy for binding processes [none (default) | hwthread | core | l1cache | l2cache | l3cache | socket | numa | board] (supported qualifiers: overload-allowed,if-supported)
               MCA hwloc: parameter "hwloc_base_bind_to_core" (current value: <0>, data source: default value)
                          Bind processes to cores
               MCA hwloc: parameter "hwloc_base_bind_to_socket" (current value: <0>, data source: default value)
                          Bind processes to sockets
               MCA hwloc: parameter "hwloc_base_report_bindings" (current value: <0>, data source: default value)
                          Report bindings to stderr
               MCA hwloc: parameter "hwloc_base_slot_list" (current value: <none>, data source: default value)
                          List of processor IDs to bind processes to [default=NULL]
               MCA hwloc: parameter "hwloc_base_cpu_set" (current value: <none>, data source: default value)
                          Comma-separated list of ranges specifying logical cpus allocated to this job [default: none]
               MCA hwloc: parameter "hwloc" (current value: <none>, data source: default value)
                          Default selection set of components for the hwloc framework (<none> means use all components that can be found)
               MCA hwloc: parameter "hwloc_hwloc152_priority" (current value: <0>, data source: default value)
               MCA hwloc: parameter "hwloc_base_use_hwthreads_as_cpus" (current value: <0>, data source: default value)
                          Use hardware threads as independent cpus
                 MCA mca: parameter "mca_param_files" (current value: </Volumes/arve/users/jkaplan/.openmpi/mca-params.conf:/usr/local/etc/openmpi-mca-params.conf>, data source: default value)
                          Path for MCA configuration files containing default parameter values
                 MCA mca: parameter "mca_base_param_file_prefix" (current value: <none>, data source: default value)
                          Aggregate MCA parameter file sets
                 MCA mca: parameter "mca_base_param_file_path" (current value: </usr/local/share/openmpi/amca-param-sets:/Volumes/arve/users/jkaplan/science_projects/mpitest/openmpi_examples>, data source:
                          default value)
                          Aggregate MCA parameter Search path
                 MCA mca: parameter "mca_base_param_file_path_force" (current value: <none>, data source: default value)
                          Forced Aggregate MCA parameter Search path
                 MCA mca: parameter "mca_component_path" (current value: </usr/local/lib/openmpi:/Volumes/arve/users/jkaplan/.openmpi/components>, data source: default value)
                          Path where to look for Open MPI and ORTE components
                 MCA mca: parameter "mca_component_show_load_errors" (current value: <1>, data source: default value)
                          Whether to show errors for components that failed to load or not
                 MCA mca: parameter "mca_component_disable_dlopen" (current value: <0>, data source: default value)
                          Whether to attempt to disable opening dynamic components or not
                 MCA mca: parameter "mca_verbose" (current value: <stderr>, data source: default value)
                          Specifies where the default error output stream goes (this is separate from distinct help messages). Accepts a comma-delimited list of: stderr, stdout, syslog,
                          syslogpri:<notice|info|debug>, syslogid:<str> (where str is the prefix string for all syslog notices), file[:filename] (if filename is not specified, a default filename is used),
                          fileappend (if not specified, the file is opened for truncation), level[:N] (if specified, integer verbose level; otherwise, 0 is implied)
          MCA memchecker: parameter "memchecker" (current value: <none>, data source: default value)
                          Default selection set of components for the memchecker framework (<none> means use all components that can be found)
              MCA memory: parameter "memory" (current value: <none>, data source: default value)
                          Default selection set of components for the memory framework (<none> means use all components that can be found)
              MCA memory: parameter "memory_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the memory framework (0 = no verbosity)
              MCA memcpy: parameter "memcpy" (current value: <none>, data source: default value)
                          Default selection set of components for the memcpy framework (<none> means use all components that can be found)
              MCA memcpy: parameter "memcpy_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the memcpy framework (0 = no verbosity)
                MCA opal: parameter "opal_net_private_ipv4" (current value: <10.0.0.0/8;172.16.0.0/12;192.168.0.0/16;169.254.0.0/16>, data source: default value)
                          Semicolon-delimited list of CIDR notation entries specifying what networks are considered "private" (default value based on RFC1918 and RFC3330)
                MCA opal: parameter "opal_signal" (current value: <6,10,8,11>, data source: default value)
                          Comma-delimited list of integer signal numbers to Open MPI to attempt to intercept. Upon receipt of the intercepted signal, Open MPI will display a stack trace and abort. Open MPI
                          will *not* replace signals if handlers are already installed by the time MPI_INIT is invoked. Optionally append ":complain" to any signal number in the comma-delimited list to make
                          Open MPI complain if it detects another signal handler (and therefore does not insert its own).
                MCA opal: parameter "opal_set_max_sys_limits" (current value: <0>, data source: default value)
                          Set to non-zero to automatically set any system-imposed limits to the maximum allowed
                MCA opal: parameter "opal_event_include" (current value: <select>, data source: default value)
                          Comma-delimited list of libevent subsystems to use (poll, select -- available on your platform)
                MCA opal: parameter "opal_if_do_not_resolve" (current value: <0>, data source: default value)
                          If nonzero, do not attempt to resolve interfaces
                MCA opal: parameter "opal_if_retain_loopback" (current value: <0>, data source: default value)
                          If nonzero, retain loopback interfaces
               MCA shmem: parameter "shmem_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level of the shmem framework
               MCA shmem: parameter "shmem" (current value: <none>, data source: default value)
                          Default selection set of components for the shmem framework (<none> means use all components that can be found)
               MCA shmem: parameter "shmem_mmap_enable_nfs_warning" (current value: <1>, data source: default value)
                          Enable the warning emitted when Open MPI detects that its shared memory backing file is located on a network filesystem (1 = enabled, 0 = disabled).
               MCA shmem: parameter "shmem_mmap_priority" (current value: <50>, data source: default value)
                          Priority of the mmap shmem component
               MCA shmem: parameter "shmem_mmap_relocate_backing_file" (current value: <0>, data source: default value)
                          Whether to change the default placement of backing files or not (Negative = try to relocate backing files to an area rooted at the path specified by
                          shmem_mmap_opal_shmem_mmap_backing_file_base_dir, but continue with the default path if the relocation fails, 0 = do not relocate, Positive = same as the negative option, but will fail
                          if the relocation fails.
               MCA shmem: parameter "shmem_mmap_backing_file_base_dir" (current value: </dev/shm>, data source: default value)
                          Specifies where backing files will be created when shmem_mmap_relocate_backing_file is in use.
               MCA shmem: parameter "shmem_posix_priority" (current value: <40>, data source: default value)
                          Priority of the posix shmem component
               MCA shmem: parameter "shmem_sysv_priority" (current value: <30>, data source: default value)
                          Priority of the sysv shmem component
               MCA timer: parameter "timer" (current value: <none>, data source: default value)
                          Default selection set of components for the timer framework (<none> means use all components that can be found)
               MCA timer: parameter "timer_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the timer framework (0 = no verbosity)
               MCA timer: parameter "timer_darwin_priority" (current value: <0>, data source: default value)
                  MCA db: parameter "db" (current value: <none>, data source: default value)
                          Default selection set of components for the db framework (<none> means use all components that can be found)
                  MCA db: parameter "db_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the db framework (0 = no verbosity)
                  MCA db: parameter "db_hash_priority" (current value: <0>, data source: default value)
              MCA errmgr: parameter "errmgr" (current value: <none>, data source: default value)
                          Default selection set of components for the errmgr framework (<none> means use all components that can be found)
              MCA errmgr: parameter "errmgr_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the errmgr framework (0 = no verbosity)
              MCA errmgr: parameter "errmgr_default_app_priority" (current value: <1000>, data source: default value)
                          Priority of the default_app errmgr component
              MCA errmgr: parameter "errmgr_default_hnp_priority" (current value: <1000>, data source: default value)
                          Priority of the default_hnp errmgr component
              MCA errmgr: parameter "errmgr_default_orted_priority" (current value: <1000>, data source: default value)
                          Priority of the default_orted errmgr component
                 MCA ess: parameter "ess" (current value: <none>, data source: default value)
                          Default selection set of components for the ess framework (<none> means use all components that can be found)
                 MCA ess: parameter "ess_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the ess framework (0 = no verbosity)
                 MCA ess: parameter "ess_env_priority" (current value: <0>, data source: default value)
                 MCA ess: parameter "ess_hnp_priority" (current value: <0>, data source: default value)
                 MCA ess: parameter "ess_singleton_priority" (current value: <0>, data source: default value)
                 MCA ess: parameter "ess_slurm_priority" (current value: <0>, data source: default value)
                 MCA ess: parameter "ess_tool_priority" (current value: <0>, data source: default value)
               MCA filem: parameter "filem" (current value: <none>, data source: default value)
                          Which Filem component to use (empty = auto-select)
               MCA filem: parameter "filem_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the filem framework (0 = no verbosity)
             MCA grpcomm: parameter "grpcomm" (current value: <none>, data source: default value)
                          Default selection set of components for the grpcomm framework (<none> means use all components that can be found)
             MCA grpcomm: parameter "grpcomm_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the grpcomm framework (0 = no verbosity)
             MCA grpcomm: parameter "grpcomm_bad_priority" (current value: <0>, data source: default value)
                 MCA iof: parameter "iof_base_output_limit" (current value: <-1>, data source: default value)
                          Maximum backlog of output messages [default: unlimited]
                 MCA iof: parameter "iof_base_input_files" (current value: <none>, data source: default value)
                          Comma-separated list of input files to be read and sent to stdin of procs (default: NULL)
                 MCA iof: parameter "iof" (current value: <none>, data source: default value)
                          Default selection set of components for the iof framework (<none> means use all components that can be found)
                 MCA iof: parameter "iof_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the iof framework (0 = no verbosity)
                 MCA iof: parameter "iof_hnp_priority" (current value: <0>, data source: default value)
                 MCA iof: parameter "iof_orted_priority" (current value: <0>, data source: default value)
                 MCA iof: parameter "iof_tool_priority" (current value: <0>, data source: default value)
                MCA odls: parameter "odls_base_sigkill_timeout" (current value: <1>, data source: default value)
                          Time to wait for a process to die after issuing a kill signal to it
                MCA odls: parameter "odls" (current value: <none>, data source: default value)
                          Default selection set of components for the odls framework (<none> means use all components that can be found)
                MCA odls: parameter "odls_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the odls framework (0 = no verbosity)
                MCA odls: parameter "odls_default_priority" (current value: <0>, data source: default value)
                 MCA oob: parameter "oob" (current value: <none>, data source: default value)
                          Default selection set of components for the oob framework (<none> means use all components that can be found)
                 MCA oob: parameter "oob_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the oob framework (0 = no verbosity)
                 MCA oob: parameter "oob_tcp_verbose" (current value: <0>, data source: default value)
                          Verbose level for the OOB tcp component
                 MCA oob: parameter "oob_tcp_peer_limit" (current value: <-1>, data source: default value)
                          Maximum number of peer connections to simultaneously maintain (-1 = infinite)
                 MCA oob: parameter "oob_tcp_peer_retries" (current value: <60>, data source: default value)
                          Number of times to try shutting down a connection before giving up
                 MCA oob: parameter "oob_tcp_debug" (current value: <0>, data source: default value)
                          Enable (1) / disable (0) debugging output for this component
                 MCA oob: parameter "oob_tcp_sndbuf" (current value: <131072>, data source: default value)
                          TCP socket send buffering size (in bytes)
                 MCA oob: parameter "oob_tcp_rcvbuf" (current value: <131072>, data source: default value)
                          TCP socket receive buffering size (in bytes)
                 MCA oob: parameter "oob_tcp_if_include" (current value: <none>, data source: default value)
                          Comma-delimited list of devices and/or CIDR notation of networks to use for Open MPI bootstrap communication (e.g., "eth0,192.168.0.0/16"). Mutually exclusive with oob_tcp_if_exclude.
                 MCA oob: parameter "oob_tcp_if_exclude" (current value: <none>, data source: default value)
                          Comma-delimited list of devices and/or CIDR notation of networks to NOT use for Open MPI bootstrap communication -- all devices not matching these specifications will be used (e.g.,
                          "eth0,192.168.0.0/16"). If set to a non-default value, it is mutually exclusive with oob_tcp_if_include.
                 MCA oob: parameter "oob_tcp_connect_sleep" (current value: <1>, data source: default value)
                          Enable (1) / disable (0) random sleep for connection wireup.
                 MCA oob: parameter "oob_tcp_listen_mode" (current value: <event>, data source: default value)
                          Mode for HNP to accept incoming connections: event, listen_thread.
                 MCA oob: parameter "oob_tcp_listen_thread_max_queue" (current value: <10>, data source: default value)
                          High water mark for queued accepted socket list size. Used only when listen_mode is listen_thread.
                 MCA oob: parameter "oob_tcp_listen_thread_wait_time" (current value: <10>, data source: default value)
                          Time in milliseconds to wait before actively checking for new connections when listen_mode is listen_thread.
                 MCA oob: parameter "oob_tcp_static_ports" (current value: <none>, data source: default value)
                          Static ports for daemons and procs (IPv4)
                 MCA oob: parameter "oob_tcp_dynamic_ports" (current value: <none>, data source: default value)
                          Range of ports to be dynamically used by daemons and procs (IPv4)
                 MCA oob: parameter "oob_tcp_disable_family" (current value: <0>, data source: default value)
                          Disable IPv4 (4) or IPv6 (6)
                 MCA oob: parameter "oob_tcp_priority" (current value: <0>, data source: default value)
                MCA orte: parameter "orte_base_help_aggregate" (current value: <1>, data source: default value)
                          If orte_base_help_aggregate is true, duplicate help messages will be aggregated rather than displayed individually. This can be helpful for parallel jobs that experience multiple
                          identical failures; rather than print out the same help/failure message N times, display it once with a count of how many processes sent the same message.
                MCA orte: parameter "orte_tmpdir_base" (current value: <none>, data source: default value)
                          Base of the session directory tree to be used by all processes
                MCA orte: parameter "orte_local_tmpdir_base" (current value: <none>, data source: default value)
                          Base of the session directory tree to be used by orterun/mpirun
                MCA orte: parameter "orte_remote_tmpdir_base" (current value: <none>, data source: default value)
                          Base of the session directory tree on remote nodes, if required to be different from head node
                MCA orte: parameter "orte_no_session_dirs" (current value: <none>, data source: default value)
                          Prohibited locations for session directories (multiple locations separated by ',', default=NULL)
                MCA orte: parameter "orte_create_session_dirs" (current value: <1>, data source: default value)
                          Create session directories
                MCA orte: parameter "orte_execute_quiet" (current value: <0>, data source: default value)
                          Do not output error and help messages
                MCA orte: parameter "orte_report_silent_errors" (current value: <0>, data source: default value)
                          Report all errors, including silent ones
                MCA orte: parameter "orte_debug" (current value: <0>, data source: default value)
                          Top-level ORTE debug switch (default verbosity: 1)
                MCA orte: parameter "orte_debug_verbose" (current value: <-1>, data source: default value)
                          Verbosity level for ORTE debug messages (default: 1)
                MCA orte: parameter "orte_debug_daemons" (current value: <0>, data source: default value)
                          Whether to debug the ORTE daemons or not
                MCA orte: parameter "orte_debug_daemons_file" (current value: <0>, data source: default value)
                          Whether want stdout/stderr of daemons to go to a file or not
                MCA orte: parameter "orte_leave_session_attached" (current value: <0>, data source: default value)
                          Whether applications and/or daemons should leave their sessions attached so that any output can be received - this allows X forwarding without all the attendant debugging output
                MCA orte: parameter "orte_output_debugger_proctable" (current value: <0>, data source: default value)
                          Whether or not to output the debugger proctable after launch (default: false)
                MCA orte: parameter "orte_debugger_test_daemon" (current value: <none>, data source: default value)
                          Name of the executable to be used to simulate a debugger colaunch (relative or absolute path)
                MCA orte: parameter "orte_debugger_test_attach" (current value: <0>, data source: default value)
                          Test debugger colaunch after debugger attachment
                MCA orte: parameter "orte_debugger_check_rate" (current value: <0>, data source: default value)
                          Set rate (in secs) for auto-detect of debugger attachment (0 => do not check)
                MCA orte: parameter "orte_do_not_launch" (current value: <0>, data source: default value)
                          Perform all necessary operations to prepare to launch the application, but do not actually launch it
                MCA orte: parameter "orte_daemon_spin" (current value: <0>, data source: default value)
                          Have any orteds spin until we can connect a debugger to them
                MCA orte: parameter "orte_daemon_fail" (current value: <-1>, data source: default value)
                          Have the specified orted fail after init for debugging purposes
                MCA orte: parameter "orte_daemon_fail_delay" (current value: <0>, data source: default value)
                          Have the specified orted fail after specified number of seconds (default: 0 => no delay)
                MCA orte: parameter "orte_startup_timeout" (current value: <0>, data source: default value)
                          Seconds/daemon to wait for startup before declaring failed_to_start (default: 0 => do not check)
                MCA orte: parameter "orte_timing" (current value: <0>, data source: default value)
                          Request that critical timing loops be measured
                MCA orte: parameter "orte_timing_details" (current value: <0>, data source: default value)
                          Request that detailed timing data by reported
                MCA orte: parameter "orte_timing_file" (current value: <none>, data source: default value)
                          Name of the file where timing data is to be written (relative or absolute path)
                MCA orte: parameter "orte_base_user_debugger" (current value: <totalview @mpirun@ -a @mpirun_args@ : ddt -n @np@ -start @executable@ @executable_argv@ @single_app@ : fxp @mpirun@ -a
                          @mpirun_args@>, data source: default value)
                          Sequence of user-level debuggers to search for in orterun
                MCA orte: parameter "orte_abort_timeout" (current value: <1>, data source: default value)
                          Max time to wait [in secs] before aborting an ORTE operation (default: 1sec)
                MCA orte: parameter "orte_timeout_step" (current value: <1000>, data source: default value)
                          Time to wait [in usecs/proc] before aborting an ORTE operation (default: 1000 usec/proc)
                MCA orte: parameter "orte_default_hostfile" (current value: <none>, data source: default value)
                          Name of the default hostfile (relative or absolute path, "none" to ignore environmental or default MCA param setting)
                MCA orte: parameter "orte_node_regex" (current value: <none>, data source: default value)
                          Regular expression defining nodes in the system
                MCA orte: parameter "orte_keep_fqdn_hostnames" (current value: <0>, data source: default value)
                          Whether or not to keep FQDN hostnames [default: no]
                MCA orte: parameter "orte_retain_aliases" (current value: <0>, data source: default value)
                          Whether or not to keep aliases for host names [default: no]
                MCA orte: parameter "orte_hostname_alias_index" (current value: <1>, data source: default value)
                          If hostname aliases are being retained, which one to use for the debugger proc table [default: 1st alias]
                MCA orte: parameter "orte_tag_output" (current value: <0>, data source: default value)
                          Tag all output with [job,rank] (default: false)
                MCA orte: parameter "orte_xml_output" (current value: <0>, data source: default value)
                          Display all output in XML format (default: false)
                MCA orte: parameter "orte_xml_file" (current value: <none>, data source: default value)
                          Provide all output in XML format to the specified file
                MCA orte: parameter "orte_timestamp_output" (current value: <0>, data source: default value)
                          Timestamp all application process output (default: false)
                MCA orte: parameter "orte_output_filename" (current value: <none>, data source: default value)
                          Redirect output from application processes into filename.rank [default: NULL]
                MCA orte: parameter "orte_show_resolved_nodenames" (current value: <0>, data source: default value)
                          Display any node names that are resolved to a different name (default: false)
                MCA orte: parameter "orte_hetero_apps" (current value: <0>, data source: default value)
                          Indicates that multiple app_contexts are being provided that are a mix of 32/64 bit binaries (default: false)
                MCA orte: parameter "orte_hetero_nodes" (current value: <0>, data source: default value)
                          Nodes in cluster may differ in topology, so send the topology back from each node [Default = false]
                MCA orte: parameter "orte_launch_agent" (current value: <orted>, data source: default value)
                          Command used to start processes on remote nodes (default: orted)
                MCA orte: parameter "orte_fork_agent" (current value: <none>, data source: default value)
                          Command used to fork processes on remote nodes (default: NULL)
                MCA orte: parameter "orte_allocation_required" (current value: <0>, data source: default value)
                          Whether or not an allocation by a resource manager is required [default: no]
                MCA orte: parameter "orte_xterm" (current value: <none>, data source: default value)
                          Create a new xterm window and display output from the specified ranks there [default: none]
                MCA orte: parameter "orte_map_stddiag_to_stderr" (current value: <0>, data source: default value)
                          Map output from opal_output to stderr of the local process [default: no]
                MCA orte: parameter "orte_forward_job_control" (current value: <0>, data source: default value)
                          Forward SIGTSTP (after converting to SIGSTOP) and SIGCONT signals to the application procs [default: no]
                MCA orte: parameter "orte_report_launch_progress" (current value: <0>, data source: default value)
                          Output a brief periodic report on launch progress [default: no]
                MCA orte: parameter "orte_report_events" (current value: <none>, data source: default value)
                          URI to which events are to be reported (default: NULL)
                MCA orte: parameter "orte_enable_recovery" (current value: <0>, data source: default value)
                          Enable recovery from process failure [Default = disabled]
                MCA orte: parameter "orte_max_restarts" (current value: <0>, data source: default value)
                          Max number of times to restart a failed process
                MCA orte: parameter "orte_abort_on_non_zero_status" (current value: <1>, data source: default value)
                          Abort the job if any process returns a non-zero exit status - no restart in such cases
                MCA orte: parameter "orte_report_child_jobs_separately" (current value: <0>, data source: default value)
                          Return the exit status of the primary job only
                MCA orte: parameter "orte_child_time_to_exit" (current value: <2147483647>, data source: default value)
                          Max time a spawned child job is allowed to run after the primary job has terminated (seconds)
                MCA orte: parameter "orte_stat_history_size" (current value: <1>, data source: default value)
                          Number of stat samples to keep
                MCA orte: parameter "orte_forward_envars" (current value: <none>, data source: default value)
                          Comma-delimited environmental variables to forward, can include value to set
                MCA orte: parameter "orte_preload_binaries" (current value: <0>, data source: default value)
                          Preload the binaries on remote machines before starting remote proceses
                MCA orte: parameter "orte_max_vm_size" (current value: <-1>, data source: default value)
                          Maximum size of virtual machine - used to subdivide allocation
                MCA orte: parameter "orte_rsh_agent" (current value: <ssh : rsh>, data source: default value, deprecated, synonym of: plm_rsh_agent)
                          The command used to launch executables on remote nodes (typically either "ssh" or "rsh")
                MCA orte: parameter "orte_assume_same_shell" (current value: <1>, data source: default value, synonyms: plm_rsh_assume_same_shell)
                          If set to 1, assume that the shell on the remote node is the same as the shell on the local node. Otherwise, probe for what the remote shell [default: 1]
                MCA orte: parameter "orte_rankfile" (current value: <none>, data source: default value, synonym of: rmaps_rank_file_path)
                          Name of the rankfile to be used for mapping processes (relative or absolute path)
                 MCA plm: parameter "plm" (current value: <none>, data source: default value)
                          Default selection set of components for the plm framework (<none> means use all components that can be found)
                 MCA plm: parameter "plm_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the plm framework (0 = no verbosity)
                 MCA plm: parameter "plm_rsh_num_concurrent" (current value: <128>, data source: default value)
                          How many plm_rsh_agent instances to invoke concurrently (must be > 0)
                 MCA plm: parameter "plm_rsh_force_rsh" (current value: <0>, data source: default value)
                          Force the launcher to always use rsh
                 MCA plm: parameter "plm_rsh_disable_qrsh" (current value: <0>, data source: default value)
                          Disable the launcher to use qrsh when under the Grid Engine parallel environment
                 MCA plm: parameter "plm_rsh_daemonize_qrsh" (current value: <0>, data source: default value)
                          Daemonize the orted under the Grid Engine parallel environment
                 MCA plm: parameter "plm_rsh_disable_llspawn" (current value: <0>, data source: default value)
                          Disable the use of llspawn when under the LoadLeveler environment
                 MCA plm: parameter "plm_rsh_daemonize_llspawn" (current value: <0>, data source: default value)
                          Daemonize the orted when under the LoadLeveler environment
                 MCA plm: parameter "plm_rsh_priority" (current value: <10>, data source: default value)
                          Priority of the rsh plm component
                 MCA plm: parameter "plm_rsh_delay" (current value: <none>, data source: default value)
                          Delay between invocations of the remote agent (sec[:usec])
                 MCA plm: parameter "plm_rsh_no_tree_spawn" (current value: <0>, data source: default value)
                          If set to 1, do not launch via a tree-based topology
                 MCA plm: parameter "plm_rsh_agent" (current value: <ssh : rsh>, data source: default value, synonyms: pls_rsh_agent, orte_rsh_agent)
                          The command used to launch executables on remote nodes (typically either "ssh" or "rsh")
                 MCA plm: parameter "plm_rsh_assume_same_shell" (current value: <1>, data source: default value, deprecated, synonym of: orte_assume_same_shell)
                          If set to 1, assume that the shell on the remote node is the same as the shell on the local node. Otherwise, probe for what the remote shell [default: 1]
                 MCA plm: parameter "plm_rsh_pass_environ_mca_params" (current value: <1>, data source: default value)
                          If set to 0, do not include mca params from the environment on the orted cmd line
                 MCA plm: parameter "plm_slurm_args" (current value: <none>, data source: default value)
                          Custom arguments to srun
                 MCA plm: parameter "plm_slurm_priority" (current value: <0>, data source: default value)
                 MCA ras: parameter "ras_base_display_alloc" (current value: <0>, data source: default value)
                          Whether to display the allocation after it is determined
                 MCA ras: parameter "ras_base_display_devel_alloc" (current value: <0>, data source: default value)
                          Whether to display a developer-detail allocation after it is determined
                 MCA ras: parameter "ras" (current value: <none>, data source: default value)
                          Default selection set of components for the ras framework (<none> means use all components that can be found)
                 MCA ras: parameter "ras_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the ras framework (0 = no verbosity)
                 MCA ras: parameter "ras_simulator_slots" (current value: <1>, data source: default value)
                          Comma-separated list of number of slots on each node to simulate
                 MCA ras: parameter "ras_simulator_max_slots" (current value: <0>, data source: default value)
                          Comma-separated list of number of max slots on each node to simulate
                 MCA ras: parameter "ras_simulator_num_nodes" (current value: <none>, data source: default value)
                          Comma-separated list of number of nodes to simulate for each topology
                 MCA ras: parameter "ras_simulator_topo_files" (current value: <none>, data source: default value)
                          Comma-separated list of files containing xml topology descriptions for simulated nodes
                 MCA ras: parameter "ras_simulator_have_cpubind" (current value: <1>, data source: default value)
                          Topology supports binding to cpus
                 MCA ras: parameter "ras_simulator_have_membind" (current value: <1>, data source: default value)
                          Topology supports binding to memory
                 MCA ras: parameter "ras_simulator_priority" (current value: <0>, data source: default value)
                 MCA ras: parameter "ras_slurm_priority" (current value: <75>, data source: default value)
                          Priority of the slurm ras component
               MCA rmaps: parameter "rmaps_base_mapping_policy" (current value: <none>, data source: default value, synonyms: rmaps_base_schedule_policy)
                          Mapping Policy [slot (default) | hwthread | core | l1cache | l2cache | l3cache | socket | numa | board | node | seq], with allowed modifiers :SPAN,OVERSUBSCRIBE,NOOVERSUBSCRIBE
               MCA rmaps: parameter "rmaps_base_schedule_policy" (current value: <none>, data source: default value, deprecated, synonym of: rmaps_base_mapping_policy)
                          Mapping Policy [slot (default) | hwthread | core | l1cache | l2cache | l3cache | socket | numa | board | node | seq], with allowed modifiers :SPAN,OVERSUBSCRIBE,NOOVERSUBSCRIBE
               MCA rmaps: parameter "rmaps_base_ranking_policy" (current value: <none>, data source: default value)
                          Ranking Policy [slot (default) | hwthread | core | l1cache | l2cache | l3cache | socket | numa | board | node], with modifier :SPAN or :FILL
               MCA rmaps: parameter "rmaps_base_byslot" (current value: <0>, data source: default value)
                          Whether to map and rank processes round-robin by slot
               MCA rmaps: parameter "rmaps_base_bynode" (current value: <0>, data source: default value)
                          Whether to map and rank processes round-robin by node
               MCA rmaps: parameter "rmaps_base_no_schedule_local" (current value: <0>, data source: default value)
                          If false, allow scheduling MPI applications on the same node as mpirun (default). If true, do not schedule any MPI applications on the same node as mpirun
               MCA rmaps: parameter "rmaps_base_no_oversubscribe" (current value: <0>, data source: default value)
                          If true, then do not allow oversubscription of nodes - mpirun will return an error if there aren't enough nodes to launch all processes without oversubscribing
               MCA rmaps: parameter "rmaps_base_oversubscribe" (current value: <0>, data source: default value)
                          If true, then =allow oversubscription of nodes
               MCA rmaps: parameter "rmaps_base_display_map" (current value: <0>, data source: default value)
                          Whether to display the process map after it is computed
               MCA rmaps: parameter "rmaps_base_display_devel_map" (current value: <0>, data source: default value)
                          Whether to display a developer-detail process map after it is computed
               MCA rmaps: parameter "rmaps_base_display_topo_with_map" (current value: <0>, data source: default value)
                          Whether to display the topology with the map
               MCA rmaps: parameter "rmaps_base_display_diffable_map" (current value: <0>, data source: default value)
                          Whether to display a diffable process map after it is computed
               MCA rmaps: parameter "rmaps" (current value: <none>, data source: default value)
                          Default selection set of components for the rmaps framework (<none> means use all components that can be found)
               MCA rmaps: parameter "rmaps_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the rmaps framework (0 = no verbosity)
               MCA rmaps: parameter "rmaps_lama_priority" (current value: <0>, data source: default value)
                          Priority of the LAMA rmaps component
               MCA rmaps: parameter "rmaps_lama_timing" (current value: <0>, data source: default value)
                          Enable timing information. [Default = disabled]
               MCA rmaps: parameter "rmaps_lama_map" (current value: <none>, data source: default value)
                          LAMA Map: Process layout iteration ordering (See documentation)
               MCA rmaps: parameter "rmaps_lama_bind" (current value: <none>, data source: default value)
                          LAMA Bind: Bind to the specified number of resources (See documentation)
               MCA rmaps: parameter "rmaps_lama_mppr" (current value: <none>, data source: default value)
                          LAMA MPPR: Maximum number of the specified resources available (See documentation)
               MCA rmaps: parameter "rmaps_lama_ordering" (current value: <none>, data source: default value)
                          LAMA Ordering: Ordering (s) sequential, (n) natural - Default: n (See documentation)
               MCA rmaps: parameter "rmaps_ppr_pernode" (current value: <0>, data source: default value, synonyms: rmaps_base_pernode)
                          Launch one ppn as directed
               MCA rmaps: parameter "rmaps_base_pernode" (current value: <0>, data source: default value, synonym of: rmaps_ppr_pernode)
                          Launch one ppn as directed
               MCA rmaps: parameter "rmaps_ppr_n_pernode" (current value: <0>, data source: default value, synonyms: rmaps_base_n_pernode)
                          Launch n procs/node
               MCA rmaps: parameter "rmaps_base_n_pernode" (current value: <0>, data source: default value, synonym of: rmaps_ppr_n_pernode)
                          Launch n procs/node
               MCA rmaps: parameter "rmaps_ppr_n_persocket" (current value: <0>, data source: default value, synonyms: rmaps_base_n_persocket)
                          Launch n procs/socket
               MCA rmaps: parameter "rmaps_base_n_persocket" (current value: <0>, data source: default value, synonym of: rmaps_ppr_n_persocket)
                          Launch n procs/socket
               MCA rmaps: parameter "rmaps_ppr_pattern" (current value: <none>, data source: default value)
                          Comma-separated list of number of processes on a given resource type [default: none]
               MCA rmaps: parameter "rmaps_ppr_priority" (current value: <0>, data source: default value)
               MCA rmaps: parameter "rmaps_rank_file_priority" (current value: <0>, data source: default value)
                          Priority of the rank_file rmaps component
               MCA rmaps: parameter "rmaps_rank_file_path" (current value: <none>, data source: default value, synonyms: orte_rankfile)
                          Name of the rankfile to be used for mapping processes (relative or absolute path)
               MCA rmaps: parameter "rmaps_resilient_fault_grp_file" (current value: <none>, data source: default value)
                          Filename that contains a description of fault groups for this system
               MCA rmaps: parameter "rmaps_resilient_priority" (current value: <40>, data source: default value)
                          Priority of the resilient rmaps component
               MCA rmaps: parameter "rmaps_round_robin_priority" (current value: <10>, data source: default value)
                          Priority of the rr rmaps component
               MCA rmaps: parameter "rmaps_seq_priority" (current value: <60>, data source: default value)
                          Priority of the seq rmaps component
                 MCA rml: parameter "rml_wrapper" (current value: <none>, data source: default value)
                          Use a Wrapper component around the selected RML component
                 MCA rml: parameter "rml" (current value: <none>, data source: default value)
                          Default selection set of components for the rml framework (<none> means use all components that can be found)
                 MCA rml: parameter "rml_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the rml framework (0 = no verbosity)
                 MCA rml: parameter "rml_oob_priority" (current value: <0>, data source: default value)
              MCA routed: parameter "routed" (current value: <none>, data source: default value)
                          Default selection set of components for the routed framework (<none> means use all components that can be found)
              MCA routed: parameter "routed_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the routed framework (0 = no verbosity)
              MCA routed: parameter "routed_binomial_priority" (current value: <0>, data source: default value)
              MCA routed: parameter "routed_debruijn_priority" (current value: <0>, data source: default value)
              MCA routed: parameter "routed_direct_priority" (current value: <0>, data source: default value)
              MCA routed: parameter "routed_radix_priority" (current value: <0>, data source: default value)
               MCA state: parameter "state" (current value: <none>, data source: default value)
                          Default selection set of components for the state framework (<none> means use all components that can be found)
               MCA state: parameter "state_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the state framework (0 = no verbosity)
               MCA state: parameter "state_app_priority" (current value: <0>, data source: default value)
               MCA state: parameter "state_hnp_priority" (current value: <0>, data source: default value)
               MCA state: parameter "state_novm_select" (current value: <0>, data source: default value)
                          Use this component
               MCA state: parameter "state_novm_priority" (current value: <0>, data source: default value)
               MCA state: parameter "state_orted_priority" (current value: <0>, data source: default value)
           MCA allocator: parameter "allocator" (current value: <none>, data source: default value)
                          Default selection set of components for the allocator framework (<none> means use all components that can be found)
           MCA allocator: parameter "allocator_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the allocator framework (0 = no verbosity)
           MCA allocator: parameter "allocator_basic_priority" (current value: <0>, data source: default value)
           MCA allocator: parameter "allocator_bucket_num_buckets" (current value: <30>, data source: default value)
           MCA allocator: parameter "allocator_bucket_priority" (current value: <0>, data source: default value)
                 MCA bml: parameter "bml" (current value: <none>, data source: default value)
                          Default selection set of components for the bml framework (<none> means use all components that can be found)
                 MCA bml: parameter "bml_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the bml framework (0 = no verbosity)
                 MCA bml: parameter "bml_r2_show_unreach_errors" (current value: <1>, data source: default value)
                          Show error message when procs are unreachable
                 MCA bml: parameter "bml_r2_priority" (current value: <0>, data source: default value)
                 MCA btl: parameter "btl_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level of the BTL framework
                 MCA btl: parameter "btl" (current value: <none>, data source: default value)
                          Default selection set of components for the btl framework (<none> means use all components that can be found)
                 MCA btl: parameter "btl_self_free_list_num" (current value: <0>, data source: default value)
                          Number of fragments by default
                 MCA btl: parameter "btl_self_free_list_max" (current value: <-1>, data source: default value)
                          Maximum number of fragments
                 MCA btl: parameter "btl_self_free_list_inc" (current value: <32>, data source: default value)
                          Increment by this number of fragments
                 MCA btl: parameter "btl_self_exclusivity" (current value: <65536>, data source: default value)
                          BTL exclusivity (must be >= 0)
                 MCA btl: parameter "btl_self_flags" (current value: <10>, data source: default value)
                          BTL bit flags (general flags: SEND=1, PUT=2, GET=4, SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by the "dr" PML (ignored by others): ACK=16, CHECKSUM=32,
                          RDMA_COMPLETION=128; flags only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512)
                 MCA btl: parameter "btl_self_rndv_eager_limit" (current value: <131072>, data source: default value)
                          Size (in bytes, including header) of "phase 1" fragment sent for all large messages (must be >= 0 and <= eager_limit)
                 MCA btl: parameter "btl_self_eager_limit" (current value: <131072>, data source: default value)
                          Maximum size (in bytes, including header) of "short" messages (must be >= 1).
                 MCA btl: parameter "btl_self_max_send_size" (current value: <262144>, data source: default value)
                          Maximum size (in bytes) of a single "phase 2" fragment of a long message when using the pipeline protocol (must be >= 1)
                 MCA btl: parameter "btl_self_rdma_pipeline_send_length" (current value: <2147483647>, data source: default value)
                          Length of the "phase 2" portion of a large message (in bytes) when using the pipeline protocol. This part of the message will be split into fragments of size max_send_size and sent
                          using send/receive semantics (must be >= 0; only relevant when the PUT flag is set)
                 MCA btl: parameter "btl_self_rdma_pipeline_frag_size" (current value: <2147483647>, data source: default value)
                          Maximum size (in bytes) of a single "phase 3" fragment from a long message when using the pipeline protocol. These fragments will be sent using RDMA semantics (must be >= 1; only
                          relevant when the PUT flag is set)
                 MCA btl: parameter "btl_self_min_rdma_pipeline_size" (current value: <0>, data source: default value)
                          Messages smaller than this size (in bytes) will not use the RDMA pipeline protocol. Instead, they will be split into fragments of max_send_size and sent using send/receive semantics
                          (must be >=0, and is automatically adjusted up to at least (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag is set)
                 MCA btl: parameter "btl_self_bandwidth" (current value: <100>, data source: default value)
                          Approximate maximum bandwidth of interconnect(0 = auto-detect value at run-time [not supported in all BTL modules], >= 1 = bandwidth in Mbps)
                 MCA btl: parameter "btl_self_latency" (current value: <0>, data source: default value)
                          Approximate latency of interconnect (must be >= 0)
                 MCA btl: parameter "btl_self_priority" (current value: <0>, data source: default value)
                 MCA btl: information "btl_sm_have_knem_support" (value: <0>, data source: default value)
                          Whether this component supports the knem Linux kernel module or not
                 MCA btl: parameter "btl_sm_use_knem" (current value: <0>, data source: default value)
                          Whether knem support is desired or not (negative = try to enable knem support, but continue even if it is not available, 0 = do not enable knem support, positive = try to enable knem
                          support and fail if it is not available)
                 MCA btl: parameter "btl_sm_knem_dma_min" (current value: <0>, data source: default value)
                          Minimum message size (in bytes) to use the knem DMA mode; ignored if knem does not support DMA mode (0 = do not use the knem DMA mode)
                 MCA btl: parameter "btl_sm_knem_max_simultaneous" (current value: <0>, data source: default value)
                          Max number of simultaneous ongoing knem operations to support (0 = do everything synchronously, which probably gives the best large message latency; >0 means to do all operations
                          asynchronously, which supports better overlap for simultaneous large message sends)
                 MCA btl: parameter "btl_sm_use_cma" (current value: <0>, data source: default value)
                          Whether or not to enable CMA
                 MCA btl: parameter "btl_sm_free_list_num" (current value: <8>, data source: default value)
                 MCA btl: parameter "btl_sm_free_list_max" (current value: <-1>, data source: default value)
                 MCA btl: parameter "btl_sm_free_list_inc" (current value: <64>, data source: default value)
                 MCA btl: parameter "btl_sm_max_procs" (current value: <-1>, data source: default value)
                 MCA btl: parameter "btl_sm_mpool" (current value: <sm>, data source: default value)
                 MCA btl: parameter "btl_sm_fifo_size" (current value: <4096>, data source: default value)
                 MCA btl: parameter "btl_sm_num_fifos" (current value: <1>, data source: default value)
                 MCA btl: parameter "btl_sm_fifo_lazy_free" (current value: <120>, data source: default value)
                 MCA btl: parameter "btl_sm_sm_extra_procs" (current value: <0>, data source: default value)
                 MCA btl: parameter "btl_sm_exclusivity" (current value: <65535>, data source: default value)
                          BTL exclusivity (must be >= 0)
                 MCA btl: parameter "btl_sm_flags" (current value: <1>, data source: default value)
                          BTL bit flags (general flags: SEND=1, PUT=2, GET=4, SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by the "dr" PML (ignored by others): ACK=16, CHECKSUM=32,
                          RDMA_COMPLETION=128; flags only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512)
                 MCA btl: parameter "btl_sm_rndv_eager_limit" (current value: <4096>, data source: default value)
                          Size (in bytes, including header) of "phase 1" fragment sent for all large messages (must be >= 0 and <= eager_limit)
                 MCA btl: parameter "btl_sm_eager_limit" (current value: <4096>, data source: default value)
                          Maximum size (in bytes, including header) of "short" messages (must be >= 1).
                 MCA btl: parameter "btl_sm_max_send_size" (current value: <32768>, data source: default value)
                          Maximum size (in bytes) of a single "phase 2" fragment of a long message when using the pipeline protocol (must be >= 1)
                 MCA btl: parameter "btl_sm_bandwidth" (current value: <9000>, data source: default value)
                          Approximate maximum bandwidth of interconnect(0 = auto-detect value at run-time [not supported in all BTL modules], >= 1 = bandwidth in Mbps)
                 MCA btl: parameter "btl_sm_latency" (current value: <1>, data source: default value)
                          Approximate latency of interconnect (must be >= 0)
                 MCA btl: parameter "btl_sm_priority" (current value: <0>, data source: default value)
                 MCA btl: parameter "btl_tcp_links" (current value: <1>, data source: default value)
                 MCA btl: parameter "btl_tcp_if_include" (current value: <none>, data source: default value)
                          Comma-delimited list of devices and/or CIDR notation of networks to use for MPI communication (e.g., "eth0,192.168.0.0/16"). Mutually exclusive with btl_tcp_if_exclude.
                 MCA btl: parameter "btl_tcp_if_exclude" (current value: <127.0.0.1/8,sppp>, data source: default value)
                          Comma-delimited list of devices and/or CIDR notation of networks to NOT use for MPI communication -- all devices not matching these specifications will be used (e.g.,
                          "eth0,192.168.0.0/16"). If set to a non-default value, it is mutually exclusive with btl_tcp_if_include.
                 MCA btl: parameter "btl_tcp_free_list_num" (current value: <8>, data source: default value)
                 MCA btl: parameter "btl_tcp_free_list_max" (current value: <-1>, data source: default value)
                 MCA btl: parameter "btl_tcp_free_list_inc" (current value: <32>, data source: default value)
                 MCA btl: parameter "btl_tcp_sndbuf" (current value: <131072>, data source: default value)
                 MCA btl: parameter "btl_tcp_rcvbuf" (current value: <131072>, data source: default value)
                 MCA btl: parameter "btl_tcp_endpoint_cache" (current value: <30720>, data source: default value)
                          The size of the internal cache for each TCP connection. This cache is used to reduce the number of syscalls, by replacing them with memcpy. Every read will read the expected data plus
                          the amount of the endpoint_cache
                 MCA btl: parameter "btl_tcp_use_nagle" (current value: <0>, data source: default value)
                          Whether to use Nagle's algorithm or not (using Nagle's algorithm may increase short message latency)
                 MCA btl: parameter "btl_tcp_port_min_v4" (current value: <1024>, data source: default value)
                          The minimum port where the TCP BTL will try to bind (default 1024)
                 MCA btl: parameter "btl_tcp_port_range_v4" (current value: <64511>, data source: default value)
                          The number of ports where the TCP BTL will try to bind (default 64511). This parameter together with the port min, define a range of ports where Open MPI will open sockets.
                 MCA btl: parameter "btl_tcp_exclusivity" (current value: <100>, data source: default value)
                          BTL exclusivity (must be >= 0)
                 MCA btl: parameter "btl_tcp_flags" (current value: <314>, data source: default value)
                          BTL bit flags (general flags: SEND=1, PUT=2, GET=4, SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by the "dr" PML (ignored by others): ACK=16, CHECKSUM=32,
                          RDMA_COMPLETION=128; flags only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512)
                 MCA btl: parameter "btl_tcp_rndv_eager_limit" (current value: <65536>, data source: default value)
                          Size (in bytes, including header) of "phase 1" fragment sent for all large messages (must be >= 0 and <= eager_limit)
                 MCA btl: parameter "btl_tcp_eager_limit" (current value: <65536>, data source: default value)
                          Maximum size (in bytes, including header) of "short" messages (must be >= 1).
                 MCA btl: parameter "btl_tcp_max_send_size" (current value: <131072>, data source: default value)
                          Maximum size (in bytes) of a single "phase 2" fragment of a long message when using the pipeline protocol (must be >= 1)
                 MCA btl: parameter "btl_tcp_rdma_pipeline_send_length" (current value: <131072>, data source: default value)
                          Length of the "phase 2" portion of a large message (in bytes) when using the pipeline protocol. This part of the message will be split into fragments of size max_send_size and sent
                          using send/receive semantics (must be >= 0; only relevant when the PUT flag is set)
                 MCA btl: parameter "btl_tcp_rdma_pipeline_frag_size" (current value: <2147483647>, data source: default value)
                          Maximum size (in bytes) of a single "phase 3" fragment from a long message when using the pipeline protocol. These fragments will be sent using RDMA semantics (must be >= 1; only
                          relevant when the PUT flag is set)
                 MCA btl: parameter "btl_tcp_min_rdma_pipeline_size" (current value: <0>, data source: default value)
                          Messages smaller than this size (in bytes) will not use the RDMA pipeline protocol. Instead, they will be split into fragments of max_send_size and sent using send/receive semantics
                          (must be >=0, and is automatically adjusted up to at least (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag is set)
                 MCA btl: parameter "btl_tcp_bandwidth" (current value: <100>, data source: default value)
                          Approximate maximum bandwidth of interconnect(0 = auto-detect value at run-time [not supported in all BTL modules], >= 1 = bandwidth in Mbps)
                 MCA btl: parameter "btl_tcp_latency" (current value: <100>, data source: default value)
                          Approximate latency of interconnect (must be >= 0)
                 MCA btl: parameter "btl_tcp_disable_family" (current value: <0>, data source: default value)
                 MCA btl: parameter "btl_tcp_if_seq" (current value: <none>, data source: default value)
                          If specified, a comma-delimited list of TCP interfaces. Interfaces will be assigned, one to each MPI process, in a round-robin fashion on each server. For example, if the list is
                          "eth0,eth1" and four MPI processes are run on a single server, then local ranks 0 and 2 will use eth0 and local ranks 1 and 3 will use eth1.
                 MCA btl: parameter "btl_tcp_priority" (current value: <0>, data source: default value)
                 MCA btl: parameter "btl_base_include" (current value: <none>, data source: default value)
                 MCA btl: parameter "btl_base_exclude" (current value: <none>, data source: default value)
                 MCA btl: parameter "btl_base_warn_component_unused" (current value: <1>, data source: default value)
                          This parameter is used to turn on warning messages when certain NICs are not used
                MCA coll: parameter "coll" (current value: <none>, data source: default value)
                          Default selection set of components for the coll framework (<none> means use all components that can be found)
                MCA coll: parameter "coll_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the coll framework (0 = no verbosity)
                MCA coll: parameter "coll_basic_priority" (current value: <10>, data source: default value)
                          Priority of the basic coll component
                MCA coll: parameter "coll_basic_crossover" (current value: <4>, data source: default value)
                          Minimum number of processes in a communicator before using the logarithmic algorithms
                MCA coll: parameter "coll_hierarch_priority" (current value: <0>, data source: default value)
                          Priority of the hierarchical coll component
                MCA coll: parameter "coll_hierarch_verbose" (current value: <0>, data source: default value)
                          Turn verbose message of the hierarchical coll component on/off
                MCA coll: parameter "coll_hierarch_use_rdma" (current value: <0>, data source: default value)
                          Switch from the send btl list used to detect hierarchies to the rdma btl list
                MCA coll: parameter "coll_hierarch_ignore_sm" (current value: <0>, data source: default value)
                          Ignore sm protocol when detecting hierarchies. Required to enable the usage of protocol specific collective operations
                MCA coll: parameter "coll_hierarch_detection_alg" (current value: <2>, data source: default value)
                          Used to specify the algorithm for detecting Hierarchy.Choose between all or two levels of hierarchy
                MCA coll: parameter "coll_hierarch_bcast_alg" (current value: <4>, data source: default value)
                          Used to specify the algorithm used for bcast operations.
                MCA coll: parameter "coll_hierarch_segment_size" (current value: <32768>, data source: default value)
                          Used to specify the segment size for segmented algorithms.
                MCA coll: parameter "coll_inter_priority" (current value: <40>, data source: default value)
                          Priority of the inter coll component
                MCA coll: parameter "coll_inter_verbose" (current value: <0>, data source: default value)
                          Turn verbose message of the inter coll component on/off
                MCA coll: parameter "coll_libnbc_priority" (current value: <10>, data source: default value)
                          Priority of the libnbc coll component
                MCA coll: parameter "coll_ml_priority" (current value: <0>, data source: default value)
                          ML component priority(from 0(low) to 90 (high))
                MCA coll: parameter "coll_ml_verbose" (current value: <0>, data source: default value)
                          Output some verbose ML information (0 = no output, nonzero = output)
                MCA coll: parameter "coll_ml_n_levels" (current value: <1>, data source: default value)
                          number of levels in the hierarchy
                MCA coll: parameter "coll_ml_max_comm" (current value: <12>, data source: default value)
                          max of communicators available to run ML
                MCA coll: parameter "coll_ml_min_comm_size" (current value: <0>, data source: default value)
                          min size of comm to be available to run ML
                MCA coll: parameter "coll_ml_n_payload_mem_banks" (current value: <2>, data source: default value)
                          number of payload memory banks
                MCA coll: parameter "coll_ml_n_payload_buffs_per_bank" (current value: <16>, data source: default value)
                          number of payload buffers per bank
                MCA coll: parameter "coll_ml_payload_buffer_size" (current value: <4096>, data source: default value)
                          size of payload buffer
                MCA coll: parameter "coll_ml_pipeline_depth" (current value: <2>, data source: default value)
                          size of fragmentation pipeline
                MCA coll: parameter "coll_ml_free_list_init_size" (current value: <128>, data source: default value)
                          Initial size for free lists in ML
                MCA coll: parameter "coll_ml_free_list_grow_size" (current value: <64>, data source: default value)
                          Initial size for free lists in ML
                MCA coll: parameter "coll_ml_free_list_max_size" (current value: <-1>, data source: default value)
                          Initial size for free lists in ML
                MCA coll: parameter "coll_ml_use_knomial_allreduce" (current value: <1>, data source: default value)
                          Use k-nomial Allreduce supports only p2p currently
                MCA coll: parameter "coll_ml_use_static_bcast" (current value: <1>, data source: default value)
                          Use new bcast static algorithm
                MCA coll: parameter "coll_ml_use_sequential_bcast" (current value: <0>, data source: default value)
                          Use new bcast static algorithm
                MCA coll: parameter "coll_ml_disable_allgather" (current value: <0>, data source: default value)
                          Allgather disabling
                MCA coll: parameter "coll_ml_disable_alltoall" (current value: <0>, data source: default value)
                          Alltoall disabling
                MCA coll: parameter "coll_ml_enable_fragmentation" (current value: <0>, data source: default value)
                          Disable/Enable fragmentation for large messages
                MCA coll: parameter "coll_ml_use_brucks_smsg_alltoall" (current value: <0>, data source: default value)
                          Use Bruck's Algo for Small Msg Alltoall1 - Bruck's Algo with RDMA; 2 - Bruck's with Send Recv
                MCA coll: parameter "coll_ml_config_file" (current value: </usr/local/share/openmpi/mca-coll-ml.config>, data source: default value)
                          ML collectives configuration file
                MCA coll: parameter "coll_ml_memory_manager_list_size" (current value: <8>, data source: default value)
                          Memory manager list size
                MCA coll: parameter "coll_ml_memory_manager_block_size" (current value: <1572864>, data source: default value)
                          Memory manager block size
                MCA coll: parameter "coll_ml_memory_manager_alignment" (current value: <4096>, data source: default value)
                          Memory manager alignment
                MCA coll: parameter "coll_self_priority" (current value: <75>, data source: default value)
                MCA coll: parameter "coll_sm_priority" (current value: <0>, data source: default value)
                          Priority of the sm coll component
                MCA coll: parameter "coll_sm_control_size" (current value: <4096>, data source: default value)
                          Length of the control data -- should usually be either the length of a cache line on most SMPs, or the size of a page on machines that support direct memory affinity page placement (in
                          bytes)
                MCA coll: parameter "coll_sm_fragment_size" (current value: <8192>, data source: default value)
                          Fragment size (in bytes) used for passing data through shared memory (will be rounded up to the nearest control_size size)
                MCA coll: parameter "coll_sm_comm_in_use_flags" (current value: <2>, data source: default value)
                          Number of "in use" flags, used to mark a message passing area segment as currently being used or not (must be >= 2 and <= comm_num_segments)
                MCA coll: parameter "coll_sm_comm_num_segments" (current value: <8>, data source: default value)
                          Number of segments in each communicator's shared memory message passing area (must be >= 2, and must be a multiple of comm_in_use_flags)
                MCA coll: parameter "coll_sm_tree_degree" (current value: <4>, data source: default value)
                          Degree of the tree for tree-based operations (must be => 1 and <= min(control_size, 255))
                MCA coll: parameter "coll_sm_info_num_procs" (current value: <4>, data source: default value)
                          Number of processes to use for the calculation of the shared_mem_size MCA information parameter (must be => 2)
                MCA coll: information "coll_sm_shared_mem_used_data" (value: <548864>, data source: default value)
                          Amount of shared memory used, per communicator, in the shared memory data area for info_num_procs processes (in bytes)
                MCA coll: parameter "coll_tuned_priority" (current value: <30>, data source: default value)
                          Priority of the tuned coll component
                MCA coll: parameter "coll_tuned_pre_allocate_memory_comm_size_limit" (current value: <32768>, data source: default value)
                          Size of communicator were we stop pre-allocating memory for the fixed internal buffer used for message requests etc that is hung off the communicator data segment. I.e. if you have a
                          100'000 nodes you might not want to pre-allocate 200'000 request handle slots per communicator instance!
                MCA coll: parameter "coll_tuned_init_tree_fanout" (current value: <4>, data source: default value)
                          Inital fanout used in the tree topologies for each communicator. This is only an initial guess, if a tuned collective needs a different fanout for an operation, it build it dynamically.
                          This parameter is only for the first guess and might save a little time
                MCA coll: parameter "coll_tuned_init_chain_fanout" (current value: <4>, data source: default value)
                          Inital fanout used in the chain (fanout followed by pipeline) topologies for each communicator. This is only an initial guess, if a tuned collective needs a different fanout for an
                          operation, it build it dynamically. This parameter is only for the first guess and might save a little time
                MCA coll: parameter "coll_tuned_use_dynamic_rules" (current value: <0>, data source: default value)
                          Switch used to decide if we use static (compiled/if statements) or dynamic (built at runtime) decision function rules
                 MCA dpm: parameter "dpm" (current value: <none>, data source: default value)
                          Default selection set of components for the dpm framework (<none> means use all components that can be found)
                 MCA dpm: parameter "dpm_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the dpm framework (0 = no verbosity)
                 MCA dpm: parameter "dpm_orte_priority" (current value: <0>, data source: default value)
                MCA fbtl: parameter "fbtl" (current value: <none>, data source: default value)
                          Default selection set of components for the fbtl framework (<none> means use all components that can be found)
                MCA fbtl: parameter "fbtl_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the fbtl framework (0 = no verbosity)
                MCA fbtl: parameter "fbtl_posix_priority" (current value: <0>, data source: default value)
               MCA fcoll: parameter "fcoll" (current value: <none>, data source: default value)
                          Default selection set of components for the fcoll framework (<none> means use all components that can be found)
               MCA fcoll: parameter "fcoll_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the fcoll framework (0 = no verbosity)
               MCA fcoll: parameter "fcoll_dynamic_priority" (current value: <10>, data source: default value)
                          Priority of the dynamic fcoll component
               MCA fcoll: parameter "fcoll_dynamic_num_io_procs" (current value: <-1>, data source: default value)
                          Number of writers in the dynamic fcoll component
               MCA fcoll: parameter "fcoll_dynamic_constant_cbs" (current value: <0>, data source: default value)
                          wether we are using constant or scaling cycle buffer size in the dynamic fcoll component
               MCA fcoll: parameter "fcoll_dynamic_cycle_buffer_size" (current value: <33554432>, data source: default value)
                          Cycle Buffer Size of the dynamic fcoll component
               MCA fcoll: parameter "fcoll_individual_priority" (current value: <10>, data source: default value)
                          Priority of the individual fcoll component
               MCA fcoll: parameter "fcoll_individual_constant_cbs" (current value: <0>, data source: default value)
                          wether we are using constant or scaling cycle buffer size in the individual fcoll component
               MCA fcoll: parameter "fcoll_individual_cycle_buffer_size" (current value: <33554432>, data source: default value)
                          Cycle Buffer Size of the individual fcoll component
               MCA fcoll: parameter "fcoll_static_priority" (current value: <10>, data source: default value)
                          Priority of the static fcoll component
               MCA fcoll: parameter "fcoll_static_num_io_procs" (current value: <-1>, data source: default value)
                          Number of writers in the static fcoll component
               MCA fcoll: parameter "fcoll_static_constant_cbs" (current value: <1>, data source: default value)
                          wether we are using constant or scaling cycle buffer size in the static fcoll component
               MCA fcoll: parameter "fcoll_static_cycle_buffer_size" (current value: <33554432>, data source: default value)
                          Cycle Buffer Size of the static fcoll component
               MCA fcoll: parameter "fcoll_two_phase_priority" (current value: <10>, data source: default value)
                          Priority of the two_phase fcoll component
               MCA fcoll: parameter "fcoll_two_phase_num_io_procs" (current value: <-1>, data source: default value)
                          Number of writers in the two_phase fcoll component
               MCA fcoll: parameter "fcoll_two_phase_constant_cbs" (current value: <0>, data source: default value)
                          wether we are using constant or scaling cycle buffer size in the two_phase fcoll component
               MCA fcoll: parameter "fcoll_two_phase_cycle_buffer_size" (current value: <33554432>, data source: default value)
                          Cycle Buffer Size of the two_phase fcoll component
               MCA fcoll: parameter "fcoll_ylib_priority" (current value: <0>, data source: default value)
                          Priority of the ylib fcoll component
               MCA fcoll: parameter "fcoll_ylib_num_io_procs" (current value: <1>, data source: default value)
                          Number of writers in the ylib fcoll component
               MCA fcoll: parameter "fcoll_ylib_stripe_size" (current value: <1048576>, data source: default value)
                          Stripe Size of the ylib fcoll component
               MCA fcoll: parameter "fcoll_ylib_blocks_per_cycle" (current value: <20>, data source: default value)
                          Blocks to write per cycle of the ylib fcoll component
                  MCA fs: parameter "fs" (current value: <none>, data source: default value)
                          Default selection set of components for the fs framework (<none> means use all components that can be found)
                  MCA fs: parameter "fs_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the fs framework (0 = no verbosity)
                  MCA fs: parameter "fs_ufs_priority" (current value: <0>, data source: default value)
                  MCA io: parameter "io_base_freelist_initial_size" (current value: <16>, data source: default value)
                          Initial MPI-2 IO request freelist size
                  MCA io: parameter "io_base_freelist_max_size" (current value: <64>, data source: default value)
                          Max size of the MPI-2 IO request freelist
                  MCA io: parameter "io_base_freelist_increment" (current value: <16>, data source: default value)
                          Increment size of the MPI-2 IO request freelist
                  MCA io: parameter "io" (current value: <none>, data source: default value)
                          Default selection set of components for the io framework (<none> means use all components that can be found)
                  MCA io: parameter "io_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the io framework (0 = no verbosity)
                  MCA io: parameter "io_ompio_priority" (current value: <10>, data source: default value)
                          Priority of the io ompio component
                  MCA io: parameter "io_ompio_delete_priority" (current value: <10>, data source: default value)
                          Delete priority of the io ompio component
                  MCA io: information "io_ompio_version" (value: <none>, data source: default value)
                          Version of OMPIO
                  MCA io: parameter "io_ompio_cycle_buffer_size" (current value: <33554432>, data source: default value)
                          Cycle Buffer Size of individual reads/writes
                  MCA io: parameter "io_ompio_bytes_per_agg" (current value: <33554432>, data source: default value)
                          Bytes per aggregator process for automatic selection
                  MCA io: parameter "io_romio_priority" (current value: <10>, data source: default value)
                          Priority of the io romio component
                  MCA io: parameter "io_romio_delete_priority" (current value: <10>, data source: default value)
                          Delete priority of the io romio component
                  MCA io: information "io_romio_version" (value: <from MPICH2 v1.3.1 with an additional patch from romio-maint_at_[hidden] about an attribute issue>, data source: default value)
                          Version of ROMIO
                  MCA io: information "io_romio_user_configure_params" (value: <none>, data source: default value)
                          User-specified command line parameters passed to ROMIO's configure script
                  MCA io: information "io_romio_complete_configure_params" (value: < CFLAGS='-O3 -DNDEBUG -finline-functions -fno-strict-aliasing -restrict -Qoption,cpp,--extended_float_types ' CPPFLAGS='
                          -D_REENTRANT -I/Users/admin/Downloads/openmpi/openmpi-1.7.2/opal/mca/hwloc/hwloc152/hwloc/include -I/Users/admin/Downloads/openmpi/openmpi-1.7.2/opal/mca/event/libevent2019/libevent
                          -I/Users/admin/Downloads/openmpi/openmpi-1.7.2/opal/mca/event/libevent2019/libevent/include -I/usr/include/infiniband -I/usr/include/infiniband -I/usr/include/infiniband
                          -I/usr/include/infiniband' FFLAGS='' LDFLAGS=' ' --enable-shared --disable-static --prefix=/usr/local --with-mpi=open_mpi --disable-aio>, data source: default value)
                          Complete set of command line parameters passed to ROMIO's configure script
                 MCA mpi: parameter "mpi_param_check" (current value: <1>, data source: default value)
                          Whether you want MPI API parameters checked at run-time or not. Possible values are 0 (no checking) and 1 (perform checking at run-time)
                 MCA mpi: parameter "mpi_yield_when_idle" (current value: <-1>, data source: default value)
                          Yield the processor when waiting for MPI communication (for MPI processes, will default to 1 when oversubscribing nodes)
                 MCA mpi: parameter "mpi_event_tick_rate" (current value: <-1>, data source: default value)
                          How often to progress TCP communications (0 = never, otherwise specified in microseconds)
                 MCA mpi: parameter "mpi_show_handle_leaks" (current value: <0>, data source: default value)
                          Whether MPI_FINALIZE shows all MPI handles that were not freed or not
                 MCA mpi: parameter "mpi_no_free_handles" (current value: <0>, data source: default value)
                          Whether to actually free MPI objects when their handles are freed
                 MCA mpi: parameter "mpi_show_mpi_alloc_mem_leaks" (current value: <0>, data source: default value)
                          If >0, MPI_FINALIZE will show up to this many instances of memory allocated by MPI_ALLOC_MEM that was not freed by MPI_FREE_MEM
                 MCA mpi: parameter "mpi_show_mca_params" (current value: <none>, data source: default value)
                          Whether to show all MCA parameter values during MPI_INIT or not (good for reproducability of MPI jobs for debug purposes). Accepted values are all, default, file, api, and enviro - or a
                          comma delimited combination of them
                 MCA mpi: parameter "mpi_show_mca_params_file" (current value: <none>, data source: default value)
                          If mpi_show_mca_params is true, setting this string to a valid filename tells Open MPI to dump all the MCA parameter values into a file suitable for reading via the mca_param_files
                          parameter (good for reproducability of MPI jobs)
                 MCA mpi: parameter "mpi_keep_peer_hostnames" (current value: <1>, data source: default value)
                          If nonzero, save the string hostnames of all MPI peer processes (mostly for error / debugging output messages). This can add quite a bit of memory usage to each MPI process.
                 MCA mpi: parameter "mpi_abort_delay" (current value: <0>, data source: default value)
                          If nonzero, print out an identifying message when MPI_ABORT is invoked (hostname, PID of the process that called MPI_ABORT) and delay for that many seconds before exiting (a negative
                          delay value means to never abort). This allows attaching of a debugger before quitting the job.
                 MCA mpi: parameter "mpi_abort_print_stack" (current value: <0>, data source: default value)
                          If nonzero, print out a stack trace when MPI_ABORT is invoked
                 MCA mpi: parameter "mpi_leave_pinned" (current value: <-1>, data source: default value)
                          Whether to use the "leave pinned" protocol or not. Enabling this setting can help bandwidth performance when repeatedly sending and receiving large messages with the same buffers over
                          RDMA-based networks (0 = do not use "leave pinned" protocol, 1 = use "leave pinned" protocol, -1 = allow network to choose at runtime).
                 MCA mpi: parameter "mpi_leave_pinned_pipeline" (current value: <0>, data source: default value)
                          Whether to use the "leave pinned pipeline" protocol or not.
                 MCA mpi: parameter "mpi_warn_on_fork" (current value: <1>, data source: default value)
                          If nonzero, issue a warning if program forks under conditions that could cause system errors
                 MCA mpi: information "mpi_have_sparse_group_storage" (value: <0>, data source: default value)
                          Whether this Open MPI installation supports storing of data in MPI groups in "sparse" formats (good for extremely large process count MPI jobs that create many communicators/groups)
                 MCA mpi: parameter "mpi_use_sparse_group_storage" (current value: <0>, data source: default value)
                          Whether to use "sparse" storage formats for MPI groups (only relevant if mpi_have_sparse_group_storage is 1)
               MCA mpool: parameter "mpool" (current value: <none>, data source: default value)
                          Default selection set of components for the mpool framework (<none> means use all components that can be found)
               MCA mpool: parameter "mpool_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the mpool framework (0 = no verbosity)
               MCA mpool: parameter "mpool_grdma_rcache_name" (current value: <vma>, data source: default value)
                          The name of the registration cache the mpool should use
               MCA mpool: parameter "mpool_grdma_rcache_size_limit" (current value: <0>, data source: default value)
                          the maximum size of registration cache in bytes. 0 is unlimited (default 0)
               MCA mpool: parameter "mpool_grdma_print_stats" (current value: <0>, data source: default value)
                          print pool usage statistics at the end of the run
               MCA mpool: parameter "mpool_grdma_priority" (current value: <0>, data source: default value)
               MCA mpool: parameter "mpool_sm_allocator" (current value: <bucket>, data source: default value)
                          Name of allocator component to use with sm mpool
               MCA mpool: parameter "mpool_sm_min_size" (current value: <67108864>, data source: default value)
                          Minimum size of the sm mpool shared memory file
               MCA mpool: parameter "mpool_sm_verbose" (current value: <0>, data source: default value)
                          Enable verbose output for mpool sm component
               MCA mpool: parameter "mpool_sm_priority" (current value: <0>, data source: default value)
                 MCA mtl: parameter "mtl" (current value: <none>, data source: default value)
                          Default selection set of components for the mtl framework (<none> means use all components that can be found)
                 MCA mtl: parameter "mtl_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the mtl framework (0 = no verbosity)
                  MCA op: parameter "op_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level of the op framework
                  MCA op: parameter "op" (current value: <none>, data source: default value)
                          Default selection set of components for the op framework (<none> means use all components that can be found)
                 MCA osc: parameter "osc" (current value: <none>, data source: default value)
                          Default selection set of components for the osc framework (<none> means use all components that can be found)
                 MCA osc: parameter "osc_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the osc framework (0 = no verbosity)
                 MCA osc: parameter "osc_pt2pt_eager_limit" (current value: <16384>, data source: default value)
                          Max size of eagerly sent data
                 MCA osc: parameter "osc_pt2pt_priority" (current value: <0>, data source: default value)
                 MCA osc: parameter "osc_rdma_eager_send" (current value: <1>, data source: default value)
                          Attempt to start data movement during communication call, instead of at synchrnoization time. Info key of same name overrides this value.
                 MCA osc: parameter "osc_rdma_use_buffers" (current value: <1>, data source: default value)
                          Coalesce messages during an epoch to reduce network utilization. Info key of same name overrides this value.
                 MCA osc: parameter "osc_rdma_use_rdma" (current value: <0>, data source: default value)
                          Use real RDMA operations to transfer data. Info key of same name overrides this value.
                 MCA osc: parameter "osc_rdma_rdma_completion_wait" (current value: <1>, data source: default value)
                          Wait for all completion of rdma events before sending acknowledgment. Info key of same name overrides this value.
                 MCA osc: parameter "osc_rdma_no_locks" (current value: <0>, data source: default value)
                          Enable optimizations available only if MPI_LOCK is not used. Info key of same name overrides this value.
                 MCA osc: parameter "osc_rdma_priority" (current value: <0>, data source: default value)
                 MCA pml: parameter "pml_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level of the PML framework
                 MCA pml: parameter "pml" (current value: <none>, data source: default value)
                          Default selection set of components for the pml framework (<none> means use all components that can be found)
                 MCA pml: parameter "pml_v_priority" (current value: <-1>, data source: default value)
                 MCA pml: parameter "pml_v_output" (current value: <stderr>, data source: default value)
                 MCA pml: parameter "pml_v_verbose" (current value: <0>, data source: default value)
                 MCA pml: parameter "pml_bfo_verbose" (current value: <0>, data source: default value)
                 MCA pml: parameter "pml_bfo_free_list_num" (current value: <4>, data source: default value)
                 MCA pml: parameter "pml_bfo_free_list_max" (current value: <-1>, data source: default value)
                 MCA pml: parameter "pml_bfo_free_list_inc" (current value: <64>, data source: default value)
                 MCA pml: parameter "pml_bfo_priority" (current value: <5>, data source: default value)
                 MCA pml: parameter "pml_bfo_send_pipeline_depth" (current value: <3>, data source: default value)
                 MCA pml: parameter "pml_bfo_recv_pipeline_depth" (current value: <4>, data source: default value)
                 MCA pml: parameter "pml_bfo_rdma_put_retries_limit" (current value: <5>, data source: default value)
                 MCA pml: parameter "pml_bfo_max_rdma_per_request" (current value: <4>, data source: default value)
                 MCA pml: parameter "pml_bfo_max_send_per_range" (current value: <4>, data source: default value)
                 MCA pml: parameter "pml_bfo_unexpected_limit" (current value: <128>, data source: default value)
                 MCA pml: parameter "pml_bfo_allocator" (current value: <bucket>, data source: default value)
                          Name of allocator component for unexpected messages
                 MCA pml: parameter "pml_cm_free_list_num" (current value: <4>, data source: default value)
                          Initial size of request free lists
                 MCA pml: parameter "pml_cm_free_list_max" (current value: <-1>, data source: default value)
                          Maximum size of request free lists
                 MCA pml: parameter "pml_cm_free_list_inc" (current value: <64>, data source: default value)
                          Number of elements to add when growing request free lists
                 MCA pml: parameter "pml_cm_priority" (current value: <10>, data source: default value)
                          CM PML selection priority
                 MCA pml: parameter "pml_ob1_verbose" (current value: <0>, data source: default value)
                 MCA pml: parameter "pml_ob1_free_list_num" (current value: <4>, data source: default value)
                 MCA pml: parameter "pml_ob1_free_list_max" (current value: <-1>, data source: default value)
                 MCA pml: parameter "pml_ob1_free_list_inc" (current value: <64>, data source: default value)
                 MCA pml: parameter "pml_ob1_priority" (current value: <20>, data source: default value)
                 MCA pml: parameter "pml_ob1_send_pipeline_depth" (current value: <3>, data source: default value)
                 MCA pml: parameter "pml_ob1_recv_pipeline_depth" (current value: <4>, data source: default value)
                 MCA pml: parameter "pml_ob1_max_rdma_per_request" (current value: <4>, data source: default value)
                 MCA pml: parameter "pml_ob1_max_send_per_range" (current value: <4>, data source: default value)
                 MCA pml: parameter "pml_ob1_unexpected_limit" (current value: <128>, data source: default value)
                 MCA pml: parameter "pml_ob1_allocator" (current value: <bucket>, data source: default value)
                          Name of allocator component for unexpected messages
              MCA pubsub: parameter "pubsub" (current value: <none>, data source: default value)
                          Default selection set of components for the pubsub framework (<none> means use all components that can be found)
              MCA pubsub: parameter "pubsub_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the pubsub framework (0 = no verbosity)
              MCA pubsub: parameter "pubsub_orte_priority" (current value: <50>, data source: default value)
                          Priority of the pubsub pmi component
              MCA rcache: parameter "rcache" (current value: <none>, data source: default value)
                          Default selection set of components for the rcache framework (<none> means use all components that can be found)
              MCA rcache: parameter "rcache_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the rcache framework (0 = no verbosity)
              MCA rcache: parameter "rcache_vma_priority" (current value: <0>, data source: default value)
            MCA sharedfp: parameter "sharedfp" (current value: <none>, data source: default value)
                          Default selection set of components for the sharedfp framework (<none> means use all components that can be found)
            MCA sharedfp: parameter "sharedfp_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the sharedfp framework (0 = no verbosity)
            MCA sharedfp: parameter "sharedfp_dummy_priority" (current value: <0>, data source: default value)
                MCA topo: parameter "topo" (current value: <none>, data source: default value)
                          Default selection set of components for the topo framework (<none> means use all components that can be found)
                MCA topo: parameter "topo_base_verbose" (current value: <0>, data source: default value)
                          Verbosity level for the topo framework (0 = no verbosity)
                MCA topo: parameter "topo_unity_priority" (current value: <0>, data source: default value)