It's not just one plugin, it was about 6 of them. I just deleted the error message from the others as I believed that opal_init was the problem.

However, I have done a full build multiple times and have blown away all the plugins and other remnants of the build and install and get the same results every time.

Here's the output from running ompi_info (same result with or w/o OPAL_PREFIX are the same; LD_LIBRARY_PATH set; path points to both bin and lib directory ):

[bolstadm-lm3:~/papillon/build/src] bolstadm% ompi_info                                          Package: Open MPI bolstadm@bolstadm-lm3 Distribution
                Open MPI: 1.6.3
   Open MPI SVN revision: r27472
   Open MPI release date: Oct 24, 2012
                Open RTE: 1.6.3
   Open RTE SVN revision: r27472
   Open RTE release date: Oct 24, 2012
                    OPAL: 1.6.3
       OPAL SVN revision: r27472
       OPAL release date: Oct 24, 2012
                 MPI API: 2.1
            Ident string: 1.6.3
                  Prefix: /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3
 Configured architecture: x86_64-apple-darwin12.2.1
          Configure host: bolstadm-lm3
           Configured by: bolstadm
           Configured on: Sun Feb 10 19:09:36 EST 2013
          Configure host: bolstadm-lm3
                Built by: bolstadm
                Built on: Sun Feb 10 19:16:52 EST 2013
              Built host: bolstadm-lm3
              C bindings: yes
            C++ bindings: yes
      Fortran77 bindings: no
      Fortran90 bindings: no
 Fortran90 bindings size: na
              C compiler: gcc
     C compiler absolute: /usr/bin/gcc
  C compiler family name: GNU
      C compiler version: 4.2.1
            C++ compiler: g++
   C++ compiler absolute: /usr/bin/g++
      Fortran77 compiler: gfortran
  Fortran77 compiler abs: /usr/bin/gfortran
      Fortran90 compiler: none
  Fortran90 compiler abs: none
             C profiling: yes
           C++ profiling: yes
     Fortran77 profiling: no
     Fortran90 profiling: no
          C++ exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: no
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
         libltdl support: yes
   Heterogeneous support: no
 mpirun default --prefix: no
         MPI I/O support: yes
       MPI_WTIME support: gettimeofday
     Symbol vis. support: yes
   Host topology support: yes
          MPI extensions: affinity example
   FT Checkpoint support: no (checkpoint thread: no)
     VampirTrace support: yes
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_paffinity_hwloc: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_carto_auto_detect: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_carto_file: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_shmem_mmap: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_shmem_posix: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_shmem_sysv: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_maffinity_first_use: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_maffinity_hwloc: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_sysinfo_darwin: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_errmgr_default: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_grpcomm_bad: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_grpcomm_basic: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ess_env: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ess_hnp: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ess_singleton: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ess_slave: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ess_slurm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ess_slurmd: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ess_tool: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_notifier_command: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_notifier_syslog: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_oob_tcp: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_odls_default: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_iof_hnp: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_iof_orted: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_iof_tool: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ras_cm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_ras_slurm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_rmaps_load_balance: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_rmaps_rank_file: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_rmaps_resilient: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_rmaps_round_robin: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_rmaps_seq: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_rmaps_topo: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_rml_oob: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_routed_binomial: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_routed_cm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_routed_direct: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_routed_linear: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_routed_radix: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_routed_slave: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_plm_rsh: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_plm_slurm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_filem_rsh: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_allocator_basic: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_allocator_bucket: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_coll_basic: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_coll_hierarch: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_coll_inter: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_coll_self: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_coll_sm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_coll_sync: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_coll_tuned: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_io_romio: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_rcache_vma: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_mpool_fake: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_mpool_rdma: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_mpool_sm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_pml_bfo: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_pml_cm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_pml_csum: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_pml_ob1: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_pml_v: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_osc_pt2pt: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_osc_rdma: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_btl_self: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_btl_sm: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_btl_tcp: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_topo_unity: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_pubsub_orte: lt_dlerror() returned NULL! (ignored)
[bolstadm-lm3:86426] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_dpm_orte: lt_dlerror() returned NULL! (ignored)
           MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6.3)
               MCA timer: darwin (MCA v2.0, API v2.0, Component v1.6.3)
         MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6.3)
         MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6.3)
               MCA hwloc: hwloc132 (MCA v2.0, API v2.0, Component v1.6.3)

And here's an example of a plugin reported as an error:
[bolstadm-lm3:~/papillon/build/src] bolstadm% l /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_paffinity_hwloc*
/Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_paffinity_hwloc.a
/Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_paffinity_hwloc.la <http://mca_paffinity_hwloc.la> *

On Mon, Feb 11, 2013 at 8:27 AM, Jeff Squyres (jsquyres) <jsquyres@cisco.com> wrote:

That's very idd; I cant think of why that would happen offhand. I build and run all the time on ML with no problems. Can you deleted that plugin and run ok?

Sent from my phone. No type good.

On Feb 10, 2013, at 10:22 PM, "Mark Bolstad" <the.render.dude@gmail.com> wrote:

> I having some difficulties with building/running 1.6.3 on Mountain Lion (10.8.2). I build with no errors and install into a prefix directory. I get the following errors:
> ...
> [bolstadm-lm3:38486] mca: base: component_find: unable to open /Users/bolstadm/papillon/build/macosx-x86_64/Release/openmpi-1.6.3/lib/openmpi/mca_shmem_sysv: lt_dlerror() returned NULL! (ignored)
> --------------------------------------------------------------------------
> It looks like opal_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during opal_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
>
>   opal_shmem_base_select failed
>   --> Returned value -1 instead of OPAL_SUCCESS
> --------------------------------------------------------------------------
> [bolstadm-lm3:38486] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file runtime/orte_init.c at line 79
> [bolstadm-lm3:38486] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file orterun.c at line 694
>
> I've fiddled with LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, OPAL_PREFIX, in combination and separately, and none of these seem to have much effect.
>
> So, I decided to try a straight build. The only option was --disable-mpi-f77. It installed into /usr/local. There is no other mpi version installed on the system, and I still get the same errors.
>
> However, I did install the version from MacPorts (also 1.6.3), and it works correctly.
>
> I would appreciate if anyone had some insight into building on OS X 10.8.
>
> Mark
>
> _______________________________________________
> users mailing list
> users@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

---

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users