Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] [omx-devel] Open-mx issue with ompi 1.6.1
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2012-09-10 14:44:38


I replied a couple days ago (with OMPI users in CC) but got an error
last night:

Action: failed
Status: 5.0.0 (permanent failure)
Diagnostic-Code: smtp; 5.4.7 - Delivery expired (message too old) 'timeout' (delivery attempts: 0)

I resent the mail this morning, it looks like it wasn't delivered yet
either?
I'll try again with another SMTP.

Brice

Le 10/09/2012 20:37, Douglas Eadline a écrit :
> I got no response for this question. Is Open-MX
> no longer supported in Open MPI? Or is there someplace else
> I should submit this information? I also attached my ompi_info
> and omx_info output
>
> --
> Doug
>
>
>> I built open-mpi 1.6.1 using the open-mx libraries.
>> This worked previously and now I get the following
>> error. Here is my system:
>>
>> kernel: 2.6.32-279.5.1.el6.x86_64
>> open-mx: 1.5.2
>>
>> BTW, open-mx worked previously with open-mpi and the current
>> version works with mpich2
>>
>>
>> $ mpiexec -np 8 -machinefile machines cpi
>> Process 0 on limulus
>> FatalError: Failed to lookup peer by addr, driver replied Bad file
>> descriptor
>> cpi: ../omx_misc.c:89: omx__ioctl_errno_to_return_checked: Assertion `0'
>> failed.
>> [limulus:04448] *** Process received signal ***
>> [limulus:04448] Signal: Aborted (6)
>> [limulus:04448] Signal code: (-6)
>> [limulus:04448] [ 0] /lib64/libpthread.so.0() [0x3324e0f500]
>> [limulus:04448] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x33246328a5]
>> [limulus:04448] [ 2] /lib64/libc.so.6(abort+0x175) [0x3324634085]
>> [limulus:04448] [ 3] /lib64/libc.so.6() [0x332462ba1e]
>> [limulus:04448] [ 4] /lib64/libc.so.6(__assert_perror_fail+0)
>> [0x332462bae0]
>> [limulus:04448] [ 5]
>> /usr/open-mx/lib/libopen-mx.so.0(omx__ioctl_errno_to_return_checked+0x197)
>> [0x7fb587418b37]
>> [limulus:04448] [ 6]
>> /usr/open-mx/lib/libopen-mx.so.0(omx__peer_addr_to_index+0x55)
>> [0x7fb58741a5d5]
>> [limulus:04448] [ 7] /usr/open-mx/lib/libopen-mx.so.0(+0xdc7a)
>> [0x7fb587419c7a]
>> [limulus:04448] [ 8] /usr/open-mx/lib/libopen-mx.so.0(omx_connect+0x8c)
>> [0x7fb58741a27c]
>> [limulus:04448] [ 9] /usr/open-mx/lib/libopen-mx.so.0(mx_connect+0x15)
>> [0x7fb587425865]
>> [limulus:04448] [10]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(mca_btl_mx_proc_connect+0x5e)
>> [0x7fb5876fe40e]
>> [limulus:04448] [11]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(mca_btl_mx_send+0x2d4)
>> [0x7fb5876fbd94]
>> [limulus:04448] [12]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(mca_pml_ob1_send_request_start_prepare+0xcb)
>> [0x7fb58777d6fb]
>> [limulus:04448] [13]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(mca_pml_ob1_isend+0x4cb)
>> [0x7fb58777509b]
>> [limulus:04448] [14]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(ompi_coll_tuned_bcast_intra_generic+0x37b)
>> [0x7fb58770b55b]
>> [limulus:04448] [15]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(ompi_coll_tuned_bcast_intra_binomial+0xd8)
>> [0x7fb58770b8b8]
>> [limulus:04448] [16]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(ompi_coll_tuned_bcast_intra_dec_fixed+0xcc)
>> [0x7fb587702d8c]
>> [limulus:04448] [17]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(mca_coll_sync_bcast+0x78)
>> [0x7fb587712e88]
>> [limulus:04448] [18]
>> /opt/mpi/openmpi-gnu4/lib64/libmpi.so.1(MPI_Bcast+0x130) [0x7fb5876ce1b0]
>> [limulus:04448] [19] cpi(main+0x10b) [0x400cc4]
>> [limulus:04448] [20] /lib64/libc.so.6(__libc_start_main+0xfd)
>> [0x332461ecdd]
>> [limulus:04448] [21] cpi() [0x400ac9]
>> [limulus:04448] *** End of error message ***
>> Process 2 on limulus
>> Process 4 on limulus
>> Process 6 on limulus
>> Process 1 on n0
>> Process 7 on n0
>> Process 3 on n0
>> Process 5 on n0
>> --------------------------------------------------------------------------
>> mpiexec noticed that process rank 0 with PID 4448 on node limulus exited
>> on signal 6 (Aborted).
>> --------------------------------------------------------------------------
>>
> [root_at_limulus ~]# ompi_info
> Package: Open MPI deadline_at_limulus Distribution
> Open MPI: 1.6.1
> Open MPI SVN revision: r27106
> Open MPI release date: Aug 22, 2012
> Open RTE: 1.6.1
> Open RTE SVN revision: r27106
> Open RTE release date: Aug 22, 2012
> OPAL: 1.6.1
> OPAL SVN revision: r27106
> OPAL release date: Aug 22, 2012
> MPI API: 2.1
> Ident string: 1.6.1
> Prefix: /opt/mpi/openmpi-gnu4-1.6.1
> Configured architecture: x86_64-unknown-linux-gnu
> Configure host: limulus
> Configured by: deadline
> Configured on: Wed Sep 5 15:03:09 EDT 2012
> Configure host: limulus
> Built by: deadline
> Built on: Wed Sep 5 15:21:19 EDT 2012
> Built host: limulus
> C bindings: yes
> C++ bindings: yes
> Fortran77 bindings: yes (all)
> Fortran90 bindings: yes
> Fortran90 bindings size: small
> C compiler: gcc
> C compiler absolute: /usr/bin/gcc
> C compiler family name: GNU
> C compiler version: 4.4.6
> C++ compiler: g++
> C++ compiler absolute: /usr/bin/g++
> Fortran77 compiler: gfortran
> Fortran77 compiler abs: /usr/bin/gfortran
> Fortran90 compiler: /usr/bin/gfortran
> Fortran90 compiler abs:
> C profiling: yes
> C++ profiling: yes
> Fortran77 profiling: yes
> Fortran90 profiling: yes
> C++ exceptions: no
> Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no)
> Sparse Groups: no
> Internal debug support: no
> MPI interface warnings: no
> MPI parameter check: runtime
> Memory profiling support: no
> Memory debugging support: no
> libltdl support: yes
> Heterogeneous support: no
> mpirun default --prefix: no
> MPI I/O support: yes
> MPI_WTIME support: gettimeofday
> Symbol vis. support: yes
> Host topology support: yes
> MPI extensions: affinity example
> FT Checkpoint support: no (checkpoint thread: no)
> VampirTrace support: yes
> MPI_MAX_PROCESSOR_NAME: 256
> MPI_MAX_ERROR_STRING: 256
> MPI_MAX_OBJECT_NAME: 64
> MPI_MAX_INFO_KEY: 36
> MPI_MAX_INFO_VAL: 256
> MPI_MAX_PORT_NAME: 1024
> MPI_MAX_DATAREP_STRING: 128
> MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6.1)
> MCA memory: linux (MCA v2.0, API v2.0, Component v1.6.1)
> MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.1)
> MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.6.1)
> MCA carto: file (MCA v2.0, API v2.0, Component v1.6.1)
> MCA shmem: mmap (MCA v2.0, API v2.0, Component v1.6.1)
> MCA shmem: posix (MCA v2.0, API v2.0, Component v1.6.1)
> MCA shmem: sysv (MCA v2.0, API v2.0, Component v1.6.1)
> MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.6.1)
> MCA maffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.1)
> MCA timer: linux (MCA v2.0, API v2.0, Component v1.6.1)
> MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6.1)
> MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6.1)
> MCA sysinfo: linux (MCA v2.0, API v2.0, Component v1.6.1)
> MCA hwloc: hwloc132 (MCA v2.0, API v2.0, Component v1.6.1)
> MCA dpm: orte (MCA v2.0, API v2.0, Component v1.6.1)
> MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.6.1)
> MCA allocator: basic (MCA v2.0, API v2.0, Component v1.6.1)
> MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.6.1)
> MCA coll: basic (MCA v2.0, API v2.0, Component v1.6.1)
> MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.6.1)
> MCA coll: inter (MCA v2.0, API v2.0, Component v1.6.1)
> MCA coll: self (MCA v2.0, API v2.0, Component v1.6.1)
> MCA coll: sm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA coll: sync (MCA v2.0, API v2.0, Component v1.6.1)
> MCA coll: tuned (MCA v2.0, API v2.0, Component v1.6.1)
> MCA io: romio (MCA v2.0, API v2.0, Component v1.6.1)
> MCA mpool: fake (MCA v2.0, API v2.0, Component v1.6.1)
> MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.6.1)
> MCA mpool: sm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA pml: bfo (MCA v2.0, API v2.0, Component v1.6.1)
> MCA pml: cm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA pml: csum (MCA v2.0, API v2.0, Component v1.6.1)
> MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.6.1)
> MCA pml: v (MCA v2.0, API v2.0, Component v1.6.1)
> MCA bml: r2 (MCA v2.0, API v2.0, Component v1.6.1)
> MCA rcache: vma (MCA v2.0, API v2.0, Component v1.6.1)
> MCA btl: self (MCA v2.0, API v2.0, Component v1.6.1)
> MCA btl: mx (MCA v2.0, API v2.0, Component v1.6.1)
> MCA btl: sm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA btl: tcp (MCA v2.0, API v2.0, Component v1.6.1)
> MCA mtl: mx (MCA v2.0, API v2.0, Component v1.6.1)
> MCA topo: unity (MCA v2.0, API v2.0, Component v1.6.1)
> MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.6.1)
> MCA osc: rdma (MCA v2.0, API v2.0, Component v1.6.1)
> MCA iof: hnp (MCA v2.0, API v2.0, Component v1.6.1)
> MCA iof: orted (MCA v2.0, API v2.0, Component v1.6.1)
> MCA iof: tool (MCA v2.0, API v2.0, Component v1.6.1)
> MCA oob: tcp (MCA v2.0, API v2.0, Component v1.6.1)
> MCA odls: default (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ras: cm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ras: loadleveler (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.6.1)
> MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.6.1)
> MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.6.1)
> MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.6.1)
> MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.6.1)
> MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.6.1)
> MCA rml: oob (MCA v2.0, API v2.0, Component v1.6.1)
> MCA routed: binomial (MCA v2.0, API v2.0, Component v1.6.1)
> MCA routed: cm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA routed: direct (MCA v2.0, API v2.0, Component v1.6.1)
> MCA routed: linear (MCA v2.0, API v2.0, Component v1.6.1)
> MCA routed: radix (MCA v2.0, API v2.0, Component v1.6.1)
> MCA routed: slave (MCA v2.0, API v2.0, Component v1.6.1)
> MCA plm: rsh (MCA v2.0, API v2.0, Component v1.6.1)
> MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA filem: rsh (MCA v2.0, API v2.0, Component v1.6.1)
> MCA errmgr: default (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ess: env (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ess: hnp (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ess: singleton (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ess: slave (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6.1)
> MCA ess: tool (MCA v2.0, API v2.0, Component v1.6.1)
> MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.6.1)
> MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.6.1)
> MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.6.1)
> MCA notifier: command (MCA v2.0, API v1.0, Component v1.6.1)
> MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.6.1)
>
>
> [root_at_limulus ~]# /usr/open-mx/bin/omx_info
> Open-MX version 1.5.2
> build: deadline_at_limulus:/raid1/home/deadline/rpms-sl6/BUILD/open-mx-1.5.2
> Mon Sep 10 08:44:16 EDT 2012
>
> Found 1 boards (32 max) supporting 32 endpoints each:
> limulus:0 (board #0 name eth0 addr e0:69:95:35:d7:71)
> managed by driver 'e1000e'
>
> Peer table is ready, mapper is 00:00:00:00:00:00
> ================================================
> 0) e0:69:95:35:d7:71 limulus:0
> 1) 00:1c:c0:9b:66:d0 n1:0
> 2) 00:1a:4d:4a:bf:85 n0:0
> 3) 00:1a:4d:4a:bf:83 n2:0
>
>
>
>
>
>
> --
> Doug
>