Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] scif btl side effects
From: Gilles Gouaillardet (gilles.gouaillardet_at_[hidden])
Date: 2014-05-07 07:23:32


Dear OpenMPI Folks,

i noticed some crashes when running OpenMPI (both latest v1.8 and trunk
from svn) on a single linux system where a MIC is available.
/* strictly speaking, MIC hardware is not needed: libscif.so, mic kernel
module and accessible /dev/mic/* are enough */

the attached test_scif program can be used in order to evidence this issue.
/* this is an over simplified version of collective/bcast_struct.c from
the ibm test suite,
it is currently failing on the bend-rsh cluster at intel */

this program will cause a silent failure
(MPI_Recv receives truncated data without issuing any warning)

i ran a few investigations and basically, here is what i found :
MPI_Send will split the message into two fragments. the first fragment
is sent via the vader btl
and the second fragment is sent with the scif btl.

the program will success if the scif btl is disabled (mpirun --mca btl
^scif)
interestingly, i found that
mpirun -host localhost -np 2 --mca btl scif,self ./test_scif
does produce correct results with ompi v1.8 r31309 (a crash might happen
in MPI_Finalize)
and it procuce incorrect results with ompi v1.8 r31671 and trunk (r31667)

imho :
a) the scif btl could/should be automatically disabled if no MIC is
detected on a host
b) the scif btl could/should not be used to communicates between two
cores of the host
(e.g. it should be used *only* when at least one peer is on the MIC)
c) that being said, that should work so there is a bug
d) there is a regression in v1.8 and a bug that might have been always here

i attached a patch that will automatically disable the scif btl if no
MIC is found
(e.g. scif_get_nodeIDs(...) returns 1), i believe it is safe to use it
(that being said, we might want to add an option to force the use of the
scif btl no matter what)

could you please share your thoughts on my asumptions a) b) c) and d) ?
if b) is what we want to implement, then mca_btl_scif_add_procs could be
modified as follow :

        if (!OPAL_PROC_ON_LOCAL_HOST(ompi_proc->proc_flags) ||
            my_proc == ompi_proc) {
            /* scif can only be used with procs on this board */
            continue;
        }

becomes

        if (!OPAL_PROC_ON_LOCAL_HOST(ompi_proc->proc_flags) ||
            my_proc == ompi_proc || (!IS_MIC(my_proc) &&
!IS_MIC(ompi_proc)) {
            /* scif can only be used with procs on this board unless
both procs are not on MIC */
            continue;
        }

and IS_MIC(proc) has to be implemented ...
/* is hwloc 1.7.2 already able to do this ? if yes, pointers will be
highly appreciated */

Cheers,

Gilles