Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Deadlock with mpi_init_thread + mpi_file_set_view
From: Rob Latham (robl_at_[hidden])
Date: 2011-04-04 10:18:40


On Sat, Apr 02, 2011 at 04:59:34PM -0400, fah10_at_[hidden] wrote:
>
> opal_mutex_lock(): Resource deadlock avoided
> #0 0x0012e416 in __kernel_vsyscall ()
> #1 0x01035941 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #2 0x01038e42 in abort () at abort.c:92
> #3 0x00d9da68 in ompi_attr_free_keyval (type=COMM_ATTR, key=0xbffda0e4, predefined=0 '\000') at attribute/attribute.c:656
> #4 0x00dd8aa2 in PMPI_Keyval_free (keyval=0xbffda0e4) at pkeyval_free.c:52
> #5 0x01bf3e6a in ADIOI_End_call (comm=0xf1c0c0, keyval=10, attribute_val=0x0, extra_state=0x0) at ad_end.c:82
> #6 0x00da01bb in ompi_attr_delete. (type=UNUSED_ATTR, object=0x6, attr_hash=0x2c64, key=14285602, predefined=232 '\350', need_lock=128 '\200') at attribute/attribute.c:726
> #7 0x00d9fb22 in ompi_attr_delete_all (type=COMM_ATTR, object=0xf1c0c0, attr_hash=0x8d0fee8) at attribute/attribute.c:1043
> #8 0x00dbda65 in ompi_mpi_finalize () at runtime/ompi_mpi_finalize.c:133
> #9 0x00dd12c2 in PMPI_Finalize () at pfinalize.c:46
> #10 0x00d6b515 in mpi_finalize_f (ierr=0xbffda2b8) at pfinalize_f.c:62

I guess I need some OpenMPI eyeballs on this...

ROMIO hooks into the attribute keyval deletion mechanism to clean up
the internal data structures it has allocated. I suppose since this
is MPI_Finalize, we could just leave those internal data structures
alone and let the OS deal with it.

What I see happening here is the OpenMPI finalize routine is deleting
attributes. one of those attributes is ROMIO's, which in turn tries
to free keyvals. Is the deadlock that noting "under" ompi_attr_delete
can itself call ompi_* routines? (as ROMIO triggers a call to
ompi_attr_free_keyval) ?

Here's where ROMIO sets up the keyval and the delete handler:
https://trac.mcs.anl.gov/projects/mpich2/browser/mpich2/trunk/src/mpi/romio/mpi-io/mpir-mpioinit.c#L39

that routine gets called upon any "MPI-IO entry point" (open, delete,
register-datarep). The keyvals help ensure that ROMIO's internal
structures get initialized exactly once, and the delete hooks help us
be good citizens and clean up on exit.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA