Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Hang in collectives involving shared memory
From: Ashley Pittman (ashley_at_[hidden])
Date: 2009-06-15 14:29:37


On Wed, 2009-06-10 at 09:07 -0600, Ralph Castain wrote:
> Hi Ashley
>
> Thanks! I would definitely be interested and will look at the tool.
> Meantime, I have filed a bunch of data on this in ticket #1944, so
> perhaps you might take a glance at that and offer some thoughts?
>
> https://svn.open-mpi.org/trac/ompi/ticket/1944
>
> Will be back after I look at the tool.

Have you made any progress?

Whilst the fact that it appears to only happen on your machine implies
it's not a general problem with OpenMPI the fact that it happens in the
same location/rep count every time does swing the blame back the other
way. Perhaps it's some special configure or runtime option you are
setting? One thing that springs to mind is the numa-maps could be
exposing some timimg problem with shared memory calls however this
doesn't sit well with it always failing on the same iteration.

Can you provide stack traces from when it's hung and crucially are they
the same for every hang? If you change the coll_sync_barrier_before
value to make it hang on a different repetition does this change the
stack trace at all?

Likewise when you have applied the collectives patch is the collective
state the same for every hang and how does this differ when you change
the coll_sync_barrier_before variable?

It would be useful to see stack traces and collective state from the
three collectives you report as causing problems, MPI_Bcast, MPI_Reduce
and MPI_Allgather because as I said before these three collectives have
radically different communication patterns.

Ashley,

-- 
Ashley Pittman, Bath, UK.
Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk