To determine if an MPI process is waiting for a message do what Rayson suggested and attach a debugger to the processes and see if any of them are stuck in MPI.  Either internally in a MPI_Recv or MPI_Wait call or looping on a MPI_Test call.

Other things to consider. 
  Is this the first time you've ran it (with Open MPI? with any MPI?)?
  How many processes is the job using?  Are you oversubscribing your processors?
  What version of Open MPI are you using?
  Have you tested all network connections? 
  It might help us to know the size of cluster you are running and what type of network?

On 4/24/2012 2:42 AM, Syed Ahsan Ali wrote:
Dear Rayson,

That is a Nuemrical model that is written by National weather service of a country. The logs of the model show every detail about the simulation progress. I have checked on the remote nodes as well the application binary is running but the logs show no progress, it is just waiting at a point. The input data is correct everything is fine. How can I check if the MPI task is waiting for a message?

On Tue, Apr 24, 2012 at 11:03 AM, Rayson Ho <> wrote:
Seems like there's a bug in the application. Did you or someone else
write it, or did you get it from an ISV??

You can log onto one of the nodes, attach a debugger, and see if the
MPI task is waiting for a message (looping in one of the MPI receive


Open Grid Scheduler / Grid Engine

Scalable Grid Engine Support Program

On Tue, Apr 24, 2012 at 12:49 AM, Syed Ahsan Ali <> wrote:
> Dear All,
> I am having problem with running an application on Dell cluster . The model
> starts well but no further progress is shown. It just stuck. I have checked
> the systems, no apparent hardware error is there. Other open mpi
> applications are running well on the same cluster. I have tried running the
> application on cores of the same server as well but the problem is same. The
> application just don't move further. The same application is also running
> well on a backup cluster. Please help.
> Thanks and Best Regards
> Ahsan
> _______________________________________________
> users mailing list

Open Grid Scheduler - The Official Open Source Grid Engine

users mailing list

users mailing list

Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle - Performance Technologies
95 Network Drive, Burlington, MA 01803