Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] VampirTrace: time not increasing
From: Holger Mickler (holger.mickler_at_[hidden])
Date: 2012-07-06 10:54:15


Hi Gene,

this error is often caused by insufficiently synchronized TSCs (time stamp
counter) of different processors/cores.
When VT uses the TSC for timing the events (it does that by default), and the
processes switch to another core during execution, it may well happen that the
next recorded time stamp is earlier in time than the last one.

One possibility to avoid this situation is pinning the processes to cores - Open
MPI has functionality for realizing this, see
http://www.open-mpi.org/faq/?category=tuning#using-paffinity

If this is not feasible, you may use another clock source with VT which provides
global time. To do this, you need to compile another version of VT. Run
configure as usual, then edit config.h: replace the value of
#define TIMER [...]
with e.g.
#define TIMER TIMER_CLOCK_GETTIME
or
#define TIMER TIMER_GETTIMEOFDAY
depending on what is available on your system. Be aware that the resolution of
those clocks is not as high as the TSC's.

Have a look inside config.h at the place of the mentioned variables - there is
some documentation there.
Afterwards, compile and install VT. Using this version, you should not encounter
the errors anymore.

Regards,
  Holger

On 07/06/2012 04:04 PM, Fluder, Eugene wrote:
> I got the following error running a VT enabled run of AMBER. This was reported
> in December of 2009 under almost identical conditions but the thread does not
> contain a resolution. I reran the test with VT_UNIFY=no and it completed
> normally. The same error occurred when I ran vtunify separately. Any help?
>
> Was this ever resolved?
>
> Gene
>
> [fludee01_at_node7-10 trace_noiox]$ vtunify 8 a
> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file: OTF_WBuffer.c, line: 308:
> time not increasing. (t= 99459634, p= 6)
> vtunify: Error: Could not read events of OTF stream [namestub ./a__ufy.tmp id 6]
> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file: OTF_WBuffer.c, line: 308:
> time not increasing. (t= 105413860, p= 5)
> vtunify: Error: Could not read events of OTF stream [namestub ./a__ufy.tmp id 5]
> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file: OTF_WBuffer.c, line: 308:
> time not increasing. (t= 103189146, p= 7)
> vtunify: Error: Could not read events of OTF stream [namestub ./a__ufy.tmp id 7]
> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file: OTF_WBuffer.c, line: 308:
> time not increasing. (t= 100509810, p= 8)
> vtunify: Error: Could not read events of OTF stream [namestub ./a__ufy.tmp id 8]
> vtunify: An error occurred during unifying events - Terminating ...
>
> -- /Eugene M Fluder, Jr, PhD/
> /Computational Scientist/
> /Scientific Computing/
> /
> /
> /The Mt. Sinai School of Medicine/
> /One Gustave L. Levy Place, Box 1498/
>
> /New York, NY 10029-6574/
>
> /
> /
>
> /T: 212 659 8608/
>
> /F: 646 537 8660/
>
> /E: eugene.fluder_at_[hidden]/
>
> /
> /
>
> //
>
>
>
>
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel