Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] VampirTrace: time not increasing
From: Holger Mickler (holger.mickler_at_[hidden])
Date: 2012-07-06 10:57:13


Oh, I just realized that you are probably using the Open MPI version of VT which
builds as part of the Open MPI build. I'm not 100% sure if the modification of
config.h works as laid out, but it should... you need to look out for VT's
config.h then, not Open MPI's.

Holger

On 07/06/2012 04:54 PM, Holger Mickler wrote:
> Hi Gene,
>
> this error is often caused by insufficiently synchronized TSCs (time stamp
> counter) of different processors/cores.
> When VT uses the TSC for timing the events (it does that by default), and the
> processes switch to another core during execution, it may well happen that the
> next recorded time stamp is earlier in time than the last one.
>
> One possibility to avoid this situation is pinning the processes to cores - Open
> MPI has functionality for realizing this, see
> http://www.open-mpi.org/faq/?category=tuning#using-paffinity
>
> If this is not feasible, you may use another clock source with VT which provides
> global time. To do this, you need to compile another version of VT. Run
> configure as usual, then edit config.h: replace the value of
> #define TIMER [...]
> with e.g.
> #define TIMER TIMER_CLOCK_GETTIME
> or
> #define TIMER TIMER_GETTIMEOFDAY
> depending on what is available on your system. Be aware that the resolution of
> those clocks is not as high as the TSC's.
>
> Have a look inside config.h at the place of the mentioned variables - there is
> some documentation there.
> Afterwards, compile and install VT. Using this version, you should not encounter
> the errors anymore.
>
> Regards,
> Holger
>
>
>
>
> On 07/06/2012 04:04 PM, Fluder, Eugene wrote:
>> I got the following error running a VT enabled run of AMBER. This was reported
>> in December of 2009 under almost identical conditions but the thread does not
>> contain a resolution. I reran the test with VT_UNIFY=no and it completed
>> normally. The same error occurred when I ran vtunify separately. Any help?
>>
>> Was this ever resolved?
>>
>> Gene
>>
>> [fludee01_at_node7-10 trace_noiox]$ vtunify 8 a
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file: OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 99459634, p= 6)
>> vtunify: Error: Could not read events of OTF stream [namestub ./a__ufy.tmp id 6]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file: OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 105413860, p= 5)
>> vtunify: Error: Could not read events of OTF stream [namestub ./a__ufy.tmp id 5]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file: OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 103189146, p= 7)
>> vtunify: Error: Could not read events of OTF stream [namestub ./a__ufy.tmp id 7]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file: OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 100509810, p= 8)
>> vtunify: Error: Could not read events of OTF stream [namestub ./a__ufy.tmp id 8]
>> vtunify: An error occurred during unifying events - Terminating ...
>>
>> -- /Eugene M Fluder, Jr, PhD/
>> /Computational Scientist/
>> /Scientific Computing/
>> /
>> /
>> /The Mt. Sinai School of Medicine/
>> /One Gustave L. Levy Place, Box 1498/
>>
>> /New York, NY 10029-6574/
>>
>> /
>> /
>>
>> /T: 212 659 8608/
>>
>> /F: 646 537 8660/
>>
>> /E: eugene.fluder_at_[hidden]/
>>
>> /
>> /
>>
>> //
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Dipl.-Inf. Holger Mickler
Technische Universität Dresden
Center for Information Services
and High Performance Computing (ZIH)
01062 Dresden
Germany
Office:  Willers-Bau (WIL) A36
Tel.:    +49 (351) 463-37903
Fax:     +49 (351) 463-37773
E-Mail:  holger.mickler_at_[hidden]