Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] VampirTrace: time not increasing
From: Fluder, Eugene (eugene.fluder_at_[hidden])
Date: 2012-07-06 11:33:49


Holger,

Thanks. I appreciate the detail.

Gene
-- Eugene M Fluder, Jr, PhD
        Computational Scientist
        Scientific Computing

        The Mt. Sinai School of Medicine
        One Gustave L. Levy Place, Box 1498
        New York, NY 10029-6574

        T: 212 659 8608
        F: 646 537 8660
        E: eugene.fluder_at_[hidden]

        

On 7/6/12 10:54 AM, "Holger Mickler" <holger.mickler_at_[hidden]> wrote:

>Hi Gene,
>
>this error is often caused by insufficiently synchronized TSCs (time stamp
>counter) of different processors/cores.
>When VT uses the TSC for timing the events (it does that by default), and
>the
>processes switch to another core during execution, it may well happen
>that the
>next recorded time stamp is earlier in time than the last one.
>
>One possibility to avoid this situation is pinning the processes to cores
>- Open
>MPI has functionality for realizing this, see
>http://www.open-mpi.org/faq/?category=tuning#using-paffinity
>
>If this is not feasible, you may use another clock source with VT which
>provides
>global time. To do this, you need to compile another version of VT. Run
>configure as usual, then edit config.h: replace the value of
>#define TIMER [...]
>with e.g.
>#define TIMER TIMER_CLOCK_GETTIME
>or
>#define TIMER TIMER_GETTIMEOFDAY
>depending on what is available on your system. Be aware that the
>resolution of
>those clocks is not as high as the TSC's.
>
>Have a look inside config.h at the place of the mentioned variables -
>there is
>some documentation there.
>Afterwards, compile and install VT. Using this version, you should not
>encounter
>the errors anymore.
>
>Regards,
> Holger
>
>
>
>
>On 07/06/2012 04:04 PM, Fluder, Eugene wrote:
>> I got the following error running a VT enabled run of AMBER. This was
>>reported
>> in December of 2009 under almost identical conditions but the thread
>>does not
>> contain a resolution. I reran the test with VT_UNIFY=no and it completed
>> normally. The same error occurred when I ran vtunify separately. Any
>>help?
>>
>> Was this ever resolved?
>>
>> Gene
>>
>> [fludee01_at_node7-10 trace_noiox]$ vtunify 8 a
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 99459634, p= 6)
>> vtunify: Error: Could not read events of OTF stream [namestub
>>./a__ufy.tmp id 6]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 105413860, p= 5)
>> vtunify: Error: Could not read events of OTF stream [namestub
>>./a__ufy.tmp id 5]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 103189146, p= 7)
>> vtunify: Error: Could not read events of OTF stream [namestub
>>./a__ufy.tmp id 7]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 100509810, p= 8)
>> vtunify: Error: Could not read events of OTF stream [namestub
>>./a__ufy.tmp id 8]
>> vtunify: An error occurred during unifying events - Terminating ...
>>
>> -- /Eugene M Fluder, Jr, PhD/
>> /Computational Scientist/
>> /Scientific Computing/
>> /
>> /
>> /The Mt. Sinai School of Medicine/
>> /One Gustave L. Levy Place, Box 1498/
>>
>> /New York, NY 10029-6574/
>>
>> /
>> /
>>
>> /T: 212 659 8608/
>>
>> /F: 646 537 8660/
>>
>> /E: eugene.fluder_at_[hidden]/
>>
>> /
>> /
>>
>> //
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>_______________________________________________
>devel mailing list
>devel_at_[hidden]
>http://www.open-mpi.org/mailman/listinfo.cgi/devel