Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] VampirTrace: time not increasing
From: Fluder, Eugene (eugene.fluder_at_[hidden])
Date: 2012-07-06 11:33:49


Holger,

Thanks. I appreciate the detail.

Gene
-- Eugene M Fluder, Jr, PhD
        Computational Scientist
        Scientific Computing

        The Mt. Sinai School of Medicine
        One Gustave L. Levy Place, Box 1498
        New York, NY 10029-6574

        T: 212 659 8608
        F: 646 537 8660
        E: eugene.fluder_at_[hidden]

        

On 7/6/12 10:54 AM, "Holger Mickler" <holger.mickler_at_[hidden]> wrote:

>Hi Gene,
>
>this error is often caused by insufficiently synchronized TSCs (time stamp
>counter) of different processors/cores.
>When VT uses the TSC for timing the events (it does that by default), and
>the
>processes switch to another core during execution, it may well happen
>that the
>next recorded time stamp is earlier in time than the last one.
>
>One possibility to avoid this situation is pinning the processes to cores
>- Open
>MPI has functionality for realizing this, see
>http://www.open-mpi.org/faq/?category=tuning#using-paffinity
>
>If this is not feasible, you may use another clock source with VT which
>provides
>global time. To do this, you need to compile another version of VT. Run
>configure as usual, then edit config.h: replace the value of
>#define TIMER [...]
>with e.g.
>#define TIMER TIMER_CLOCK_GETTIME
>or
>#define TIMER TIMER_GETTIMEOFDAY
>depending on what is available on your system. Be aware that the
>resolution of
>those clocks is not as high as the TSC's.
>
>Have a look inside config.h at the place of the mentioned variables -
>there is
>some documentation there.
>Afterwards, compile and install VT. Using this version, you should not
>encounter
>the errors anymore.
>
>Regards,
> Holger
>
>
>
>
>On 07/06/2012 04:04 PM, Fluder, Eugene wrote:
>> I got the following error running a VT enabled run of AMBER. This was
>>reported
>> in December of 2009 under almost identical conditions but the thread
>>does not
>> contain a resolution. I reran the test with VT_UNIFY=no and it completed
>> normally. The same error occurred when I ran vtunify separately. Any
>>help?
>>
>> Was this ever resolved?
>>
>> Gene
>>
>> [fludee01_at_node7-10 trace_noiox]$ vtunify 8 a
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 99459634, p= 6)
>> vtunify: Error: Could not read events of OTF stream [namestub
>>./a__ufy.tmp id 6]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 105413860, p= 5)
>> vtunify: Error: Could not read events of OTF stream [namestub
>>./a__ufy.tmp id 5]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 103189146, p= 7)
>> vtunify: Error: Could not read events of OTF stream [namestub
>>./a__ufy.tmp id 7]
>> OTF ERROR in function OTF_WBuffer_setTimeAndProcess, file:
>>OTF_WBuffer.c, line: 308:
>> time not increasing. (t= 100509810, p= 8)
>> vtunify: Error: Could not read events of OTF stream [namestub
>>./a__ufy.tmp id 8]
>> vtunify: An error occurred during unifying events - Terminating ...
>>
>> -- /Eugene M Fluder, Jr, PhD/
>> /Computational Scientist/
>> /Scientific Computing/
>> /
>> /
>> /The Mt. Sinai School of Medicine/
>> /One Gustave L. Levy Place, Box 1498/
>>
>> /New York, NY 10029-6574/
>>
>> /
>> /
>>
>> /T: 212 659 8608/
>>
>> /F: 646 537 8660/
>>
>> /E: eugene.fluder_at_[hidden]/
>>
>> /
>> /
>>
>> //
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>_______________________________________________
>devel mailing list
>devel_at_[hidden]
>http://www.open-mpi.org/mailman/listinfo.cgi/devel