I'm seeing some sporadic strange behavior in one of our MPI codes.  Here are selected portions of the output:

-----------------------------------------------------------------------
|   |   |im |jm |km |  phi0   |         | iter | sync |mcalc |        |
|grp|itn|loc|loc|loc|Max Error|   NSR   |t(sec)|t(sec)|t(sec)| sysbal |
-----------------------------------------------------------------------
1   2   1   1   9 1.000E+00 1.000E+00 16.789 15.923  0.079 1.00E+00
1   3   1   1   5 1.000E+00 1.000E+00 16.800 15.935  0.078 1.00E+00
1   4   1   1   1 1.000E+00 1.000E+00 17.500 15.906  0.079 1.00E+00
...
11   7  18 118  84 1.485E-01 1.117E+00 16.600 15.929  0.077 1.00E+00
11   8  20 124  84 1.516E-01 1.021E+00 16.600 15.929  0.077 1.00E+00
11   9  21 127  86 1.596E-01 1.053E+00  1.253  0.450  0.083 1.00E+00
11  10   7 131  88 1.290E-01 8.083E-01  0.808  0.014  0.272 1.00E+00
11  11   7 131  85 8.267E-02 6.408E-01  1.000  0.002  0.262 1.00E+00
...
101  10  25 111  77 5.690E-02 8.179E-01  0.480  0.023  0.087 1.00E+00
101  11  32 113  77 4.782E-02 8.404E-01  0.479  0.023  0.087 1.00E+00
101  12  37 116  79 4.330E-02 9.055E-01  0.479  0.023  0.087 1.00E+00

This is an iterative calculation.  The critical quantity of interest is "iter t(sec)", which is the time per iteration.  (The other "t(sec)" quantities are subsets of "iter t(sec)".)  Between "grp" 1 and 111, the calculation is not becoming appreciably more or less difficult, yet there is a factor of ~30 difference in performance between the beginning and the end.  This problem does not appear all of the time.  In many cases, performance is good throughout the entire calculation.  ("Good", here, is being defined as what is seen in grp 101 above, which is roughly what I expect to be seeing.)  However, when the problem does appear, it seems to mysteriously go away after grinding through the calculation for a while.

Has anyone ever seen behavior like this?  Any thoughts as to what could be causing it?

I tried to recompile the code with mpif90-vt and mpicc-vt, in hopes that the vampirtrace outputs might shine some light as to the true nature of the problem.  After recompiling, the code complains:

[lx102:15254] *** An error occurred in MPI_Cart_create
[lx102:15254] *** on communicator MPI_COMM_WORLD
[lx102:15254] *** MPI_ERR_ARG: invalid argument of some other kind
[lx102:15254] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)

...and then crashes out before doing anything useful.  My understanding is that I only need to use the -vt compiler wrappers, and it will automatically "instrument" my code.  Is there something else I should be doing?

Thanks
Greg