I was debugging something that had the same properties.
I was following the comm struct for corruption-on-free in the read
phase of a write-then-read test, in mpich-1.2.7p1 (and mpich-1.2.6).
It only manifested on 64bit nodes, and only when compiled on 64bit
nodes. The binary when compiled on 32bit and run on 64bit did not
My smallest test case was two machines, and failure was intermittant,
both in reported stack trace, and upon size of the buffers being used.
The app that triggered this was a benchmark from one of the (US) labs
used for benchmarking i/o on a storage product.