As far as I can tell, both the PATH and LD_LIBRARY_PATH are set correctly. I've tried with the full path to the mpirun executable and using the --prefix command line option. Neither works. The debug output seems to contain a lot of system specific information (IPs, usernames and such), which I'm a little reticent to share on an open mailing list. As such I've censored that information. Hopefully the rest is still of use. One thing I did notice is that Open MPI seems to want to use sh instead of bash (which is the shell I use). Is that what's meant by the following lines?
[gamma2.censored_domain:22554] pls:rsh: local csh: 0, local sh: 1
[gamma2.censored_domain:22554] pls:rsh: assuming same remote shell as local shell
[gamma2.censored_domain:22554] pls:rsh: remote csh: 0, remote sh: 1
If so is there a way to make it use bash?