Hi Cristobal

Does it run only on the head node alone?
(Fuego? Agua? Acatenango?)
Try to put only the head node on the hostfile and execute with mpiexec.

This may help sort out what is going on.
Hopefully it will run on the head node.

Also, do you have Infinband connecting the nodes?
The error messages refer to the openib btl (i.e. Infiniband),
and complains of

"perhaps a missing symbol, or compiled for a different

version of Open MPI?".
It sounds as a mixup of versions/builds.

Did you configure/build OpenMPI from source, or did you install
it with apt-get?
It may be easier/less confusing to install from source.
If you did, what configure options did you use?

Also, as for the OpenMPI runtime environment,
it is not enough to set it on
the command line, because it will be effective only on the head node.
You need to either add them to the PATH and LD_LIBRARY_PATH
on your .bashrc/.cshrc files (assuming these files and your home directory are *also* shared with the nodes via NFS),
or use the --prefix option of mpiexec to point to the OpenMPI main directory.

Needless to say, you need to check and ensure that the OpenMPI directory (and maybe your home directory, and your work directory) is (are)
really mounted on the nodes.

I hope this helps,

Gus Correa

Update: i just reinstalled openMPI, with the same parameters, and it seems that the problem has gone, i couldnt test entirely but when i get back to lab ill confirm.