Is this a TCP-based cluster?

yes

If so, do you have multiple IP addresses on your frontend machine?
Check out these two FAQ entries to see if they help:

http://www.open-mpi.org/faq/?category=tcp#tcp-routability
http://www.open-mpi.org/faq/?category=tcp#tcp-selection

ok, using the internal interfaces only fixed the problem.

it is a little confusing that when this happens, one machine would make
it past the barrier, and the others would not.

thanks Jeff!

--tim