On Aug 12, 2009, at  19:09 PM, Ralph Castain wrote:

Hmmm...well, I'm going to ask our TCP friends for some help here.

Meantime, I do see one thing that stands out. Port 4 is an awfully low port number that usually sits in the reserved range. I checked the /etc/services file on my Mac, and it was commented out as unassigned, which should mean it was okay.

Still, that is an unusual number. The default minimum port number is 1024, so I'm puzzled how you wound up down there. Of course, could just be an error in the print statement, but let's try moving it to be safe? Set

-mca btl_tcp_port_min_v4 36900 -mca btl_tcp_port_range_v4 32
and see what happens.

What happens is that everything works now!  Both connectivity_c and the MITgcm.  I haven't tried under torque yet, but lets declare an openMPI victory at this point.  

On Aug 13, 2009, at  8:28 AM, Jeff Squyres wrote:

Agreed -- ports 4 and 260 should be in the reserved ports range.  Are you running as root, perchance?

Errrr, no, but yes.  My user account has admin privledges.  A sloppy workstation OS X habit I now regret propagating to my cluster. I'm sorry to not mention it earlier as possibly relevant.

As a suggestion, btl_base_verbose could be mentioned as a good debugging tool in the troubleshooting section of the FAQ.  Its on the page to do with tcp, which I admit I should have read as soon as I realized there was a communication issue, but having it in the troubleshooting section would be helpful too.  i.e. maybe a more erudite version of: 

Checking connections between nodes:

Sometimes the configuration of a cluster makes it impossible for nodes to communicate properly.  To debug this it helps to include  

--mca btl_base_verbose 30 as a command line argument (see http://www.open-mpi.org/faq/?category=tcp for more information).  The program example/connectivity_c.c is also a useful minimal program for testing communication on the cluster.  

Thanks again for everyone's help, particularly Ralph, Jeff and Gus.  

Cheers,  Jody