Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] IBCM and ordering of OpenFabrics devices
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-07-17 07:17:24


I have a case where ib_cm_open_device() is failing for an odd reason:
I have 12 servers that contain both HCAs and iWARP NICs. In most
cases, everything is fine. But one one of these servers, IBCM refuses
to work -- ib_cm_open_device() fails with the following:

libibcm: unable to open /dev/infiniband/ucm1

Looking closer, this device does, indeed, exist:

[4:01] svbu-mpi044:~/mpi % ls -l /dev/infiniband/ucm*
crw-rw-rw- 1 root root 231, 224 Jul 16 04:30 /dev/infiniband/ucm0
crw-rw-rw- 1 root root 231, 225 Jul 16 04:30 /dev/infiniband/ucm1
[4:08] svbu-mpi044:~/mpi %

Granted; I had to create these devices manually because they are not
created automatically for me upon boot in RHEL4U4 and U6. These
device major/minor numbers work fine for me on all my other servers.

So what's different between the 11 machines that work and the 1 that
doesn't? It seems that the kernel ordering of devices is what is
different. On most of the machines:

[4:10] svbu-mpi045:~ % ibv_devinfo
hca_id: mlx4_0
         fw_ver: 2.3.000
         node_guid: 0002:c903:0000:036c
         sys_image_guid: 0002:c903:0000:036f
         vendor_id: 0x02c9
         vendor_part_id: 25418
         hw_ver: 0xA0
         board_id: MT_04A0110002
         phys_port_cnt: 2
                 port: 1
                         state: PORT_ACTIVE (4)
                         max_mtu: 2048 (4)
                         active_mtu: 2048 (4)
                         sm_lid: 2
                         port_lid: 7
                         port_lmc: 0x00

                 port: 2
                         state: PORT_ACTIVE (4)
                         max_mtu: 2048 (4)
                         active_mtu: 2048 (4)
                         sm_lid: 2
                         port_lid: 52
                         port_lmc: 0x00

hca_id: nes0
         node_guid: 0012:5502:63c0:0000
         sys_image_guid: 0012:5502:63c0:0000
         vendor_id: 0x0000
         vendor_part_id: 0
         hw_ver: 0x5
         board_id: NES020 Board ID
         phys_port_cnt: 1
                 port: 1
                         state: PORT_ACTIVE (4)
                         max_mtu: 2048 (4)
                         active_mtu: 2048 (4)
                         sm_lid: 0
                         port_lid: 1
                         port_lmc: 0x00

But on this one problematic machine:

4:10] svbu-mpi044:~/mpi % ibv_devinfo
hca_id: nes0
         node_guid: 0012:5502:63b8:0000
         sys_image_guid: 0012:5502:63b8:0000
         vendor_id: 0x0000
         vendor_part_id: 0
         hw_ver: 0x5
         board_id: NES020 Board ID
         phys_port_cnt: 1
                 port: 1
                         state: PORT_ACTIVE (4)
                         max_mtu: 2048 (4)
                         active_mtu: 2048 (4)
                         sm_lid: 0
                         port_lid: 1
                         port_lmc: 0x00

hca_id: mlx4_0
         fw_ver: 2.3.000
         node_guid: 0002:c903:0000:03b0
         sys_image_guid: 0002:c903:0000:03b3
         vendor_id: 0x02c9
         vendor_part_id: 25418
         hw_ver: 0xA0
         board_id: MT_04A0110002
         phys_port_cnt: 2
                 port: 1
                         state: PORT_ACTIVE (4)
                         max_mtu: 2048 (4)
                         active_mtu: 2048 (4)
                         sm_lid: 2
                         port_lid: 6
                         port_lmc: 0x00

                 port: 2
                         state: PORT_ACTIVE (4)
                         max_mtu: 2048 (4)
                         active_mtu: 2048 (4)
                         sm_lid: 2
                         port_lid: 136
                         port_lmc: 0x00

Notice that the ordering is different.

So I'm not enough of a kernel guy to know where the problem is:

1. Technically, mlx4_0 is the first IB device. Should it therefore be
using ucm0? I.e., is libibcm wrong for trying to use ucm1? (note
that OMPI's openib BTL is currently replicating the logic from libibcm
to check for the Right ucm* file so that we can silently fail before
ib_cm_open_device() fails with a warning message -- so if libibcm's
logic to find the Right ucm* file changes, we'll also need to change
OMPI's logic to mirror it. OMPI's logic becomes moot in newer libibcm
versions where Sean removed the warning message, though).

2. Or are my major/minor numbers incorrect for the devices that I
created manually? If the major/minor device numbers were created by
the OS upon bootup (as they should be -- there's an open OpenFabrics
bugzilla ticket about this), would they be correct?

-- 
Jeff Squyres
Cisco Systems