Matt,
Can you please run " cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/* " on your d2-ib,d3-ib.
I would like to check the partition configuration.
Ohh, BTW I see that the command line in previous email was wrong,
Please use follow command line (the parameter name should be "btl_openib_ib_pkey_val" for ompi-1.2.6 and my patch accepts HEX/DEC values):
/opt/openmpi-ib/1.2.6/bin/mpirun -np 2 -H d2-ib,d3-ib -mca btl openib,self -mca btl_openib_ib_pkey_val 0x8109 /cluster/pallas/x86_64-ib/IMB-MPI1
Ompi 1.2.6 version should work ok with this patch.
Thanks,
Pasha
Matt Burgess wrote:
Pasha,
Thanks for the patch. Unfortunately, it doesn't seem like that fixed the problem. I realized earlier I didn't mention what version of OpenMPI I was trying - it's 1.2.6. <http://1.2.6.> Should I be trying 1.2.7 with this patch?
Thanks,
Matt
2008/10/7 Pavel Shamis (Pasha) <pasha@dev.mellanox.co.il <mailto:pasha@dev.mellanox.co.il>>
Matt,
Can you please try attached patch ? I guess it will resolve this
issue.
Thanks,
Pasha
Matt Burgess wrote:
Lenny,
Thanks for the info. It doesn't seem to be be working still.
My command line is:
/opt/openmpi-ib/1.2.6/bin/mpirun -np 2 -H d2-ib,d3-ib -mca btl
openib,self -mca btl_openib_of_pkey_val 33033
/cluster/pallas/x86_64-ib/IMB-MPI1
I don't have a "/sys/class/infiniband/mthca0/ports/1/pkeys/"
but I do have "/sys/class/infiniband/mlx4_0/ports/1/pkeys/".
It's contents are:
0 106 114 122 16 24 32 40 49 57 65 73 81
9 98
1 107 115 123 17 25 33 41 5 58 66 74 82
90 99
10 108 116 124 18 26 34 42 50 59 67 75 83
91 100 109 117 125 19 27 35 43 51 6 68 76 84 92 101 11 118 126 2 28 36 44 52 60
69 77 85 93 102 110 119 127 20 29 37 45 53 61 7 78 86 94 103 111 12 13 21 3 38
46 54 62 70 79 87 95 104 112 120 14 22 30 39 47 55 63 71 8 88 96 105 113 121 15
23 31 4 48 56 64 72 80 89 97
We aren't using the opensm, but voltaire's SM on a 2012 switch.
Thanks again,
Matt
On Tue, Oct 7, 2008 at 9:37 AM, Lenny Verkhovsky
<lenny.verkhovsky@gmail.com
<mailto:lenny.verkhovsky@gmail.com>
<mailto:lenny.verkhovsky@gmail.com
<mailto:lenny.verkhovsky@gmail.com>>> wrote:
Hi Matt,
It seems that the right way to do it is the fallowing:
-mca btl openib,self -mca btl_openib_ib_pkey_val 33033
when the value is a decimal number of the pkey, in your case
0x8109 = 33033, and no need for btl_openib_ib_pkey_ix value.
ex.
mpirun -np 2 -H witch2,witch3 -mca btl openib,self -mca
btl_openib_ib_pkey_val 32769 ./mpi_p1_4_1_2 -t lt
LT (2) (size min max avg) 1 3.511429 3.511429 3.511429
if it's not working check cat
/sys/class/infiniband/mthca0/ports/1/pkeys/* for pkeys ans SM,
maybe it's a setup.
Pasha is currently checking this issue.
Best regards,
Lenny.
On 10/7/08, *Jeff Squyres* <jsquyres@cisco.com
<mailto:jsquyres@cisco.com><mailto:jsquyres@cisco.com <mailto:jsquyres@cisco.com>>> wrote:<mailto:jsquyres@cisco.com
FWIW, if this configuration is for all of your users, you
might want to specify these MCA params in the default MCA
param file, or the environment, ...etc. Just so that you
don't have to specify it on every mpirun command line.
See
http://www.open-mpi.org/faq/?category=tuning#setting-mca-params.
On Oct 7, 2008, at 5:43 AM, Lenny Verkhovsky wrote:
Sorry, misunderstood the question,
thanks for Pasha the right command line will be
-mca btl openib,self -mca btl_openib_of_pkey_val 0x8109
-mca btl_openib_of_pkey_ix 1
ex.
#mpirun -np 2 -H witch2,witch3 -mca btl openib,self
-mca
btl_openib_of_pkey_val 0x8001 -mca
btl_openib_of_pkey_ix 1
./mpi_p1_4_TRUNK -t lt
LT (2) (size min max avg) 1 3.443480 3.443480 3.443480
Best regards
Lenny.
On 10/6/08, Jeff Squyres <jsquyres@cisco.com
<mailto:jsquyres@cisco.com><mailto:users@open-mpi.org <mailto:users@open-mpi.org>>
<mailto:jsquyres@cisco.com>>> wrote: On Oct 5, 2008, at
1:22 PM, Lenny Verkhovsky wrote:
you should probably use -mca tcp,self -mca
btl_openib_if_include ib0.8109
Really? I thought we only took OpenFabrics device
names
in the openib_if_include MCA param...? It looks like
ib0.8109 is an IPoIB device name.
Lenny.
On 10/3/08, Matt Burgess <burgess.matt@gmail.com
<mailto:burgess.matt@gmail.com>
<mailto:burgess.matt@gmail.com
<mailto:burgess.matt@gmail.com>>> wrote:
Hi,
I'm trying to get openmpi working over openib
partitions.
On this cluster, the partition number is 0x109. The ib
interfaces are pingable over the appropriate ib0.8109
interface:
d2:/opt/openmpi-ib # ifconfig ib0.8109
ib0.8109 Link encap:UNSPEC HWaddr
80-00-00-4A-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:10.21.48.2 <http://10.21.48.2>
<http://10.21.48.2>
Bcast:10.21.255.255 <http://10.21.255.255>
<http://10.21.255.255>
Mask:255.255.0.0 <http://255.255.0.0>
<http://255.255.0.0>
inet6 addr: fe80::202:c902:26:ca01/64
Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520
Metric:1
RX packets:16811 errors:0 dropped:0
overruns:0 frame:0
TX packets:15848 errors:0 dropped:1 overruns:0
carrier:0
collisions:0 txqueuelen:256
RX bytes:102229428 (97.4 Mb) TX
bytes:102324172
(97.5 Mb)
I have tried the following:
/opt/openmpi-ib/1.2.6/bin/mpirun -np 2 -machinefile
machinefile -mca btl openib,self -mca
btl_openib_max_btls
1 -mca btl_openib_ib_pkey_val 0x8109 -mca
btl_openib_ib_pkey_ix 1
/cluster/pallas/x86_64-ib/IMB-MPI1
but I just get a RETRY EXCEEDED ERROR. Is there a MCA
parameter I am missing?
I was successful using tcp only:
/opt/openmpi-ib/1.2.6/bin/mpirun -np 2 -machinefile
machinefile -mca btl tcp,self -mca
btl_openib_max_btls 1
-mca btl_openib_ib_pkey_val 0x8109
/cluster/pallas/x86_64-ib/IMB-MPI1
Thanks,
Matt Burgess
_______________________________________________
users mailing list
users@open-mpi.org <mailto:users@open-mpi.org><mailto:users@open-mpi.org <mailto:users@open-mpi.org>>
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
users@open-mpi.org <mailto:users@open-mpi.org><mailto:users@open-mpi.org <mailto:users@open-mpi.org>>
http://www.open-mpi.org/mailman/listinfo.cgi/users
-- Jeff Squyres
Cisco Systems
_______________________________________________
users mailing list
users@open-mpi.org <mailto:users@open-mpi.org>devel@open-mpi.org <mailto:devel@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/users
-- Jeff Squyres
Cisco Systems
------------------------------------------------------------------------
_______________________________________________
devel mailing listdevel@open-mpi.org <mailto:devel@open-mpi.org>
http://www.open-mpi.org/mailman/listinfo.cgi/devel
-- --
Pavel Shamis (Pasha)
Mellanox Technologies LTD.
Index: ompi/mca/btl/openib/btl_openib_component.c
===================================================================
--- ompi/mca/btl/openib/btl_openib_component.c (revision 19490)
+++ ompi/mca/btl/openib/btl_openib_component.c (working copy)
@@ -558,7 +558,7 @@ static int init_one_hca(opal_list_t *btl
goto dealloc_pd;
}
- ret = OMPI_SUCCESS;
+ ret = OMPI_SUCCESS;
/* Note ports are 1 based hence j = 1 */
for(i = 1; i <= hca->ib_dev_attr.phys_port_cnt; i++){
struct ibv_port_attr ib_port_attr;
@@ -580,7 +580,7 @@ static int init_one_hca(opal_list_t *btl
uint16_t pkey,j;
for (j=0; j < hca->ib_dev_attr.max_pkeys; j++) {
ibv_query_pkey(hca->ib_dev_context, i, j, &pkey);
- pkey=ntohs(pkey);
+ pkey=ntohs(pkey) & 0x7fff;
if(pkey == mca_btl_openib_component.ib_pkey_val){
ret = init_one_port(btl_list, hca, i, j,
&ib_port_attr);
break;
Index: ompi/mca/btl/openib/btl_openib_ini.c
===================================================================
--- ompi/mca/btl/openib/btl_openib_ini.c (revision 19490)
+++ ompi/mca/btl/openib/btl_openib_ini.c (working copy)
@@ -90,8 +90,6 @@ static int parse_line(parsed_section_val
static void reset_section(bool had_previous_value,
parsed_section_values_t *s);
static void reset_values(ompi_btl_openib_ini_values_t *v);
static int save_section(parsed_section_values_t *s);
-static int intify(char *string);
-static int intify_list(char *str, uint32_t **values, int *len);
static inline void show_help(const char *topic);
@@ -364,14 +362,14 @@ static int parse_line(parsed_section_val
all whitespace at the beginning and ending of the value. */
if (0 == strcasecmp(key_buffer, "vendor_id")) {
- if (OMPI_SUCCESS != (ret = intify_list(value,
&sv->vendor_ids,
+ if (OMPI_SUCCESS != (ret =
ompi_btl_openib_ini_intify_list(value, &sv->vendor_ids,
&sv->vendor_ids_len))) {
return ret;
}
}
else if (0 == strcasecmp(key_buffer, "vendor_part_id")) {
- if (OMPI_SUCCESS != (ret = intify_list(value,
&sv->vendor_part_ids,
+ if (OMPI_SUCCESS != (ret =
ompi_btl_openib_ini_intify_list(value, &sv->vendor_part_ids,
&sv->vendor_part_ids_len))) {
return ret;
}
@@ -379,13 +377,13 @@ static int parse_line(parsed_section_val
else if (0 == strcasecmp(key_buffer, "mtu")) {
/* Single value */
- sv->values.mtu = (uint32_t) intify(value);
+ sv->values.mtu = (uint32_t)
ompi_btl_openib_ini_intify(value);
sv->values.mtu_set = true;
}
else if (0 == strcasecmp(key_buffer, "use_eager_rdma")) {
/* Single value */
- sv->values.use_eager_rdma = (uint32_t) intify(value);
+ sv->values.use_eager_rdma = (uint32_t)
ompi_btl_openib_ini_intify(value);
sv->values.use_eager_rdma_set = true;
}
@@ -547,7 +545,7 @@ static int save_section(parsed_section_v
/*
* Do string-to-integer conversion, for both hex and decimal numbers
*/
-static int intify(char *str)
+int ompi_btl_openib_ini_intify(char *str)
{
while (isspace(*str)) {
++str;
@@ -568,7 +566,7 @@ static int intify(char *str)
/*
* Take a comma-delimited list and infity them all
*/
-static int intify_list(char *value, uint32_t **values, int *len)
+int ompi_btl_openib_ini_intify_list(char *value, uint32_t
**values, int *len)
{
char *comma;
char *str = value;
@@ -584,7 +582,7 @@ static int intify_list(char *value, uint
if (NULL == *values) {
return OMPI_ERR_OUT_OF_RESOURCE;
}
- *values[0] = (uint32_t) intify(str);
+ *values[0] = (uint32_t) ompi_btl_openib_ini_intify(str);
*len = 1;
} else {
/* If we found a comma, loop over all the values. Be a
@@ -594,7 +592,7 @@ static int intify_list(char *value, uint
do {
*comma = '\0';
*values = realloc(*values, sizeof(uint32_t) * (*len + 2));
- (*values)[*len] = (int32_t) intify(str);
+ (*values)[*len] = (int32_t)
ompi_btl_openib_ini_intify(str);
++(*len);
str = comma + 1;
comma = strchr(str, ',');
@@ -602,7 +600,7 @@ static int intify_list(char *value, uint
/* Get the last value (i.e., the value after the last
comma, because it won't have been snarfed in the
loop) */
- (*values)[*len] = (uint32_t) intify(str);
+ (*values)[*len] = (uint32_t) ompi_btl_openib_ini_intify(str);
++(*len);
}
Index: ompi/mca/btl/openib/btl_openib_ini.h
===================================================================
--- ompi/mca/btl/openib/btl_openib_ini.h (revision 19490)
+++ ompi/mca/btl/openib/btl_openib_ini.h (working copy)
@@ -49,6 +49,9 @@ extern "C" {
*/
int ompi_btl_openib_ini_finalize(void);
+ int ompi_btl_openib_ini_intify(char *string);
+ int ompi_btl_openib_ini_intify_list(char *str, uint32_t
**values, int *len);
+
#if defined(c_plusplus) || defined(__cplusplus)
}
#endif
Index: ompi/mca/btl/openib/btl_openib_mca.c
===================================================================
--- ompi/mca/btl/openib/btl_openib_mca.c (revision 19490)
+++ ompi/mca/btl/openib/btl_openib_mca.c (working copy)
@@ -27,6 +27,7 @@
#include "opal/mca/base/mca_base_param.h"
#include "btl_openib.h"
#include "btl_openib_mca.h"
+#include "btl_openib_ini.h"
/*
* Local flags
@@ -97,7 +98,7 @@ static inline int reg_int(const char* pa
*/
int btl_openib_register_mca_params(void)
{
- char *msg, *str;
+ char *msg, *str, *pkey;
int ival, ival2, ret, tmp;
ret = OMPI_SUCCESS;
@@ -192,13 +193,15 @@ int btl_openib_register_mca_params(void)
0, &ival, REGINT_GE_ZERO));
mca_btl_openib_component.ib_pkey_ix = (uint32_t) ival;
- CHECK(reg_int("ib_pkey_val", "InfiniBand pkey value"
+ CHECK(reg_string("ib_pkey_val", "InfiniBand pkey value"
"(must be > 0 and < 0xffff)",
- 0, &ival, REGINT_GE_ZERO));
- if (ival > 0xffff) {
+ "0", &pkey, 0));
+ mca_btl_openib_component.ib_pkey_val =
ompi_btl_openib_ini_intify(pkey) & 0x7fff;
+ if (mca_btl_openib_component.ib_pkey_val > 0xffff ||
+ mca_btl_openib_component.ib_pkey_val < 0) {
ret = OMPI_ERR_BAD_PARAM;
}
- mca_btl_openib_component.ib_pkey_val = (uint32_t) ival;
+ free(pkey);
CHECK(reg_int("ib_psn", "InfiniBand packet sequence starting
number "
"(must be >= 0)",
_______________________________________________
devel mailing list
--
--
Pavel Shamis (Pasha)
Mellanox Technologies LTD.