Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2007-06-13 11:03:09


Hey Gleb --

Can you explain the rationale for this change? Is there a reason why
the bandwidths reported by the IBV API are not sufficient? Are you
trying to do creative things with multi-LID scenarios (perhaps QOS-
like things)? If so, this looks like a good idea, but I'm not sure
that MCA params are the right vehicle for it.

I'm not particularly fond of creating variable MCA parameters after
the btl open call because they won't show up in ompi_info. Should we
do something else if you want to override bandwidths, perhaps
something similar to the HCA params file? If you recall, we made the
HCA params stuff a file because of exactly this issue: we didn't want
variable MCA parameters and we didn't want to have to make the user
type out a potentially long set of complex MCA parameters that were
dependent upon the exact environment that the user was running in.

On Jun 13, 2007, at 8:47 AM, gleb_at_[hidden] wrote:

> Author: gleb
> Date: 2007-06-13 08:47:38 EDT (Wed, 13 Jun 2007)
> New Revision: 15041
> URL: https://svn.open-mpi.org/trac/ompi/changeset/15041
>
> Log:
> Allow to configure bandwidth and latency with finer granularity.
>
> Set bandwidth for all ports of mthca0:
> --mca btl_openib_bandwidth_mthca0 1000
>
> Set bandwidth for port 1 of mthca1:
> --mca btl_openib_bandwidth_mthca1:1 1000
>
> Set latency for port 2 lid 123 on mthca0:
> --mca btl_openib_latency_mthca0:2:123 20
>
> Text files modified:
> trunk/ompi/mca/btl/openib/btl_openib_component.c | 43 ++++++++
> ++++++++++++++++++++++++++++++++
> 1 files changed, 43 insertions(+), 0 deletions(-)
>
> Modified: trunk/ompi/mca/btl/openib/btl_openib_component.c
> ======================================================================
> ========
> --- trunk/ompi/mca/btl/openib/btl_openib_component.c (original)
> +++ trunk/ompi/mca/btl/openib/btl_openib_component.c 2007-06-13
> 08:47:38 EDT (Wed, 13 Jun 2007)
> @@ -299,6 +299,14 @@
> openib_reg->mr = NULL;
> return OMPI_SUCCESS;
> }
> +static inline int param_register_int(const char* param_name, int
> default_value)
> +{
> + int param_value = default_value;
> + int id = mca_base_param_register_int("btl", "openib",
> param_name, NULL,
> + default_value);
> + mca_base_param_lookup_int(id, &param_value);
> + return param_value;
> +}
>
> static int init_one_port(opal_list_t *btl_list,
> mca_btl_openib_hca_t *hca,
> uint8_t port_num, uint16_t pkey_index,
> @@ -331,6 +339,7 @@
> for(lid = ib_port_attr->lid;
> lid < ib_port_attr->lid + lmc; lid++){
> for(i = 0; i < mca_btl_openib_component.btls_per_lid; i++){
> + char param[40];
> openib_btl = malloc(sizeof(mca_btl_openib_module_t));
> if(NULL == openib_btl) {
> BTL_ERROR(("Failed malloc: %s:%d\n", __FILE__,
> __LINE__));
> @@ -353,6 +362,40 @@
> openib_btl->ib_reg[MCA_BTL_TAG_BTL].cbfunc =
> btl_openib_control;
> openib_btl->ib_reg[MCA_BTL_TAG_BTL].cbdata = NULL;
>
> + /* Check bandwidth configured for this HCA */
> + sprintf(param, "bandwidth_%s", ibv_get_device_name(hca-
> >ib_dev));
> + openib_btl->super.btl_bandwidth =
> + param_register_int(param, openib_btl-
> >super.btl_bandwidth);
> +
> + /* Check bandwidth configured for this HCA/port */
> + sprintf(param, "bandwidth_%s:%d", ibv_get_device_name
> (hca->ib_dev),
> + port_num);
> + openib_btl->super.btl_bandwidth =
> + param_register_int(param, openib_btl-
> >super.btl_bandwidth);
> +
> + /* Check bandwidth configured for this HCA/port/LID */
> + sprintf(param, "bandwidth_%s:%d:%d",
> + ibv_get_device_name(hca->ib_dev), port_num, lid);
> + openib_btl->super.btl_bandwidth =
> + param_register_int(param, openib_btl-
> >super.btl_bandwidth);
> +
> + /* Check latency configured for this HCA */
> + sprintf(param, "latency_%s", ibv_get_device_name(hca-
> >ib_dev));
> + openib_btl->super.btl_latency =
> + param_register_int(param, openib_btl-
> >super.btl_latency);
> +
> + /* Check latency configured for this HCA/port */
> + sprintf(param, "latency_%s:%d", ibv_get_device_name
> (hca->ib_dev),
> + port_num);
> + openib_btl->super.btl_latency =
> + param_register_int(param, openib_btl-
> >super.btl_latency);
> +
> + /* Check latency configured for this HCA/port/LID */
> + sprintf(param, "latency_%s:%d:%d", ibv_get_device_name
> (hca->ib_dev),
> + port_num, lid);
> + openib_btl->super.btl_latency =
> + param_register_int(param, openib_btl-
> >super.btl_latency);
> +
> /* Auto-detect the port bandwidth */
> if (0 == openib_btl->super.btl_bandwidth) {
> /* To calculate the bandwidth available on this port,
> _______________________________________________
> svn-full mailing list
> svn-full_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/svn-full

-- 
Jeff Squyres
Cisco Systems