Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Problems with btl openib and MPI_THREAD_MULTIPLE
From: Ralph Castain (rhc_at_[hidden])
Date: 2012-11-07 08:55:43


Yes, we definitely should do so - will put it on the Trac system so it gets
done.

Thanks - and sorry it wasn't already there.

On Wed, Nov 7, 2012 at 4:49 AM, Iliev, Hristo <iliev_at_[hidden]>wrote:

> Hello, Markus,
>
> The openib BTL component is not thread-safe. It disables itself when the
> thread support level is MPI_THREAD_MULTIPLE. See this rant from one of my
> colleagues:
>
> http://www.open-mpi.org/community/lists/devel/2012/10/11584.php
>
> A message is shown but only if the library was compiled with
> developer-level
> debugging.
>
> Open MPI guys, could the debug-level message in
> btl_openib_component.c:btl_openib_component_init() be replaced by a help
> text, e.g. the same way that the help text about the amount of registerable
> memory not being enough is shown. Looks like the case of openib being
> disabled for no apparent reason when MPI_THREAD_MULTIPLE is in effect is
> not
> isolated to our users only. Or at least could you put somewhere in the FAQ
> an explicit statement that openib is not only not thread-safe, but that it
> would disable itself in a multithreaded environment.
>
> Kind regards,
> Hristo
> --
> Hristo Iliev, Ph.D. -- High Performance Computing
> RWTH Aachen University, Center for Computing and Communication
> Rechen- und Kommunikationszentrum der RWTH Aachen
> Seffenter Weg 23, D 52074 Aachen (Germany)
> Tel: +49 241 80 24367 -- Fax/UMS: +49 241 80 624367
>
> > -----Original Message-----
> > From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]]
> > On Behalf Of Markus Wittmann
> > Sent: Wednesday, November 07, 2012 1:14 PM
> > To: users_at_[hidden]
> > Subject: [OMPI users] Problems with btl openib and MPI_THREAD_MULTIPLE
> >
> > Hello,
> >
> > I've compiled Open MPI 1.6.3 with --enable-mpi-thread-multiple -with-tm -
> > with-openib --enable-opal-multi-threads.
> >
> > When I use for example the pingpong benchmark from the Intel MPI
> > Benchmarks, which call MPI_Init the btl openib is used and everything
> works
> > fine.
> >
> > When instead the benchmark calls MPI_Thread_init with
> > MPI_THREAD_MULTIPLE as requested threading level the btl openib fails to
> > load but gives no further hints for the reason:
> >
> > mpirun -v -n 2 -npernode 1 -gmca btl_base_verbose 200 ./imb- tm-openmpi-
> > ts pingpong
> >
> > ...
> > [l0519:08267] select: initializing btl component openib [l0519:08267]
> select:
> > init of component openib returned failure [l0519:08267] select: module
> > openib unloaded ...
> >
> > The question is now, is currently just the support for
> > MPI_THREADM_MULTIPLE missing in the openib module or are there other
> > errors occurring and if so, how to identify them.
> >
> > Attached ist the config.log from the Open MPI build, the ompi_info output
> > and the output of the IMB pingpong bechmarks.
> >
> > As system used were two nodes with:
> >
> > - OpenFabrics 1.5.3
> > - CentOS release 5.8 (Final)
> > - Linux Kernel 2.6.18-308.11.1.el5 x86_64
> > - OpenSM 3.3.3
> >
> > [l0519] src > ibv_devinfo
> > hca_id: mlx4_0
> > transport: InfiniBand (0)
> > fw_ver: 2.7.000
> > node_guid: 0030:48ff:fff6:31e4
> > sys_image_guid: 0030:48ff:fff6:31e7
> > vendor_id: 0x02c9
> > vendor_part_id: 26428
> > hw_ver: 0xB0
> > board_id: SM_2122000001000
> > phys_port_cnt: 1
> > port: 1
> > state: PORT_ACTIVE (4)
> > max_mtu: 2048 (4)
> > active_mtu: 2048 (4)
> > sm_lid: 48
> > port_lid: 278
> > port_lmc: 0x00
> >
> > Thanks for the help in advance.
> >
> > Regards,
> > Markus
> >
> >
> > --
> > Markus Wittmann, HPC Services
> > Friedrich-Alexander-Universität Erlangen-Nürnberg Regionales
> > Rechenzentrum Erlangen (RRZE) Martensstrasse 1, 91058 Erlangen, Germany
> > Tel.: +49 9131 85-20104
> > markus.wittmann_at_[hidden]
> > http://www.rrze.fau.de/hpc/
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>