Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Problems with btl openib and MPI_THREAD_MULTIPLE
From: Markus Wittmann (markus.wittmann_at_[hidden])
Date: 2012-11-08 03:23:52


Hi,

OK, that makes it clear.

Thank you for the fast response.

Regards,
Markus

Am 07.11.2012 13:49, schrieb Iliev, Hristo:
> Hello, Markus,
>
> The openib BTL component is not thread-safe. It disables itself when
> the thread support level is MPI_THREAD_MULTIPLE. See this rant from
> one of my colleagues:
>
> http://www.open-mpi.org/community/lists/devel/2012/10/11584.php
>
> A message is shown but only if the library was compiled with
> developer-level debugging.
>
> Open MPI guys, could the debug-level message in
> btl_openib_component.c:btl_openib_component_init() be replaced by a
> help text, e.g. the same way that the help text about the amount of
> registerable memory not being enough is shown. Looks like the case of
> openib being disabled for no apparent reason when MPI_THREAD_MULTIPLE
> is in effect is not isolated to our users only. Or at least could you
> put somewhere in the FAQ an explicit statement that openib is not
> only not thread-safe, but that it would disable itself in a
> multithreaded environment.
>
> Kind regards, Hristo -- Hristo Iliev, Ph.D. -- High Performance
> Computing RWTH Aachen University, Center for Computing and
> Communication Rechen- und Kommunikationszentrum der RWTH Aachen
> Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241 80 24367 --
> Fax/UMS: +49 241 80 624367
>
>> -----Original Message----- From: users-bounces_at_[hidden]
>> [mailto:users-bounces_at_[hidden]] On Behalf Of Markus Wittmann
>> Sent: Wednesday, November 07, 2012 1:14 PM To: users_at_[hidden]
>> Subject: [OMPI users] Problems with btl openib and
>> MPI_THREAD_MULTIPLE
>>
>> Hello,
>>
>> I've compiled Open MPI 1.6.3 with --enable-mpi-thread-multiple
>> -with-tm - with-openib --enable-opal-multi-threads.
>>
>> When I use for example the pingpong benchmark from the Intel MPI
>> Benchmarks, which call MPI_Init the btl openib is used and
>> everything
> works
>> fine.
>>
>> When instead the benchmark calls MPI_Thread_init with
>> MPI_THREAD_MULTIPLE as requested threading level the btl openib
>> fails to load but gives no further hints for the reason:
>>
>> mpirun -v -n 2 -npernode 1 -gmca btl_base_verbose 200 ./imb-
>> tm-openmpi- ts pingpong
>>
>> ... [l0519:08267] select: initializing btl component openib
>> [l0519:08267]
> select:
>> init of component openib returned failure [l0519:08267] select:
>> module openib unloaded ...
>>
>> The question is now, is currently just the support for
>> MPI_THREADM_MULTIPLE missing in the openib module or are there
>> other errors occurring and if so, how to identify them.
>>
>> Attached ist the config.log from the Open MPI build, the ompi_info
>> output and the output of the IMB pingpong bechmarks.
>>
>> As system used were two nodes with:
>>
>> - OpenFabrics 1.5.3 - CentOS release 5.8 (Final) - Linux Kernel
>> 2.6.18-308.11.1.el5 x86_64 - OpenSM 3.3.3
>>
>> [l0519] src > ibv_devinfo hca_id: mlx4_0 transport:
>> InfiniBand (0) fw_ver: 2.7.000 node_guid:
>> 0030:48ff:fff6:31e4 sys_image_guid:
>> 0030:48ff:fff6:31e7 vendor_id: 0x02c9
>> vendor_part_id: 26428 hw_ver:
>> 0xB0 board_id: SM_2122000001000
>> phys_port_cnt: 1 port: 1 state:
>> PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu:
>> 2048 (4) sm_lid: 48 port_lid: 278
>> port_lmc: 0x00
>>
>> Thanks for the help in advance.
>>
>> Regards, Markus
>>
>>
>> -- Markus Wittmann, HPC Services Friedrich-Alexander-Universität
>> Erlangen-Nürnberg Regionales Rechenzentrum Erlangen (RRZE)
>> Martensstrasse 1, 91058 Erlangen, Germany Tel.: +49 9131 85-20104
>> markus.wittmann_at_[hidden] http://www.rrze.fau.de/hpc/
>>
>>
>> _______________________________________________ users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Markus Wittmann, HPC Services
Friedrich-Alexander-Universität Erlangen-Nürnberg
Regionales Rechenzentrum Erlangen (RRZE)
Martensstrasse 1, 91058 Erlangen, Germany
Tel.: +49 9131 85-20104
markus.wittmann_at_[hidden]
http://www.rrze.fau.de/hpc/