Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] How are the Open MPI processes spawned?
From: Paul Kapinos (kapinos_at_[hidden])
Date: 2011-11-22 12:10:45


Hello Ralph, hello all.

> No real ideas, I'm afraid. We regularly launch much larger jobs than that using ssh without problem,
I was also able to run a 288-node-job yesterday - the size alone is not
the problem...

> so it is likely something about the local setup of that node that is causing the problem.
> Offhand, it sounds like either the mapper isn't getting things right, or for some reason
> the daemon on 005 isn't properly getting or processing the launch command.
>
> What you could try is adding --display-map to see if the map is being correctly generated.
> If that works, then (using a debug build) try adding
--leave-session-attached and see if
> any daemons are outputting an error.
>
> You could add -mca odls_base_verbose 5 --leave-session-attached to your cmd line.
> You'll see debug output from each daemon as it receives and processes
> the launch command. See if the daemon on 005 is behaving differently than the others.

I've tried the options.
The map seem to be correctly build, also the output if the daemons seem
to be the same (see helloworld.txt)

> You should also try putting that long list of nodes in a hostfile - see if that makes a difference.
> It will process the nodes thru a different code path, so if there is
some problem in --host,
> this will tell us.

No, with the host file instead of host list on command line the
behaviour is the same.

But, I just found out that the 1.4.3 does *not* hang on this
constellation. The next thing I will try will be the installation of
1.5.4 :o)

Best,

Paul

P.S. started:

$ /opt/MPI/openmpi-1.5.3/linux/intel/bin/mpiexec --hostfile
hostfile-mini -mca odls_base_verbose 5 --leave-session-attached
--display-map helloworld 2>&1 | tee helloworld.txt

>
>
> On Nov 21, 2011, at 9:33 AM, Paul Kapinos wrote:
>
>> Hello Open MPI volks,
>>
>> We use OpenMPI 1.5.3 on our pretty new 1800+ nodes InfiniBand cluster, and we have some strange hangups if starting OpenMPI processes.
>>
>> The nodes are named linuxbsc001,linuxbsc002,... (with some lacuna due of offline nodes). Each node is accessible from each other over SSH (without password), also MPI programs between any two nodes are checked to run.
>>
>>
>> So long, I tried to start some bigger number of processes, one process per node:
>> $ mpiexec -np NN --host linuxbsc001,linuxbsc002,... MPI_FastTest.exe
>>
>> Now the problem: there are some constellations of names in the host list on which mpiexec reproducible hangs forever; and more surprising: other *permutation* of the *same* node names may run without any errors!
>>
>> Example: the command in laueft.txt runs OK, the command in haengt.txt hangs. Note: the only difference is that the node linuxbsc025 is put on the end of the host list. Amazed, too?
>>
>> Looking on the particular nodes during the above mpiexec hangs, we found the orted daemons started on *each* node and the binary on all but one node (orted.txt, MPI_FastTest.txt).
>> Again amazing that the node with no user process started (leading to hangup in MPI_Init of all processes and thus to hangup, I believe) was always the same, linuxbsc005, which is NOT the permuted item linuxbsc025...
>>
>> This behaviour is reproducible. The hang-on only occure if the started application is a MPI application ("hostname" does not hang).
>>
>>
>> Any Idea what is gonna on?
>>
>>
>> Best,
>>
>> Paul Kapinos
>>
>>
>> P.S: no alias names used, all names are real ones
>>
>>
>>
>>
>>
>>
>>
>> --
>> Dipl.-Inform. Paul Kapinos - High Performance Computing,
>> RWTH Aachen University, Center for Computing and Communication
>> Seffenter Weg 23, D 52074 Aachen (Germany)
>> Tel: +49 241/80-24915
>> linuxbsc001: STDOUT: 24323 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc002: STDOUT: 2142 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc003: STDOUT: 69266 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc004: STDOUT: 58899 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc006: STDOUT: 68255 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc007: STDOUT: 62026 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc008: STDOUT: 54221 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc009: STDOUT: 55482 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc010: STDOUT: 59380 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc011: STDOUT: 58312 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc014: STDOUT: 56013 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc016: STDOUT: 58563 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc017: STDOUT: 54693 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc018: STDOUT: 54187 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc020: STDOUT: 55811 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc021: STDOUT: 54982 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc022: STDOUT: 50032 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc023: STDOUT: 54044 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc024: STDOUT: 51247 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc025: STDOUT: 18575 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc027: STDOUT: 48969 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc028: STDOUT: 52397 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc029: STDOUT: 52780 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc030: STDOUT: 47537 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc031: STDOUT: 54609 ? SLl 0:00 MPI_FastTest.exe
>> linuxbsc032: STDOUT: 52833 ? SLl 0:00 MPI_FastTest.exe
>> $ timex /opt/MPI/openmpi-1.5.3/linux/intel/bin/mpiexec -np 27 --host
>> linuxbsc001,linuxbsc002,linuxbsc003,linuxbsc004,linuxbsc005,linuxbsc006,linuxbsc007,linuxbsc008,linuxbsc009,linuxbsc010,linuxbsc011,linuxbsc014,linuxbsc016,linuxbsc017,linuxbsc018,linuxbsc020,linuxbsc021,linuxbsc022,linuxbsc023,linuxbsc024,linuxbsc025,linuxbsc027,linuxbsc028,linuxbsc029,linuxbsc030,linuxbsc031,linuxbsc032
>> MPI_FastTest.exe
>> $ timex /opt/MPI/openmpi-1.5.3/linux/intel/bin/mpiexec -np 27 --host
>> linuxbsc001,linuxbsc002,linuxbsc003,linuxbsc004,linuxbsc005,linuxbsc006,linuxbsc007,linuxbsc008,linuxbsc009,linuxbsc010,linuxbsc011,linuxbsc014,linuxbsc016,linuxbsc017,linuxbsc018,linuxbsc020,linuxbsc021,linuxbsc022,linuxbsc023,linuxbsc024,linuxbsc027,linuxbsc028,linuxbsc029,linuxbsc030,linuxbsc031,linuxbsc032,linuxbsc025
>> MPI_FastTest.exe
>> linuxbsc001: STDOUT: 24322 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc002: STDOUT: 2141 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 2 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc003: STDOUT: 69265 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 3 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc004: STDOUT: 58898 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 4 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc005: STDOUT: 65642 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 5 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc006: STDOUT: 68254 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 6 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc007: STDOUT: 62025 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 7 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc008: STDOUT: 54220 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 8 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc009: STDOUT: 55481 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 9 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc010: STDOUT: 59379 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 10 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc011: STDOUT: 58311 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 11 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc014: STDOUT: 56012 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 12 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc016: STDOUT: 58562 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 13 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc017: STDOUT: 54692 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 14 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc018: STDOUT: 54186 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 15 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc020: STDOUT: 55810 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 16 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc021: STDOUT: 54981 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 17 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc022: STDOUT: 50031 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 18 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc023: STDOUT: 54043 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 19 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc024: STDOUT: 51246 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 20 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc025: STDOUT: 18574 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 21 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc027: STDOUT: 48968 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 22 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc028: STDOUT: 52396 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 23 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc029: STDOUT: 52779 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 24 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc030: STDOUT: 47536 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 25 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc031: STDOUT: 54608 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 26 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> linuxbsc032: STDOUT: 52832 ? Ss 0:00 /opt/MPI/openmpi-1.5.3/linux/intel/bin/orted --daemonize -mca ess env -mca orte_ess_jobid 751435776 -mca orte_ess_vpid 27 -mca orte_ess_num_procs 28 --hnp-uri 751435776.0;tcp://134.61.194.2:33210 -mca plm rsh
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

-- 
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915

linuxbsc005 slots=1
linuxbsc006 slots=1
linuxbsc007 slots=1
linuxbsc008 slots=1
linuxbsc009 slots=1
linuxbsc010 slots=1
linuxbsc011 slots=1
linuxbsc014 slots=1
linuxbsc016 slots=1
linuxbsc017 slots=1
linuxbsc018 slots=1
linuxbsc020 slots=1
linuxbsc021 slots=1
linuxbsc022 slots=1
linuxbsc023 slots=1
linuxbsc024 slots=1
linuxbsc025 slots=1

[linuxc2.rz.RWTH-Aachen.DE:22229] mca:base:select:( odls) Querying component [default]
[linuxc2.rz.RWTH-Aachen.DE:22229] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxc2.rz.RWTH-Aachen.DE:22229] mca:base:select:( odls) Selected component [default]

 ======================== JOB MAP ========================

 Data for node: linuxbsc005 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 0

 Data for node: linuxbsc006 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 1

 Data for node: linuxbsc007 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 2

 Data for node: linuxbsc008 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 3

 Data for node: linuxbsc009 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 4

 Data for node: linuxbsc010 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 5

 Data for node: linuxbsc011 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 6

 Data for node: linuxbsc014 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 7

 Data for node: linuxbsc016 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 8

 Data for node: linuxbsc017 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 9

 Data for node: linuxbsc018 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 10

 Data for node: linuxbsc020 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 11

 Data for node: linuxbsc021 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 12

 Data for node: linuxbsc022 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 13

 Data for node: linuxbsc023 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 14

 Data for node: linuxbsc024 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 15

 Data for node: linuxbsc025 Num procs: 1
         Process OMPI jobid: [87,1] Process rank: 16

 =============================================================
[linuxbsc007.rz.RWTH-Aachen.DE:07574] mca:base:select:( odls) Querying component [default]
[linuxbsc007.rz.RWTH-Aachen.DE:07574] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc007.rz.RWTH-Aachen.DE:07574] mca:base:select:( odls) Selected component [default]
[linuxbsc016.rz.RWTH-Aachen.DE:03146] mca:base:select:( odls) Querying component [default]
[linuxbsc016.rz.RWTH-Aachen.DE:03146] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc016.rz.RWTH-Aachen.DE:03146] mca:base:select:( odls) Selected component [default]
[linuxbsc005.rz.RWTH-Aachen.DE:22051] mca:base:select:( odls) Querying component [default]
[linuxbsc005.rz.RWTH-Aachen.DE:22051] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc005.rz.RWTH-Aachen.DE:22051] mca:base:select:( odls) Selected component [default]
[linuxbsc011.rz.RWTH-Aachen.DE:07131] mca:base:select:( odls) Querying component [default]
[linuxbsc011.rz.RWTH-Aachen.DE:07131] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc011.rz.RWTH-Aachen.DE:07131] mca:base:select:( odls) Selected component [default]
[linuxbsc025.rz.RWTH-Aachen.DE:43153] mca:base:select:( odls) Querying component [default]
[linuxbsc025.rz.RWTH-Aachen.DE:43153] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc025.rz.RWTH-Aachen.DE:43153] mca:base:select:( odls) Selected component [default]
[linuxbsc017.rz.RWTH-Aachen.DE:05044] mca:base:select:( odls) Querying component [default]
[linuxbsc017.rz.RWTH-Aachen.DE:05044] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc017.rz.RWTH-Aachen.DE:05044] mca:base:select:( odls) Selected component [default]
[linuxbsc018.rz.RWTH-Aachen.DE:01840] mca:base:select:( odls) Querying component [default]
[linuxbsc018.rz.RWTH-Aachen.DE:01840] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc018.rz.RWTH-Aachen.DE:01840] mca:base:select:( odls) Selected component [default]
[linuxbsc024.rz.RWTH-Aachen.DE:79549] mca:base:select:( odls) Querying component [default]
[linuxbsc024.rz.RWTH-Aachen.DE:79549] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc024.rz.RWTH-Aachen.DE:79549] mca:base:select:( odls) Selected component [default]
[linuxbsc022.rz.RWTH-Aachen.DE:73501] mca:base:select:( odls) Querying component [default]
[linuxbsc022.rz.RWTH-Aachen.DE:73501] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc022.rz.RWTH-Aachen.DE:73501] mca:base:select:( odls) Selected component [default]
[linuxbsc023.rz.RWTH-Aachen.DE:03364] mca:base:select:( odls) Querying component [default]
[linuxbsc023.rz.RWTH-Aachen.DE:03364] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc023.rz.RWTH-Aachen.DE:03364] mca:base:select:( odls) Selected component [default]
[linuxbsc006.rz.RWTH-Aachen.DE:16811] mca:base:select:( odls) Querying component [default]
[linuxbsc006.rz.RWTH-Aachen.DE:16811] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc006.rz.RWTH-Aachen.DE:16811] mca:base:select:( odls) Selected component [default]
[linuxbsc014.rz.RWTH-Aachen.DE:10206] mca:base:select:( odls) Querying component [default]
[linuxbsc014.rz.RWTH-Aachen.DE:10206] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc014.rz.RWTH-Aachen.DE:10206] mca:base:select:( odls) Selected component [default]
[linuxbsc008.rz.RWTH-Aachen.DE:00858] mca:base:select:( odls) Querying component [default]
[linuxbsc008.rz.RWTH-Aachen.DE:00858] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc008.rz.RWTH-Aachen.DE:00858] mca:base:select:( odls) Selected component [default]
[linuxbsc010.rz.RWTH-Aachen.DE:09727] mca:base:select:( odls) Querying component [default]
[linuxbsc010.rz.RWTH-Aachen.DE:09727] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc010.rz.RWTH-Aachen.DE:09727] mca:base:select:( odls) Selected component [default]
[linuxbsc020.rz.RWTH-Aachen.DE:06680] mca:base:select:( odls) Querying component [default]
[linuxbsc020.rz.RWTH-Aachen.DE:06680] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc020.rz.RWTH-Aachen.DE:06680] mca:base:select:( odls) Selected component [default]
[linuxbsc009.rz.RWTH-Aachen.DE:05145] mca:base:select:( odls) Querying component [default]
[linuxbsc009.rz.RWTH-Aachen.DE:05145] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc009.rz.RWTH-Aachen.DE:05145] mca:base:select:( odls) Selected component [default]
[linuxbsc021.rz.RWTH-Aachen.DE:01405] mca:base:select:( odls) Querying component [default]
[linuxbsc021.rz.RWTH-Aachen.DE:01405] mca:base:select:( odls) Query of component [default] set priority to 1
[linuxbsc021.rz.RWTH-Aachen.DE:01405] mca:base:select:( odls) Selected component [default]