Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Rainer Keller (Keller_at_[hidden])
Date: 2005-08-18 14:04:23


Hello Brian,
sure, attached is output of ompi_info -a on:

model name : AMD Opteron(tm) Processor 246

Linux c3-19 2.4.21-OC_NUMA_fix #4 SMP Tue Nov 30 16:03:38 CET 2004 x86_64
unknown

It's a SuSE SLES8 distribution with the following libc:

hpcraink_at_c3-19:~ > /lib64/libc.so.6
GNU C Library stable release version 2.2.5, by Roland McGrath et al.
Copyright (C) 1992-2001, 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.2.2 (SuSE Linux).
Compiled on a Linux 2.4.19 system on 2003-03-27.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        Berkeley DB glibc 2.1 compat library by Thorsten Kukuk
        linuxthreads-0.9 by Xavier Leroy
        BIND-8.2.3-T5B
        libthread_db work sponsored by Alpha Processor Inc
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Report bugs using the `glibcbug' script to <bugs_at_[hidden]>.

Compilation was done with pgcc-5.2-4

CU,
raY

On Thursday 18 August 2005 20:05, Brian Barrett wrote:
> Just to double check, can you run ompi_info and send me the results?
>
> Thanks,
>
> Brian
>
> On Aug 18, 2005, at 10:45 AM, Rainer Keller wrote:
> > Hello,
> > see the "same" (well probably not exactly same) thing here in
> > Opteron with
> > 64bit (-g and so on), I get:
> >
> > #0 0x0000000040085160 in orte_sds_base_contact_universe ()
> > at ../../../../../orte/mca/sds/base/sds_base_interface.c:29
> > 29 return orte_sds_base_module->contact_universe();
> > (gdb) where
> > #0 0x0000000040085160 in orte_sds_base_contact_universe ()
> > at ../../../../../orte/mca/sds/base/sds_base_interface.c:29
> > #1 0x0000000040063e95 in orte_init_stage1 ()
> > at ../../../orte/runtime/orte_init_stage1.c:185
> > #2 0x0000000040017e7d in orte_system_init ()
> > at ../../../orte/runtime/orte_system_init.c:38
> > #3 0x00000000400148f5 in orte_init () at ../../../orte/runtime/
> > orte_init.c:46
> > #4 0x000000004000dfc7 in main (argc=4, argv=0x7fbfffe8a8)
> > at ../../../../orte/tools/orterun/orterun.c:291
> > #5 0x0000002a95c0c017 in __libc_start_main () from /lib64/libc.so.6
> > #6 0x000000004000bf2a in _start ()
> > (gdb)
> > within mpirun
> >
> > orte_sds_base_module here is Null...
> > This is without persistent orted; Just mpirun...
> >
> > CU,
> > ray
> >
> > On Thursday 18 August 2005 16:57, Nathan DeBardeleben wrote:
> >> FYI, this only happens when I let OMPI compile 64bit on Linux.
> >> When I
> >> throw in there CFLAGS=FFLAGS=CXXFLAGS=-m32 orted, my myriad of test
> >> codes, mpirun, registry subscription codes, and JNI all work like
> >> a champ.
> >> Something's wrong with the 64bit it appears to me.
> >>
> >> -- Nathan
> >> Correspondence
> >> ---------------------------------------------------------------------
> >> Nathan DeBardeleben, Ph.D.
> >> Los Alamos National Laboratory
> >> Parallel Tools Team
> >> High Performance Computing Environments
> >> phone: 505-667-3428
> >> email: ndebard_at_[hidden]
> >> ---------------------------------------------------------------------
> >>
> >> Tim S. Woodall wrote:
> >>> Nathan,
> >>>
> >>> I'll try to reproduce this sometime this week - but I'm pretty
> >>> swamped.
> >>> Is Greg also seeing the same behavior?
> >>>
> >>> Thanks,
> >>> Tim
> >>>
> >>> Nathan DeBardeleben wrote:
> >>>> To expand on this further, orte_init() seg faults on both bluesteel
> >>>> (32bit linux) and sparkplug (64bit linux) equally. The required
> >>>> condition is that orted must be running first (which of course we
> >>>> require for our work - a persistent orte daemon and registry).
> >>>>
> >>>>> [bluesteel]~/ptp > ./dump_info
> >>>>> Segmentation fault
> >>>>> [bluesteel]~/ptp > gdb dump_info
> >>>>> GNU gdb 6.1
> >>>>> Copyright 2004 Free Software Foundation, Inc.
> >>>>> GDB is free software, covered by the GNU General Public
> >>>>> License, and
> >>>>> you are
> >>>>> welcome to change it and/or distribute copies of it under certain
> >>>>> conditions.
> >>>>> Type "show copying" to see the conditions.
> >>>>> There is absolutely no warranty for GDB. Type "show warranty" for
> >>>>> details.
> >>>>> This GDB was configured as "x86_64-suse-linux"...Using host
> >>>>> libthread_db library "/lib64/tls/libthread_db.so.1".
> >>>>>
> >>>>> (gdb) run
> >>>>> Starting program: /home/ndebard/ptp/dump_info
> >>>>>
> >>>>> Program received signal SIGSEGV, Segmentation fault.
> >>>>> 0x0000000000000000 in ?? ()
> >>>>> (gdb) where
> >>>>> #0 0x0000000000000000 in ?? ()
> >>>>> #1 0x000000000045997d in orte_init_stage1 () at
> >>>>> orte_init_stage1.c:419
> >>>>> #2 0x00000000004156a7 in orte_system_init () at
> >>>>> orte_system_init.c:38
> >>>>> #3 0x00000000004151c7 in orte_init () at orte_init.c:46
> >>>>> #4 0x0000000000414cbb in main (argc=1, argv=0x7fbffff298) at
> >>>>> dump_info.c:185
> >>>>> (gdb)
> >>>>
> >>>> -- Nathan
> >>>> Correspondence
> >>>> -------------------------------------------------------------------
> >>>> --
> >>>> Nathan DeBardeleben, Ph.D.
> >>>> Los Alamos National Laboratory
> >>>> Parallel Tools Team
> >>>> High Performance Computing Environments
> >>>> phone: 505-667-3428
> >>>> email: ndebard_at_[hidden]
> >>>> -------------------------------------------------------------------
> >>>> --
> >>>>
> >>>> Nathan DeBardeleben wrote:
> >>>>> Just to clarify:
> >>>>> 1: no orted started (meaning the MPIrun or registry programs will
> >>>>> start one by themselves) causes those programs to lock up.
> >>>>> 2: starting orted by hand (trying to get these programs to
> >>>>> connect to
> >>>>> a centralized one) causes the connecting programs to seg fault.
> >>>>>
> >>>>> -- Nathan
> >>>>> Correspondence
> >>>>> ------------------------------------------------------------------
> >>>>> ---
> >>>>> Nathan DeBardeleben, Ph.D.
> >>>>> Los Alamos National Laboratory
> >>>>> Parallel Tools Team
> >>>>> High Performance Computing Environments
> >>>>> phone: 505-667-3428
> >>>>> email: ndebard_at_[hidden]
> >>>>> ------------------------------------------------------------------
> >>>>> ---
> >>>>>
> >>>>> Nathan DeBardeleben wrote:
> >>>>>> So I dropped an .ompi_ignore into that directory,
> >>>>>> reconfigured, and
> >>>>>> compile worked (yay!).
> >>>>>> However, not a lot of progress: mpirun locks up, all my
> >>>>>> registry test
> >>>>>> programs lock up as well. If I start the orted by hand, then
> >>>>>> any of my
> >>>>>>
> >>>>>> registry calling programs cause segfault:
> >>>>>>> [sparkplug]~/ptp > gdb sub_test
> >>>>>>> GNU gdb 6.1
> >>>>>>> Copyright 2004 Free Software Foundation, Inc.
> >>>>>>> GDB is free software, covered by the GNU General Public
> >>>>>>> License, and
> >>>>>>> you are
> >>>>>>> welcome to change it and/or distribute copies of it under
> >>>>>>> certain
> >>>>>>> conditions.
> >>>>>>> Type "show copying" to see the conditions.
> >>>>>>> There is absolutely no warranty for GDB. Type "show
> >>>>>>> warranty" for
> >>>>>>> details.
> >>>>>>> This GDB was configured as "x86_64-suse-linux"...Using host
> >>>>>>> libthread_db library "/lib64/tls/libthread_db.so.1".
> >>>>>>>
> >>>>>>> (gdb) run
> >>>>>>> Starting program: /home/ndebard/ptp/sub_test
> >>>>>>>
> >>>>>>> Program received signal SIGSEGV, Segmentation fault.
> >>>>>>> 0x0000000000000000 in ?? ()
> >>>>>>> (gdb) where
> >>>>>>> #0 0x0000000000000000 in ?? ()
> >>>>>>> #1 0x00000000004598a5 in orte_init_stage1 () at
> >>>>>>> orte_init_stage1.c:419 #2 0x00000000004155cf in
> >>>>>>> orte_system_init ()
> >>>>>>> at orte_system_init.c:38 #3 0x00000000004150ef in orte_init
> >>>>>>> () at
> >>>>>>> orte_init.c:46
> >>>>>>> #4 0x00000000004148a1 in main (argc=1, argv=0x7fbffff178) at
> >>>>>>> sub_test.c:60
> >>>>>>> (gdb)
> >>>>>>
> >>>>>> Yes, I recompiled everything.
> >>>>>>
> >>>>>> Here's an example of me trying something a little more
> >>>>>> complicated
> >>>>>> (which I believe locks up for the same reason - something
> >>>>>> borked with
> >>>>>> the registry interaction).
> >>>>>>
> >>>>>>>> [sparkplug]~/ompi-test > bjssub -s 10000 -n 10 -i bash
> >>>>>>>> Waiting for interactive job nodes.
> >>>>>>>> (nodes 18 16 17 18 19 20 21 22 23 24 25)
> >>>>>>>> Starting interactive job.
> >>>>>>>> NODES=16,17,18,19,20,21,22,23,24,25
> >>>>>>>> JOBID=18
> >>>>>>>
> >>>>>>> so i got my nodes
> >>>>>>>
> >>>>>>>> ndebard_at_sparkplug:~/ompi-test> export
> >>>>>>>> OMPI_MCA_ptl_base_exclude=sm
> >>>>>>>> ndebard_at_sparkplug:~/ompi-test> export
> >>>>>>>> OMPI_MCA_pls_bproc_seed_priority=101
> >>>>>>>
> >>>>>>> and set these envvars like we need to use Greg's bproc,
> >>>>>>> without the
> >>>>>>> 2nd export the machine's load maxes and locks up.
> >>>>>>>
> >>>>>>>> ndebard_at_sparkplug:~/ompi-test> bpstat
> >>>>>>>> Node(s) Status Mode
> >>>>>>>> User Group 100-128 down
> >>>>>>>> ---------- root root 0-15
> >>>>>>>> up ---x------ vchandu vchandu
> >>>>>>>> 16-25 up ---x------
> >>>>>>>> ndebard ndebard
> >>>>>>>> 26-27 up ---x------
> >>>>>>>> root root 28-30 up
> >>>>>>>> ---x--x--x root root ndebard_at_sparkplug:~/ompi-test>
> >>>>>>>> env | grep
> >>>>>>>> NODES
> >>>>>>>> NODES=16,17,18,19,20,21,22,23,24,25
> >>>>>>>
> >>>>>>> yes, i really have the nodes
> >>>>>>>
> >>>>>>>> ndebard_at_sparkplug:~/ompi-test> mpicc -o test-mpi test-mpi.c
> >>>>>>>> ndebard_at_sparkplug:~/ompi-test>
> >>>>>>>
> >>>>>>> recompile for good measure
> >>>>>>>
> >>>>>>>> ndebard_at_sparkplug:~/ompi-test> ls /tmp/openmpi-sessions-
> >>>>>>>> ndebard*
> >>>>>>>> /bin/ls: /tmp/openmpi-sessions-ndebard*: No such file or
> >>>>>>>> directory
> >>>>>>>
> >>>>>>> proof that there's no left over old directory
> >>>>>>>
> >>>>>>>> ndebard_at_sparkplug:~/ompi-test> mpirun -np 1 test-mpi
> >>>>>>>
> >>>>>>> it never responds at this point - but I can kill it with ^C.
> >>>>>>>
> >>>>>>>> mpirun: killing job...
> >>>>>>>> Killed
> >>>>>>>> ndebard_at_sparkplug:~/ompi-test>
> >>>>>>
> >>>>>> -- Nathan
> >>>>>> Correspondence
> >>>>>> -----------------------------------------------------------------
> >>>>>> ----
> >>>>>> Nathan DeBardeleben, Ph.D.
> >>>>>> Los Alamos National Laboratory
> >>>>>> Parallel Tools Team
> >>>>>> High Performance Computing Environments
> >>>>>> phone: 505-667-3428
> >>>>>> email: ndebard_at_[hidden]
> >>>>>> -----------------------------------------------------------------
> >>>>>> ----
> >>>>>>
> >>>>>> Jeff Squyres wrote:
> >>>>>>> Is this what Tim Prins was working on?
> >>>>>>>
> >>>>>>> On Aug 16, 2005, at 5:21 PM, Tim S. Woodall wrote:
> >>>>>>>> I'm not sure why this is even building... Is someone working
> >>>>>>>> on this?
> >>>>>>>> I thought we had .ompi_ignore files in this directory.
> >>>>>>>>
> >>>>>>>> Tim
> >>>>>>>>
> >>>>>>>> Nathan DeBardeleben wrote:
> >>>>>>>>> So I'm seeing all these nice emails about people developing
> >>>>>>>>> on OMPI
> >>>>>>>>> today yet I can't get it to compile. Am I out here in
> >>>>>>>>> limbo on this
> >>>>>>>>> or
> >>>>>>>>> are others in the same boat? The errors I'm seeing are
> >>>>>>>>> about some
> >>>>>>>>> bproc
> >>>>>>>>> code calling undefined functions and they are linked again
> >>>>>>>>> below.
> >>>>>>>>>
> >>>>>>>>> -- Nathan
> >>>>>>>>> Correspondence
> >>>>>>>>> --------------------------------------------------------------
> >>>>>>>>> ------
> >>>>>>>>> - Nathan DeBardeleben, Ph.D.
> >>>>>>>>> Los Alamos National Laboratory
> >>>>>>>>> Parallel Tools Team
> >>>>>>>>> High Performance Computing Environments
> >>>>>>>>> phone: 505-667-3428
> >>>>>>>>> email: ndebard_at_[hidden]
> >>>>>>>>> --------------------------------------------------------------
> >>>>>>>>> ------
> >>>>>>>>> -
> >>>>>>>>>
> >>>>>>>>> Nathan DeBardeleben wrote:
> >>>>>>>>>> Back from training and trying to test this but now OMPI
> >>>>>>>>>> doesn't
> >>>>>>>>>> compile
> >>>>>>>>>>
> >>>>>>>>>> at all:
> >>>>>>>>>>> gcc -DHAVE_CONFIG_H -I. -I. -I../../../../include
> >>>>>>>>>>> -I../../../../include -I../../../.. -I../../../..
> >>>>>>>>>>> -I../../../../include -I../../../../opal -I../../../../orte
> >>>>>>>>>>> -I../../../../ompi -g -Wall -Wundef -Wno-long-long -Wsign-
> >>>>>>>>>>> compare
> >>>>>>>>>>> -Wmissing-prototypes -Wstrict-prototypes -Wcomment -pedantic
> >>>>>>>>>>> -Werror-implicit-function-declaration -fno-strict-
> >>>>>>>>>>> aliasing -MT
> >>>>>>>>>>> ras_lsf_bproc.lo -MD -MP -MF .deps/ras_lsf_bproc.Tpo -c
> >>>>>>>>>>> ras_lsf_bproc.c -o ras_lsf_bproc.o
> >>>>>>>>>>> ras_lsf_bproc.c: In function
> >>>>>>>>>>> `orte_ras_lsf_bproc_node_insert':
> >>>>>>>>>>> ras_lsf_bproc.c:32: error: implicit declaration of function
> >>>>>>>>>>> `orte_ras_base_node_insert'
> >>>>>>>>>>> ras_lsf_bproc.c: In function
> >>>>>>>>>>> `orte_ras_lsf_bproc_node_query':
> >>>>>>>>>>> ras_lsf_bproc.c:37: error: implicit declaration of function
> >>>>>>>>>>> `orte_ras_base_node_query'
> >>>>>>>>>>> make[4]: *** [ras_lsf_bproc.lo] Error 1
> >>>>>>>>>>> make[4]: Leaving directory
> >>>>>>>>>>> `/home/ndebard/ompi/orte/mca/ras/lsf_bproc'
> >>>>>>>>>>> make[3]: *** [all-recursive] Error 1
> >>>>>>>>>>> make[3]: Leaving directory `/home/ndebard/ompi/orte/mca/ras'
> >>>>>>>>>>> make[2]: *** [all-recursive] Error 1
> >>>>>>>>>>> make[2]: Leaving directory `/home/ndebard/ompi/orte/mca'
> >>>>>>>>>>> make[1]: *** [all-recursive] Error 1
> >>>>>>>>>>> make[1]: Leaving directory `/home/ndebard/ompi/orte'
> >>>>>>>>>>> make: *** [all-recursive] Error 1
> >>>>>>>>>>> [sparkplug]~/ompi >
> >>>>>>>>>>
> >>>>>>>>>> Clean SVN checkout this morning with configure:
> >>>>>>>>>>> [sparkplug]~/ompi > ./configure --enable-static --disable-
> >>>>>>>>>>> shared
> >>>>>>>>>>> --without-threads --prefix=/home/ndebard/local/ompi
> >>>>>>>>>>> --with-devel-headers
> >>>>>>>>>>
> >>>>>>>>>> -- Nathan
> >>>>>>>>>> Correspondence
> >>>>>>>>>> -------------------------------------------------------------
> >>>>>>>>>> ------
> >>>>>>>>>> -- Nathan DeBardeleben, Ph.D.
> >>>>>>>>>> Los Alamos National Laboratory
> >>>>>>>>>> Parallel Tools Team
> >>>>>>>>>> High Performance Computing Environments
> >>>>>>>>>> phone: 505-667-3428
> >>>>>>>>>> email: ndebard_at_[hidden]
> >>>>>>>>>> -------------------------------------------------------------
> >>>>>>>>>> ------
> >>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>> Brian Barrett wrote:
> >>>>>>>>>>> This is now fixed in SVN. You should no longer need the
> >>>>>>>>>>> --build=i586... hack to compile 32 bit code on Opterons.
> >>>>>>>>>>>
> >>>>>>>>>>> Brian
> >>>>>>>>>>>
> >>>>>>>>>>> On Aug 12, 2005, at 3:17 PM, Brian Barrett wrote:
> >>>>>>>>>>>> On Aug 12, 2005, at 3:13 PM, Nathan DeBardeleben wrote:
> >>>>>>>>>>>>> We've got a 64bit Linux (SUSE) box here. For a variety of
> >>>>>>>>>>>>> reasons (Java, JNI, linking in with OMPI libraries, etc
> >>>>>>>>>>>>> which I
> >>>>>>>>>>>>> won't get into)
> >>>>>>>>>>>>> I need to compile OMPI 32 bit (or get 64bit versions of
> >>>>>>>>>>>>> a lot of
> >>>>>>>>>>>>> other
> >>>>>>>>>>>>> libraries).
> >>>>>>>>>>>>> I get various compile errors when I try different
> >>>>>>>>>>>>> things, but
> >>>>>>>>>>>>> first
> >>>>>>>>>>>>> let
> >>>>>>>>>>>>> me explain the system we have:
> >>>>>>>>>>>>
> >>>>>>>>>>>> <snip>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> This goes on and on and on actually. And the 'is
> >>>>>>>>>>>>> incompatible
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>> i386:x86-64 output' looks to be repeated for every line
> >>>>>>>>>>>>> before
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>> error which actually caused the Make to bomb.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Any suggestions at all? Surely someone must have tried
> >>>>>>>>>>>>> to force
> >>>>>>>>>>>>> OMPI to
> >>>>>>>>>>>>> build in 32bit mode on a 64bit machine.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I don't think anyone has tried to build 32 bit on an
> >>>>>>>>>>>> Opteron,
> >>>>>>>>>>>> which is the cause of the problems...
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think I know how to fix this, but won't happen until
> >>>>>>>>>>>> later in
> >>>>>>>>>>>> the weekend. I can't think of a good workaround until
> >>>>>>>>>>>> then.
> >>>>>>>>>>>> Well, one possibility is to set the target like you were
> >>>>>>>>>>>> doing
> >>>>>>>>>>>> and disable ROMIO. Actually, you'll also need to disable
> >>>>>>>>>>>> Fortran 77. So something like:
> >>>>>>>>>>>>
> >>>>>>>>>>>> ./configure [usual options] --build=i586-suse-linux --
> >>>>>>>>>>>> disable-io-
> >>>>>>>>>>>> romio --disable-f77
> >>>>>>>>>>>>
> >>>>>>>>>>>> might just do the trick.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Brian
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Brian Barrett
> >>>>>>>>>>>> Open MPI developer
> >>>>>>>>>>>> http://www.open-mpi.org/
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> devel mailing list
> >>>>>>>>>>>> devel_at_[hidden]
> >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>>>>>>>>
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> devel mailing list
> >>>>>>>>>> devel_at_[hidden]
> >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> devel mailing list
> >>>>>>>>> devel_at_[hidden]
> >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> devel mailing list
> >>>>>>>> devel_at_[hidden]
> >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> devel mailing list
> >>>>>> devel_at_[hidden]
> >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>>>
> >>>>> _______________________________________________
> >>>>> devel mailing list
> >>>>> devel_at_[hidden]
> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>>
> >>>> _______________________________________________
> >>>> devel mailing list
> >>>> devel_at_[hidden]
> >>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>
> >>> _______________________________________________
> >>> devel mailing list
> >>> devel_at_[hidden]
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>
> >> _______________________________________________
> >> devel mailing list
> >> devel_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >
> > --
> > ---------------------------------------------------------------------
> > Dipl.-Inf. Rainer Keller email: keller_at_[hidden]
> > High Performance Computing Tel: ++49 (0)711-685 5858
> > Center Stuttgart (HLRS) Fax: ++49 (0)711-678 7626
> > Nobelstrasse 19, R. O0.030 http://www.hlrs.de/people/keller
> > 70550 Stuttgart
> > _______________________________________________
> > devel mailing list
> > devel_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
---------------------------------------------------------------------
Dipl.-Inf. Rainer Keller             email: keller_at_[hidden]
  High Performance Computing         Tel: ++49 (0)711-685 5858
    Center Stuttgart (HLRS)          Fax: ++49 (0)711-678 7626
  Nobelstrasse 19,  R. O0.030        http://www.hlrs.de/people/keller
  70550 Stuttgart