Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] WRF, OpenMPI and PGI 7.2
From: Gus Correa (gus_at_[hidden])
Date: 2009-02-20 10:23:14


Hi Gerry

I usually put configure commands (and environment variables)
on little shell scripts, which I edit to fit the combination
of hardware/compiler(s), and keep them in the build directory.
Otherwise I would forget the details next time I need to build.

If Myrinet and GigE are on separate clusters,
you'll have to install OpenMPI on each one, sorry.
However, if Myrinet and GigE are available on the same cluster,
you can build a single OpenMPI,
and choose the "byte transport layer (BTL)"
to be Myrinet or GigE (or IB, for that matter),
and even the NICs/networks to use,
on your job submission script.

Check the OpenMPI FAQ:
http://www.open-mpi.org/faq/?category=myrinet#myri-btl-gm
http://www.open-mpi.org/faq/?category=tcp#tcp-btl
http://www.open-mpi.org/faq/?category=openfabrics#ib-btl
http://www.open-mpi.org/faq/?category=tcp#tcp-multi-network
http://www.open-mpi.org/faq/?category=tcp#tcp-selection

Gus Correa

PS - BTW - Our old non-Rocks cluster has Myrinet-2000 (GM).
After I get the new cluster up and running and in production,
I am thinking of revamping the old cluster, and install Rocks on it.
I would love to learn from your experience with your
Rocks+Myrinet cluster, if you have the time to post a short
"bullet list" of "do's and don'ts".
(The Rocks list may be more appropriate than the OpenMPI for this.)

Last I checked Myrinet had a roll only for Rocks 5.0,
not 5.1, right?
Did you install it with on top of Rocks 5.0 or 5.1?
(For instance, my recollection of old postings on the list,
is that the Torque 5.0 roll worked with Rocks 5.1,
but it is always a risky business to mix different releases.)

Many thanks,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Gerry Creager wrote:
> Gus,
>
> I'll give that a try real quick (or as quickly as the compiles can run.
>
> I'd not thought of this solution. I've been context-switching too much
> lately. I've gotta look at this for a gigabit cluster as well.
>
> Thanks!
>
> Gus Correa wrote:
>> Hi Gerry
>>
>> You may need to compile a hybrid OpenMPI
>> using gcc for C, PGI f90 for Fortran on the OpenMPI configure script.
>> This should give you the required mpicc and mpif90 to do the job.
>> I guess this is what Elvedin meant on his message.
>>
>> I have these hybrids for OpenMPI and MPICH2 here
>> (not Myrinet but GigE), and they work
>> fine with a WRF relative (CAM3, atmospheric climate).
>>
>> Two cents from
>> Gus Correa
>> ---------------------------------------------------------------------
>> Gustavo Correa
>> Lamont-Doherty Earth Observatory - Columbia University
>> Palisades, NY, 10964-8000 - USA
>> ---------------------------------------------------------------------
>>
>> Gerry Creager wrote:
>>> Elvedin,
>>>
>>> Yeah, I thought about that after finding a reference to this in the
>>> archives, so I redirected the path to MPI toward the gnu-compiled
>>> version. It died in THIS manner:
>>> make[3]: Entering directory `/home/gerry/WRFv3/WRFV3/external/RSL_LITE'
>>> mpicc -cc=gcc -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c c_code.c
>>> pgcc-Error-Unknown switch: -cc=gcc
>>> make[3]: [c_code.o] Error 1 (ignored)
>>>
>>> Methinks the wrf configuration script and make file will need some
>>> tweeks.
>>>
>>> Interesting thing: I have another system (alas, with mpich) where it
>>> compiles just fine. I'm trying to sort this out, as on 2 systems,
>>> with openMPI, it does odd dances before dying.
>>>
>>> I'm still trying things. I've gotta get this up both for MY research
>>> and to support other users.
>>>
>>> Thanks, Gerry
>>>
>>> Elvedin Trnjanin wrote:
>>>> WRF almost requires that you use gcc for the C/C++ part and the PGI
>>>> Fortran compilers, if you choose that option. I'd suggest compiling
>>>> OpenMPI in the same way as that has resolved our various issues.
>>>> Have you tried that with the same result?
>>>>
>>>> Gerry Creager wrote:
>>>>> Howdy,
>>>>>
>>>>> I'm new to this list. I've done a little review but likely missed
>>>>> something specific to what I'm asking. I'll keep looking but need
>>>>> to resolve this soon.
>>>>>
>>>>> I'm running a Rocks cluster (centos 5), with PGI 7.2-3 compilers,
>>>>> Myricom MX2 hardware and drivers, and OpenMPI1.3
>>>>>
>>>>> I installed the Myricom roll which has OpenMPI compiled with gcc.
>>>>> I recently compiled the openmpi code w/ PGI.
>>>>>
>>>>> I've the MPICH_F90 pointing to the right place, and we're looking
>>>>> for the right includes and libs by means of LD_LIBRARY_PATH, etc.
>>>>>
>>>>> When I tried to run, I got the following error:
>>>>> make[3]: Entering directory
>>>>> `/home/gerry/WRFv3/WRFV3/external/RSL_LITE'
>>>>> mpicc -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c c_code.c
>>>>> PGC/x86-64 Linux 7.2-3: compilation completed with warnings
>>>>> mpicc -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c buf_for_proc.c
>>>>> PGC-S-0036-Syntax error: Recovery attempted by inserting identifier
>>>>> .Z0000 before '(' (/share/apps/openmpi-1.3-pgi/include/mpi.h: 889)
>>>>> PGC-S-0082-Function returning array not allowed
>>>>> (/share/apps/openmpi-1.3-pgi/include/mpi.h: 889)
>>>>> PGC-S-0043-Redefinition of symbol, MPI_Comm
>>>>> (/share/apps/openmpi-1.3-pgi/include/mpi.h: 903)
>>>>> PGC/x86-64 Linux 7.2-3: compilation completed with severe errors
>>>>> make[3]: [buf_for_proc.o] Error 2 (ignored)
>>>>>
>>>>> Note that I had modified the makefile to use PGI in place of gcc,
>>>>> and thus, the PGI-compiled openMPI.
>>>>>
>>>>> Thanks, Gerry
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>