Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] WRF, OpenMPI and PGI 7.2
From: Ralph Castain (rhc_at_[hidden])
Date: 2009-02-20 10:30:17

Note that (beginning with 1.3) you can also use "platform files" to
save configure and default mca params so that you build consistently.
Check the examples in contrib/platform. Most of us developers use
these religiously, as do our host organizations, for precisely this

I believe there should be something on the FAQ about platform files -
if not, I'll try to add it in the next few days.

If you want to contribute platform files to support some community
with similar configurations, please send them to me - we shouldn't
need a contributors agreement for them as there is no code involved.


On Feb 20, 2009, at 8:23 AM, Gus Correa wrote:

> Hi Gerry
> I usually put configure commands (and environment variables)
> on little shell scripts, which I edit to fit the combination
> of hardware/compiler(s), and keep them in the build directory.
> Otherwise I would forget the details next time I need to build.
> If Myrinet and GigE are on separate clusters,
> you'll have to install OpenMPI on each one, sorry.
> However, if Myrinet and GigE are available on the same cluster,
> you can build a single OpenMPI,
> and choose the "byte transport layer (BTL)"
> to be Myrinet or GigE (or IB, for that matter),
> and even the NICs/networks to use,
> on your job submission script.
> Check the OpenMPI FAQ:
> Gus Correa
> PS - BTW - Our old non-Rocks cluster has Myrinet-2000 (GM).
> After I get the new cluster up and running and in production,
> I am thinking of revamping the old cluster, and install Rocks on it.
> I would love to learn from your experience with your
> Rocks+Myrinet cluster, if you have the time to post a short
> "bullet list" of "do's and don'ts".
> (The Rocks list may be more appropriate than the OpenMPI for this.)
> Last I checked Myrinet had a roll only for Rocks 5.0,
> not 5.1, right?
> Did you install it with on top of Rocks 5.0 or 5.1?
> (For instance, my recollection of old postings on the list,
> is that the Torque 5.0 roll worked with Rocks 5.1,
> but it is always a risky business to mix different releases.)
> Many thanks,
> Gus Correa
> ---------------------------------------------------------------------
> Gustavo Correa
> Lamont-Doherty Earth Observatory - Columbia University
> Palisades, NY, 10964-8000 - USA
> ---------------------------------------------------------------------
> Gerry Creager wrote:
>> Gus,
>> I'll give that a try real quick (or as quickly as the compiles can
>> run.
>> I'd not thought of this solution. I've been context-switching too
>> much lately. I've gotta look at this for a gigabit cluster as well.
>> Thanks!
>> Gus Correa wrote:
>>> Hi Gerry
>>> You may need to compile a hybrid OpenMPI
>>> using gcc for C, PGI f90 for Fortran on the OpenMPI configure
>>> script.
>>> This should give you the required mpicc and mpif90 to do the job.
>>> I guess this is what Elvedin meant on his message.
>>> I have these hybrids for OpenMPI and MPICH2 here
>>> (not Myrinet but GigE), and they work
>>> fine with a WRF relative (CAM3, atmospheric climate).
>>> Two cents from
>>> Gus Correa
>>> ---------------------------------------------------------------------
>>> Gustavo Correa
>>> Lamont-Doherty Earth Observatory - Columbia University
>>> Palisades, NY, 10964-8000 - USA
>>> ---------------------------------------------------------------------
>>> Gerry Creager wrote:
>>>> Elvedin,
>>>> Yeah, I thought about that after finding a reference to this in
>>>> the archives, so I redirected the path to MPI toward the gnu-
>>>> compiled version. It died in THIS manner:
>>>> make[3]: Entering directory `/home/gerry/WRFv3/WRFV3/external/
>>>> RSL_LITE'
>>>> mpicc -cc=gcc -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c c_code.c
>>>> pgcc-Error-Unknown switch: -cc=gcc
>>>> make[3]: [c_code.o] Error 1 (ignored)
>>>> Methinks the wrf configuration script and make file will need
>>>> some tweeks.
>>>> Interesting thing: I have another system (alas, with mpich) where
>>>> it compiles just fine. I'm trying to sort this out, as on 2
>>>> systems, with openMPI, it does odd dances before dying.
>>>> I'm still trying things. I've gotta get this up both for MY
>>>> research and to support other users.
>>>> Thanks, Gerry
>>>> Elvedin Trnjanin wrote:
>>>>> WRF almost requires that you use gcc for the C/C++ part and the
>>>>> PGI Fortran compilers, if you choose that option. I'd suggest
>>>>> compiling OpenMPI in the same way as that has resolved our
>>>>> various issues. Have you tried that with the same result?
>>>>> Gerry Creager wrote:
>>>>>> Howdy,
>>>>>> I'm new to this list. I've done a little review but likely
>>>>>> missed something specific to what I'm asking. I'll keep
>>>>>> looking but need to resolve this soon.
>>>>>> I'm running a Rocks cluster (centos 5), with PGI 7.2-3
>>>>>> compilers, Myricom MX2 hardware and drivers, and OpenMPI1.3
>>>>>> I installed the Myricom roll which has OpenMPI compiled with
>>>>>> gcc. I recently compiled the openmpi code w/ PGI.
>>>>>> I've the MPICH_F90 pointing to the right place, and we're
>>>>>> looking for the right includes and libs by means of
>>>>>> LD_LIBRARY_PATH, etc.
>>>>>> When I tried to run, I got the following error:
>>>>>> make[3]: Entering directory `/home/gerry/WRFv3/WRFV3/external/
>>>>>> RSL_LITE'
>>>>>> mpicc -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c c_code.c
>>>>>> PGC/x86-64 Linux 7.2-3: compilation completed with warnings
>>>>>> mpicc -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c buf_for_proc.c
>>>>>> PGC-S-0036-Syntax error: Recovery attempted by inserting
>>>>>> identifier .Z0000 before '(' (/share/apps/openmpi-1.3-pgi/
>>>>>> include/mpi.h: 889)
>>>>>> PGC-S-0082-Function returning array not allowed (/share/apps/
>>>>>> openmpi-1.3-pgi/include/mpi.h: 889)
>>>>>> PGC-S-0043-Redefinition of symbol, MPI_Comm (/share/apps/
>>>>>> openmpi-1.3-pgi/include/mpi.h: 903)
>>>>>> PGC/x86-64 Linux 7.2-3: compilation completed with severe errors
>>>>>> make[3]: [buf_for_proc.o] Error 2 (ignored)
>>>>>> Note that I had modified the makefile to use PGI in place of
>>>>>> gcc, and thus, the PGI-compiled openMPI.
>>>>>> Thanks, Gerry
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
> _______________________________________________
> users mailing list
> users_at_[hidden]