I would disagree with your statement that the available fortran options can't pass a cost-benefit analysis. I have found that for scientific programming (e.g., Livermore Fortran Kernels and actual PDE solvers) that code produced by the intel compiler runs 25 to 55% faster than code from gfortran or g95. Looking at the cost of adding processors with g95/gfortran to get the same throughput as with ifort you recover the $549 compiler cost real quickly.

Doug Reeder 
On Mar 6, 2008, at 9:20 AM, Gregory John Orris wrote:

Sorry for the long delay in response. 

Let's get back to the beginning:
My original compiler configuration was gcc from the standard Leopard Developer Tools supplied off the installation DVD. This version was 4.0.1. However, it has been significantly modified by Apple to work with Leopard. If you haven't used Apple's Developer Environment, you're missing out on something. It's pretty sweet. But the price you pay for it is no fortran support (not usually a problem for me but it is relevant here) and usually a somewhat time-lagged compiler. I'm not as plugged into Apple as perhaps I should be, but I can only imagine that their philosophy is to really over test their compiler. Gratis, Apple throws into it's "frameworks" a shared library called vecLib, that includes machine optimized BLAS and CLAPACK routines. Also, with Leopard, Apple has integrated open-mpi (yea!). But they have once again not included fortran support (boo!).

Now, to get fortran on a Mac you have several options (most of which cannot really survive the cost-benefit analysis of a competent manager), but a perfectly fine freeware option is to get it off of This version is based on gcc 4.3.0. There are a few legitimate reasons to stick with Apple's older gcc. As it's not really a good idea to try an mix libraries from one compiler version with another. Especially here, because (without knowing precisely what Apple has done) there is a tremendous difference in execution speed of code written with gcc 4.0 and 4.1 as opposed to 4.2 and later. (This has been well documented on many systems.) Also, out of a bit of laziness, I really didn't want to go to the trouble of re-writing (or finding) all of the compiler scripts in the Developer Environment to use the new gcc. 

So, I compiled open-mpi-1.2.5 with gcc, g++ 4.0.1, and gfortran 4.3. Then, I compiled BLACS and ScaLAPACK using the configuration from the open-mpi FAQ page. Everything compiles perfectly ok, independent of whether you choose 32 or 64 bit addressing. First problem was that I was still calling mpicc from the Apple supplied openmpi and mpif77 from the newly installed distribution. Once again, I've not a clue what Apple has done, but while the two would compile items together, they DO NOT COMMUNICATE properly in 64-bit mode. MPI_COMM_WORLD even in the test routines of openMPI would fail! This is the point at which I originated the message asking if anyone had gotten a 64-bit version to actually work. The errors were in libSystem and were not what I'd expect from a simple openmpi error. I believe this problem is caused by a difference in how pointers were/are treated within gcc from version to version. Thus mixing versions essentially caused failure within the Apple supplied openmpi distribution and the new one I installed.

How to get over this hurdle? Install the complete gcc 4.3.0 from the site and recompile EVERYTHING! 

You might think you were done here, but there is one (or actually four) additional problem(s). Now NONE of the complex routines worked. All of the test routines returned failure. And I tracked it down the the fact that pzdotc, pzdotu, pcdotc, and pcdotu inside of the PBLAS routines were failing. Potentially this was a much more difficult problem, since rewriting these codes is really not what I'm paid to do. Tracing down these errors further I found that the actual problem is with the zdotc, zdotu, cdotc, and cdotu BLAS routines inside of Apple's vecLib. So, the problem seemed as though a faulty manufacturer supplied and optimized library was not functioning properly. Well, as it turns out there is a peculiar difference (again) between versions of the gcc suite in how it regards, returned values from complex fortran functions (I'm only assuming this since the workaround was successful). This problem has been know for some time now (perhaps 4 years or more). See,

How to get over this hurdle? Install ATLAS, CLAPACK, and CBLAS off the web site, and compile them with the gcc 4.3.0 suite.

So, where am I now? BLACS and ScaLAPACK, and PBLAS work in 64-bit mode with CLAPACK-3.1.1, ATLAS 3.8.1, Open-MPI-1.2.5, and GCC 4.3.0 and link with ATLAS and CLAPACK and NOT vecLib!

Long way of saying that the problem appears to be solved, but not well documented (until now)!


On Mar 6, 2008, at 8:25 AM, Terry Dontje wrote:

Ok, I think I found the cause of the SPARC segv when trying to use a
64-bit compiled Open MPI library.  If one does not set the WHATMPI
variable in the it defaults to UseF77Mpi which assumes all
handles are ints.  This is a correct assumption if you are using the F77
interfaces but the way BLACS seems to compile for Open MPI it uses the C
versions.  So the handles are stored as 32 bits in BLACS and passed to
the C Open MPI interfaces which expects 64 bits.  In cases where your
addresses need more than 32 bits this will cause MPI to segv when passed
an invalid address due to this coersion.

So by setting "WHATMPI= -DUseCMpi" I've gotten the SPARC version of
BLACS compiled for 64 bits to pass its tests without segv'ing.  I do
believe this issue actually exists for other platforms (ie AMD64 and
IA64) with other OSes and compilers.  Just that we've been lucky that
MPI_COMM_WORLD is allocated such that it has an address that fits in 32
bits.  I am amazed still that we haven't seen this fail in user codes.  
Note, I have not confirmed this failure with a test case but the code
stack in dbx looks the same on X64 platforms as the code on SPARC except
the address is smaller on the former.

Greg, I would be interested in knowing if you are still seeing the
problem on Leopard and whether the above setting helps any.


*Subject:* Re: [OMPI users] ScaLapack and BLACS on Leopard
*From:* Terry Dontje (/Terry.Dontje_at_[hidden]/)
*Date:* 2008-03-03 07:34:17

What kind of system lib errors are you seeing and do you have a stack
trace? Note, I was trying something similar with Solaris and 64-bit on
a SPARC machine and was seeing segv's inside the MPI Library due to a
pointer being passed through an integer (thus dropping the upper 32
bits). Funny thing is it all works under Solaris on AMD64 or IA-64


Date: Thu, 28 Feb 2008 17:50:28 -0500
From: Gregory John Orris <gregory.orris_at_[hidden]>
Subject: [OMPI users] ScaLapack and BLACS on Leopard
To: Open MPI Users <users_at_[hidden]>
Message-ID: <528FD4C0-6157-49CB-80E6-1C62684E4545_at_[hidden]>
Content-Type: text/plain; charset="us-ascii"

Hey Folks,

Anyone got ScaLapack and BLACS working and not just compiled under
OSX10.5 in 64-bit mode?
The FAQ site directions were followed and every thing compiles just
fine. But ALL of the single precision routines and many of the double
precisions routines in the TESTING directory fail with system lib

I've gotten some interesting errors and am wondering what the magic
touch is.


users mailing list

users mailing list