Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Paul H. Hargrove (PHHargrove_at_[hidden])
Date: 2007-07-14 19:59:14


Brian Barrett wrote:
> On Jul 14, 2007, at 8:26 AM, Dirk Eddelbuettel wrote:
>
>> Please let us (ie Debian's openmpi maintainers) how else we can
>> help. I am
>> ccing the porters lists (for hppa, m68k, mips) too to invite them
>> to help. I
>> hope that doesn't get the spam filters going... I may contact the
>> 'arm'
>> porters once we have a failure; s390 and sparc activity are not as
>> big these
>> days.
>
> Open MPI uses some assembly for things like atomic locks, atomic
> compare and swap, memory barriers, and the like. We currently have
> support for:
>
> * x86 (32 bit)
> * x86_64 / amd64 (32 or 64 bit)
> * UltraSparc (v8plus and v9* targets)
> * IA64
> * PowerPC (32 or 64 bit)
>
> We also have code for:
>
> * Alpha
> * MIPS (32 bit NEW ABI & 64 bit)
>
> This support isn't well tested in a while and it sounds like it
> doesn't work for MIPS. At one time, we supported the sparc v8
> target, but that The other platforms (hppa, mipsel (how is this
> different than MIPS?), s390, m68k) aren't at all supported by Open
> MPI. If you can get the real error messages, I can help on the MIPS
> issue, although it'll have to be a low priority.
>

As maintainer of the atomics code for two projects unrelated to OpenMPI,
I thought I'd pass on some of my insight. I'll not post any code here
to avoid any accidental license questions.

HPPA lacks an atomic compare-and-swap and is therefore probably a lost
cause. The Linux kernel uses HPPA's only atomic instruction,
load-and-clear, to implement a spinlock and a hashed table of spinlocks
to implement atomic operations. This works because the atomic_read and
atomic_set macros honor the spinlocks. This is not the case with ompi's
atomics, is it? OpenMPI appears to contain fragments of such an
array-of-spinlocks implementation for SPARCv8, but Brian's comments
suggest to me that this may no longer work.

ARM before v6 needs no memory barriers, but lacks atomic instructions
other than unconditional swap (though very few multi-processor systems
were built with earlier chips). However, on the libc-ports mailing
list (http://sourceware.org/ml/libc-ports/2005-10/msg00016.html) says of
the code used in glibc
<Quote>
/* Atomic compare and exchange. These sequences are not actually Atomic;
   there is a race if *MEM != OLDVAL and we are preempted between the two
   swaps. However, they are very close to atomic, and are the best that a
   pre-ARMv6 implementation can do without operating system support.
   LinuxThreads has been using these sequences for many years. */
</Quote>
So, ompi might try getting away with the same logic if an ARM port is
high priority for somebody. Alternatively, if one is on a new enough
Linux kernel (>= 2.6.12 IIRC) you get kernel support for CAS by calling
to a function in a "highpage" (like the VDSO on x86) that is implemented
natively on >=ARMv6 and traps to the kernel otherwise (the kernel
disables interrupts and then uses the not-quite-atomic sequence).

For ARMv6 you get a load-exclusive and store-exclusive pair, and you get
real memory barriers as well.

M68K has a CAS instruction and memory barriers are no-ops. This should
be an easy one to implement from the instruction set reference docs.

s390 is one I don't have any first-hand experience with but know from
peeking at the Linux kernel source that it has the eieio memory-barrier
instruction of early PPCs and a CAS instruction. Again, should be easy
from the ISA docs.

MIPS is supposed to work w/ ompi on IRIX, but there is no
atomic-mips-linux.s on OpenMPI 1.2.3. I was going to try to build 1.2.3
on an O2K (IRIX64 6.5 and gcc 3.3) today, but found that configure dies with
  configure: error: Could not determine global symbol label prefix
So, I'll not be pursuing that.

-Paul

> We don't currently have support for a non-assembly code path. We
> originally planned on having one, but the team went away from that
> route over time and there's no way to build Open MPI without assembly
> support right now.
>
>
> Brian
>

-- 
Paul H. Hargrove                          PHHargrove_at_[hidden]
Future Technologies Group
HPC Research Department                   Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900