Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] OMPI 1.4.5rc1 posted
From: David Singleton (David.Singleton_at_[hidden])
Date: 2011-12-13 15:44:14


Hi Chris,

Try setting OMPI_MCA_orte_tmpdir_base.

Going back to a related earlier OMPI users thread ("How to set up state-less node /tmp for OpenMPI usage"), here are sm pingpong latencies (using 1.4.3) for
session dirs on Lustre, an SSD and tmpfs:

[dbs900_at_v1490 ~/MPI]$ export OMPI_MCA_orte_tmpdir_base=/short/z00/dbs900/tmp
[dbs900_at_v1490 ~/MPI]$ mpirun -n 2 ./pp143
                       | Best | Average | Worst | Streaming
       n (test/loop) | usec MB/s | usec MB/s | usec MB/s | ( average, worst)
       1 ( 64/1100) | 0.3 30.45 | 0.3 29.63 | 0.3 26.00 | ( 0.3, 0.3)
       2 ( 64/1100) | 0.3 60.29 | 0.3 59.11 | 0.3 50.86 | ( 0.3, 0.3)
       4 ( 64/1100) | 0.3 117.92 | 0.3 115.01 | 0.3 96.18 | ( 0.3, 0.3)
...
[dbs900_at_v1490 ~/MPI]$ export OMPI_MCA_orte_tmpdir_base=/jobfs/flash/tmp
[dbs900_at_v1490 ~/MPI]$ mpirun -n 2 ./pp143
                       | Best | Average | Worst | Streaming
       n (test/loop) | usec MB/s | usec MB/s | usec MB/s | ( average, worst)
       1 ( 64/1100) | 0.3 30.24 | 0.3 29.54 | 0.3 24.86 | ( 0.3, 0.3)
       2 ( 64/1100) | 0.3 60.09 | 0.3 58.91 | 0.3 50.51 | ( 0.3, 0.3)
       4 ( 64/1100) | 0.3 117.92 | 0.3 115.39 | 0.3 99.55 | ( 0.3, 0.3)
...
[dbs900_at_v1490 ~/MPI]$ export OMPI_MCA_orte_tmpdir_base=/tmp
[dbs900_at_v1490 ~/MPI]$ mpirun -n 2 ./pp143
                       | Best | Average | Worst | Streaming
       n (test/loop) | usec MB/s | usec MB/s | usec MB/s | ( average, worst)
       1 ( 64/1100) | 0.3 30.34 | 0.3 29.62 | 0.3 24.96 | ( 0.3, 0.3)
       2 ( 64/1100) | 0.3 60.58 | 0.3 59.30 | 0.3 51.39 | ( 0.3, 0.3)
       4 ( 64/1100) | 0.3 116.94 | 0.3 114.69 | 0.3 99.32 | ( 0.3, 0.3)
...

This is why I was hoping sites could be given the opportunity to control that warning message. Some Lustre sites may not want their users getting that message.

David

On 12/13/2011 05:10 PM, Christopher Samuel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 09/12/11 09:13, Jeff Squyres wrote:
>
>> Gearing up for a pre-Christmas release -- please test!
>
> Had a quick test here (we're still on 1.4.2 as our default).
>
> It now complains that $TMPDIR points to our Panasas filesystem,
> and there's a minor typo in its message, saying:
>
> # Note, too, that system administrators can set a list of filesystems
> # where Open MPI is disallowed from creating temporary files by settings
> # the MCA parameter "orte_no_session_dir".
>
> I think you want s/settings/setting/ there.
>
> Also I can not seem to make it accept the orte_no_session_dir settings,
> neither:
>
> mpirun -mca orte_no_session_dir /vlsci ./hello_c
>
> nor:
>
> mpirun -mca orte_no_session_dir /vlsci/tmp ./hello_c
>
> nor:
>
> $ export OMPI_MCA_orte_no_session_dir=/vlsci
>
> nor:
>
> export OMPI_MCA_orte_no_session_dir=/vlsci/tmp
>
> prevent it whinging about (for example):
>
> Local host: bruce001
> File Name: /vlsci/tmp/979325.bruce-m.vlsci.unimelb.edu.au/openmpi-sessions-samuel_at_bruce001_0/14488/1/shared_mem_pool.bruce001
>
> Is there a way to tell it to use /tmp without changing what
> $TMPDIR is set to ?
>
> cheers,
> Chris