Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] [gridengine users] h_vmem in jobs with mixture of openmpi and openmp
From: Reuti (reuti_at_[hidden])
Date: 2012-09-06 17:26:13


Am 06.09.2012 um 13:21 schrieb Schmidt U.:

>> <snip>
> If h_vmem is defined in the script, what sense is then an additional vf option in the script ? The h_vmem has per default higher value than vf, so it must fit first to let the job run.

If you want to avoid swapping, both should have the same value anyway.

>>
>>> pe:
>>> pe_name openmp_6
>>> slots 3168
>>> user_lists standard
>>> xuser_lists NONE
>>> start_proc_args /bin/true
>>> stop_proc_args /bin/true
>>> allocation_rule 6
>>> control_slaves TRUE
>>> job_is_first_task FALSE
>>> urgency_slots min
>>> accounting_summary FALSE
>>>
>>> job script:
>>> #$ -N test
>>> #$ -cwd
>>> #$ -o $JOB_ID.out
>>> #$ -e $JOB_ID.err
>>> #$ -l h_rt=150:00:00
>>> #$ -l vf=2.3G
>>> #$ -l h_vmem=3G
>>> #$ -pe openmp_6 72
>>> export OMP_NUM_THREADS=6
>>> export MKL_NUM_THREADS=($OMP_NUM_THREADS)
>>> mpirun --mca btl openib,self -pernode -np 12 /my_mixed_job
>> With this you will get 12 machines, and on each you can use 6 threads. As all (threads on a machine) will work on the same memory, this shouldn't be a problem. But you are using MKL with $OMP_NUM_THREADS too, which could create locally 36 processes as a result. Therefore I use unthreaded versions of MKL/ACML/ATLAS usually.
> Thanks for that hint, as a workaround a could check all scripts for MKL_NUM_THREADS and set it to 1 by jsv ?

Yes.

But you could also try the opposite, i.e. OMP_NUM_THREADS=1 and MKL_NUM_THREADS=$NSLOTS

Depends of the application what's better.

-- Reuti

> Udo
>>
>> -- Reuti
>>
>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> https://gridengine.org/mailman/listinfo/users
>