On Oct 30, 2008, at 8:13 AM, Reuti wrote:
> Am 30.10.2008 um 14:46 schrieb Brock Palen:
>> Any thoughts on this?
>> We are looking writing a script that parses $PBS_NODEFILE to create
>> a machinefile and using -machinefile
>> When we do that though we have to disable tm to avoid an error (-
>> mca pls ^tm) this is far from preferable.
> What about redefining the variable $PBS_NODEFILE pointing to an
> adjusted copy of the original file? With this, you could even use
> the TM startup of the nodes, as mpirun would use the adjusted file
Probably won't work. The problem is that TM doesn't launch based on
node name - it launches based on a TM-defined "launchid". This is
computed based on the location of the slot in the list in the
So if you mess with the nodefile, there is no guarantee that the
launchid we compute when reading the file will match what Torque
thinks it assigned. This has been fixed in 1.3, but remains a
constraint in 1.2
> When you know, that you request always 2 cores per node, the startup
> of any threads is up to you on your own. As you got two cores, it's
> -- Reuti
>> Any ideas to tell mpirun to only launch on half the cpus given to
>> it by PBS, but each cpu must have adjacent to it another cpu in the
>> same node?
>> Brock Palen
>> Center for Advanced Computing
>> On Oct 25, 2008, at 5:36 PM, Brock Palen wrote:
>>> We have a user with a code that uses threaded solvers inside each
>>> MPI rank. They would like to run two threads per process.
>>> The question is how to launch this? The default -byslot puts all
>>> the processes on the first sets of cpus not leaving any cpus for
>>> the second thread for each process. And half the cpus are wasted.
>>> The -bynode option works in theory, if all our nodes had the same
>>> number of core (they do not).
>>> So right now the user did:
>>> #PBS -l nodes=22:ppn=2
>>> export OMP_NUM_THREADS=2
>>> mpirun -np 22 app
>>> Which made me aware of the problem.
>>> How can I basically tell OMPI that a 'slot' is two cores on the
>>> same machine? This needs to work inside out torque based
>>> queueing system.
>>> Sorry If I was not clear about my goal.
>>> Brock Palen
>>> Center for Advanced Computing
>>> users mailing list
>> users mailing list
> users mailing list