>> We would rather that OpenMPI use shared-mem (sm) module when running
>> intra-node processes.
> Doesn't PSM use shared memory to communicate between peers on the same
> node?
Possibly, yes (I'm not sure). Even if it does it appears to consume a
'hardware context' for each peer - this is what we want to avoid.
>> We believe that by using our scheduler's allocation policy (packing)
>> and considering our job mix, we might be able to add nodes to this
>> cluster using only one HCA per node (again, we would rather not use
>> 'shared contexts').
> Are you saying that you want Open MPI to not use PSM when the job
> entirely fits within a single node?
Yes, considering that the use of sm instead of psm would conserve
hardware contexts (and thus reduce the need for HCAs)
> If so, you might want to experiment with the pre-job hook in the job
> scheduler. You could try setting MCA parameters as environment
> variables (e.g., setenv OMPI_MCA_pml ob1 -- which would exclude the CM
> PML and therefore the PSM MTL) if your pre-job hook can tell if the job
> fits entirely on a single node.
>
> Does that help?
>
That's an interesting idea that I will investigate.
Thank you,
Tom
Tom Harvill
hcc.unl.edu
402.472.5660
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}