Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] running "openmpi" with "knem"
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2012-12-01 17:53:47

Le 01/12/2012 12:45, Leta Melkamu a écrit :
> Hello there,
> I have some doubts on the use of knem with openmpi, everything works fine.
> However, it is a bit not clear on the usage of knem flags while
> running my open-mpi program.
> Something like --mca btl_sm_knem_dma_min 4860 is enough or I have to
> add more flag like --mca btl_sm_eager_limit 4276 in the same run? or
> can you please suggest me a good documentation link about knem flag
> usage, I have tried to look around but no good info regarding those
> staff. otherwise I will end up to test each command with different
> flag value for each run.

What are you trying to do?

* Use knem for direct copy through kernel for medium/large messages:
(common use case)
"--mca btl_sm_use_knem 1" is enough. You can "cat /dev/knem" before and
after a run to see that knem counters have increased, which means OMPI
successfully passed some copy requests to knem.

* Use knem for short messages:
"--mca btl_sm_eager_limit 4276" may help. But I am not sure that's a
good idea since knem was designed for large messages.

* Offload knem copies to I/OAT hardware on Intel servers. That's what
"btl_sm_knem_dma_min" is for. Not sure you really want to do that
either, it's not much interesting on current Intel servers.