Le 01/12/2012 12:45, Leta Melkamu a écrit :
Hello there, 

I have some doubts on the use of knem with openmpi, everything works fine.
However, it is a bit not clear on the usage of knem flags while running my open-mpi program. 
Something like --mca btl_sm_knem_dma_min 4860 is enough or I have to add more flag like --mca btl_sm_eager_limit 4276 in the same run? or can you please suggest me a good documentation link about knem flag usage, I have tried to look around but no good info regarding those staff. otherwise I will end up to test each command with different flag value for each run.

What are you trying to do?

* Use knem for direct copy through kernel for medium/large messages: (common use case)
"--mca btl_sm_use_knem 1" is enough. You can "cat /dev/knem" before and after a run to see that knem counters have increased, which means OMPI successfully passed some copy requests to knem.

* Use knem for short messages:
"--mca btl_sm_eager_limit 4276" may help. But I am not sure that's a good idea since knem was designed for large messages.

* Offload knem copies to I/OAT hardware on Intel servers. That's what "btl_sm_knem_dma_min" is for. Not sure you really want to do that either, it's not much interesting on current Intel servers.