Hi Brice,
>The flag says "when the first touch occurs and the physical memory is allocated for real, don't allocate on the local node (default), but >rather allocate where specified by set_membind".
If is it already allocated for real, how set_membind() can allocate on other node?
So, what's the difference between HWLOC_MEMBIND_BIND and HWLOC_MEMBIND_FIRSTTOUCH?
>set_membind_nodeset() with BIND and the nodeset containing the nodes where physical memory should be allocated.
Mm In my tests, I have a process that allocates a memory, thread 0 is bound on node 0 and thread 1 on node1:
malloc(array)...
omp parallel region
if(tid==0)
set_area_membind_nodeset( first half of array)
if (tid==1)
set_area_membind_nodeset( second half of array)
end parallel region
for(i...)
array(i)
Checking the free memory on two nodes, supposing array is 1 Gb, I see 500 mb less on first node and 500 mb less on second node and it is ok, because first touch is done by the process, but on memory already bound on specific nodes.
Doing the follow test:
omp parallel region
if(tid==0){
malloc(array)...
set_area_membind(HWLOCMEMBIND_BIND, node 0)
}
if (tid==1){
set_area_membind(HWLOCMEMBIND_BIND, node 1)
}
end parallel region
Now, checking the free memory on two nodes, I see 1 GB less on second node, because thread 1 is doing first touch. set_membind() for first thread has no effect, I suppose, or I dont' understand how it works, because in the second example only first touch appears to have some effects, indipendently which hwloc function I'm using.
Sorry, but it is quite difficult to understand .. :(
2011/9/25 Brice Goglin
<Brice.Goglin@inria.fr>
Le 25/09/2011 11:14, Gabriele Fatigati a écrit :
I report my questions in a different way (in the first question i
did a mistake):
1) I don't understand the means of set_membind()
function. Why I should to allocate in a node "near" my
cpuset and not in my local node (where thread or process
runs?)
It's exactly the same. Your local node is near the cpuset that
contains the CPUs that are close to this node.
2) Which is the
behaviour of HWLOC_MEMBIND_BIND flag?
From the manual:
"Allocate memory on the specified nodes."
It means that I can allocate without binding the
memory?
It's about physical memory allocation (first touch causing a fault
causing a page to be allocated), not about virtual memory (malloc).
What happens if one thread allocate and another
thread in another node read/write for the first time this
memory? In a little example I see the memory is allocated on
the second thread, not where first thread does malloc(). So,
when I have to use HWLOC_MEMBIND_BIND flag? Or it has nothing
to do with binding?
If the effective allocation is done when first
thread touch the memory, which is the means of this flag?
The flag says "when the first touch occurs and the physical memory
is allocated for real, don't allocate on the local node (default),
but rather allocate where specified by set_membind".
2) My goal is to replicate the behaviour of
set_area_membind_nodeset() in some manner for all futures
allocation without call this function each time I allocate
some memory. Is it possible to do this?
set_membind_nodeset() with BIND and the nodeset containing the nodes
where physical memory should be allocated.
Brice
_______________________________________________
hwloc-users mailing list
hwloc-users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it