Well, looks like we found the problem.

The memory free callback was incorrect. We were just looking for the base address of the free in the tree. Here is why this didn't work 
Probably wouldn't compile but works for an explanation: 

buf = malloc(40*1024);  /* malloc  10 pages */ 

 /* send the second half of the buffer to the peer */ 
/* note that leave_pinned will register and cache only what it sees in the send */ 
MPI_Send(buf+5*4*1024, 5*4*1024, ........ );   

/* free the buffer, mpi will try to find the registration in the tree 
    based on the address, buf,, but won't find it so the registration 
    remains */ 
free(buf); 


So since the registration is  left in the tree, a future malloc may obtain a virtual address that is within the base and bound of the registration. When this memory is later freed we try to deregister the entire registration, part of which might be in use by another buffer, it could even be in the process of an RDMA operation. 

Anyway, I have modified the code and we are now passing a smaller linpack run with leave_pinned and the mem hooks enabled without using any mallopt trickiness. 

Thanks, 

Galen 



On Sep 25, 2005, at 10:58 AM, Galen M. Shipman wrote:

Well, after adding a bunch of debugging  output, I have found the following. 

With both leave_pinned and use_mem_hook enabled on a linpack run we get the assertion error on the memory callback in linpack. That is to say, there is a free occurring in the middle of a registration. 
 
At the point of assert we have NOT resized any registrations. 
The existing registrations in the tree are: 

Existing registrations:
BaseBoundLength
2416153602448416073226248
2448416082464288071587200
2464288082480160071587200
2480196482512458953226248
Tyring to free
247917216
From
BaseBound
246428808248016007



When we get the assert, we are trying to free: 247917216, which is in the middle of the registration. Note we have NOT resized any registrations so I am confident there is not an issue with either the tree or the resize at least as far as linpack is concerned. 
Here is the callstack: 

#0  0x0000002a95f079c9 in raise () from /lib/libc.so.6
#1  0x0000002a95f08e6e in abort () from /lib/libc.so.6
#2  0x0000002a95f01690 in __assert_fail () from /lib/libc.so.6
#3  0x0000002a9571b200 in mca_mpool_base_mem_cb (base=0xec6eaa0, size=31624,
    cbdata=0x0) at mpool_base_mem_cb.c:53
#4  0x0000002a9587fe0d in opal_mem_free_release_hook (buf=0xec6eaa0,
    length=31624) at memory.c:121
#5  0x0000002a9588bd12 in opal_mem_free_free_hook (ptr=0xec6eaa0,
    caller=0x42b052) at memory_malloc_hooks.c:66
#6  0x000000000042b052 in ATL_dmmIJK ()
#7  0x000000000064f9b1 in ATL_dgemmNN ()
#8  0x000000000057722b in ATL_dgemmNN_RB ()
#9  0x0000000000577fc3 in ATL_rtrsmRUN ()
#10 0x000000000042c63c in ATL_dtrsm ()
#11 0x0000000000423c1e in atl_f77wrap_dtrsm__ ()
#12 0x0000000000423a94 in dtrsm_ ()
#13 0x0000000000411192 in HPL_dtrsm (ORDER=17933, SIDE=17933, UPLO=8,
    TRANS=4294967295, DIAG=0, M=23458672, N=0, ALPHA=1, A=0x7fbfffefa0, LDA=0,
    B=0x202, LDB=0) at HPL_dtrsm.c:949
#14 0x000000000040cfb6 in HPL_pdupdateTT (PBCST=0x0, IFLAG=0x0,
    PANEL=0x165f040, NN=-1) at HPL_pdupdateTT.c:362
#15 0x000000000041936f in HPL_pdgesvK2 (GRID=0x7fbffff4a0, ALGO=0x7fbffff460,
    A=0x7fbffff260) at HPL_pdgesvK2.c:178
#16 0x000000000040d6f7 in HPL_pdgesv (GRID=0x7fbffff4a0, ALGO=0x460d,
    A=0x7fbffff260) at HPL_pdgesv.c:107
#17 0x0000000000405b10 in HPL_pdtest (TEST=0x7fbffff430, GRID=0x7fbffff4a0,
    ALGO=0x7fbffff460, N=10000, NB=80) at HPL_pdtest.c:193
#18 0x0000000000401840 in main (ARGC=1, ARGV=0x7fbffff928)
    at HPL_pddriver.c:223

Note that the free occurs in the ATLAS libraries, I will look into re-building linpack with another BLAS library to see what happens. Any other suggestions? 

Thanks, 

Galen 

_______________________________________________
devel mailing list