Dear All,


I meet a strange problem using MPI_Allgatherv when the send buf size becomes bigger but not that big.

The following simple testing code of MPI_Allgatherv runs fine when I use processors on same machine.

However, when I use processors on different machine, I see following problems on different send buf size.


1. Bufsize = 2^28, run on 2 processors. OK

2. Bufsize = 2^28, run on 4 processors. Error

[btl_tcp_frag.c:209:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv error (0xffffffff85f526f8, 2147483592) Bad address(1)

3. Bufsize = 2^28 – 1, run on 4 or 5 or 6 processors. OK

4. Bufsize = 2^29, run on 2 processors. Error shown below OR hang there with no error shown

[btl_tcp_frag.c:209:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv error (0xffffffff964605d0, 2147483632) Bad address(1)

5. Bufsize =2^29-1, run on 2 processors. OK

6. Bufsize =2^29-1 , run on 4 processors. hang there with no error shown


I suspect the error is that when receive buf reaches 2^30, mca_btl_tcp_frag_recv will have some number close to int limit. However, this suspect is not well supported by the testcase above. And I have no clue why there is this limit. The problem may also be in network configuration since same machine doesn’t have this problem.


Please take a look at the following test code and help me to solve the problem.


#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <unistd.h>

#include <time.h>

#include "mpi.h"

int main(int argc, char ** argv)

{

int myid,nproc;

long i,j;

long size;

long bufsize;

int *rbuf;

int *sbuf;

char hostname[MPI_MAX_PROCESSOR_NAME];

int len;

size = (long) 2*1024*1024*1024-1;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &myid);

MPI_Comm_size(MPI_COMM_WORLD, &nproc);

MPI_Get_processor_name(hostname, &len);

printf("I am process %d with pid: %d at %s\n",myid,getpid(),hostname);

sleep(2);


if (myid == 0)

printf("size : %ld\n",size);

sbuf = (int *) calloc(size,sizeof(MPI_INT));

if (sbuf == NULL) {

printf("fail to allocate memory of sbuf\n");

exit(1);

}

rbuf = (int *) calloc(size,sizeof(MPI_INT));

if (rbuf == NULL) {

printf("fail to allocate memory of rbuf\n");

exit(1);

}

int *recvCount = calloc(nproc,sizeof(int));

int *displ = calloc(nproc,sizeof(int));

bufsize = 268435456; //which is 2^28

for(i=0;i<nproc;++i) {

recvCount[i] = bufsize;

displ[i] = bufsize*i;

}

for (i=0;i<bufsize;++i)

sbuf[i] = myid+i;

printf("buffer size: %ld recvCount[0]:%d last displ index:%d\n",bufsize,recvCount[0],displ[nproc-1]);

fflush(stdout);


MPI_Allgatherv(sbuf,recvCount[0], MPI_INT,rbuf,recvCount,displ,MPI_INT,

MPI_COMM_WORLD);


printf("OK\n");

fflush(stdout);

MPI_Finalize();

return 0;

}