Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] MPI_Alltoall with Vector Datatype
From: Spenser Gilliland (spenser_at_[hidden])
Date: 2014-05-07 17:41:41


George,

> Do you mind posting your working example here on the mailing list?
> This might help future users understanding how to correctly use the
> MPI datatype.

No problem. I wrote up this simplified example so others can learn to
use the functionality. This is a matrix transpose operation using MPI
vector derived types and collective communication. (Terms added to aid
search engines)

The result of running this program with a two node cluster is

Matrix =
 0: 0 1 2 3 4 5 6 7
 0: 8 9 10 11 12 13 14 15
 0: 16 17 18 19 20 21 22 23
 0: 24 25 26 27 28 29 30 31
 1: 32 33 34 35 36 37 38 39
 1: 40 41 42 43 44 45 46 47
 1: 48 49 50 51 52 53 54 55
 1: 56 57 58 59 60 61 62 63
Matrix =
 0: 0 8 16 24 4 12 20 28
 0: 1 9 17 25 5 13 21 29
 0: 2 10 18 26 6 14 22 30
 0: 3 11 19 27 7 15 23 31
 1: 32 40 48 56 36 44 52 60
 1: 33 41 49 57 37 45 53 61
 1: 34 42 50 58 38 46 54 62
 1: 35 43 51 59 39 47 55 63
Matrix =
 0: 0 8 16 24 32 40 48 56
 0: 1 9 17 25 33 41 49 57
 0: 2 10 18 26 34 42 50 58
 0: 3 11 19 27 35 43 51 59
 1: 4 12 20 28 36 44 52 60
 1: 5 13 21 29 37 45 53 61
 1: 6 14 22 30 38 46 54 62
 1: 7 15 23 31 39 47 55 63

/* file: transpose.c
 *
 * Description: Transpose operation using MPI derived datatypes with
 * MPI collective communications.
 *
 * Author: Spenser Gilliland <spenser_at_[hidden]>
 */

#include <mpi.h>
#include <stdio.h>
#include <unistd.h>

#define N (8)
float matrix[N][N];

void print_matrix(int wrank, int wrows) {
    int i , j;

    MPI_Barrier(MPI_COMM_WORLD);
    if(wrank == 0) printf("Matrix = \n");
    MPI_Barrier(MPI_COMM_WORLD);

    for(i = 0; i < N; i++) {
        if(i >= wrank*wrows && i < (wrank+1)*wrows) {
            printf("%2d:", wrank);
            for(j = 0; j < N; j++) {
                printf("%6.2g", matrix[i][j]);
            }
            printf("\n");
        }
        usleep(1);
        MPI_Barrier(MPI_COMM_WORLD);
    }
}

int main(int argc, char *argv[]) {
    int wsize, wrank, wrows;
    int i, j, k;
    int row, col;
    float temp;
    MPI_Datatype mpi_all_unaligned_t, mpi_all_t;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &wsize);
    MPI_Comm_rank(MPI_COMM_WORLD, &wrank);

    wrows = N / wsize;

    MPI_Type_vector(N, wrows, N, MPI_FLOAT, &mpi_all_unaligned_t);
    MPI_Type_create_resized(mpi_all_unaligned_t, 0,
wrows*sizeof(float), &mpi_all_t);
    MPI_Type_free(&mpi_all_unaligned_t);
    MPI_Type_commit(&mpi_all_t);

    for(i = 0; i < N; i++) {
        /* Initialize data on the rows of the matrix owned by this rank */
        if (i >= wrank*wrows && i < (wrank+1)*wrows) {
            for(j = 0; j < N; j++) {
                matrix[i][j] = i*N + j;
            }
        }
    }

    print_matrix(wrank, wrows);

    /* Local Transpose */
    row = wrank*wrows;
    for(k = 0; k < wsize; k++) {
        col = k*wrows;
        for( i = 0; i < wrows; i++) {
            for(j =i+1; j < wrows; j++) {
                temp = matrix[row+i][col + j];
                matrix[row + i][col + j] = matrix[row + j][col + i];
                matrix[row + j][col + i] = temp;
            }
        }
    }

    print_matrix(wrank, wrows);

    /* Global Transpose */
    MPI_Alltoall(matrix[wrank*wrows], 1, mpi_all_t,
                 matrix[wrank*wrows], 1, mpi_all_t,
                 MPI_COMM_WORLD);

    print_matrix(wrank, wrows);

    MPI_Finalize();
    return 0;
}

Thanks,
Spenser