Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Application logging
From: Barry Rountree (rountree_at_[hidden])
Date: 2008-05-07 13:32:29


On Wed, May 07, 2008 at 12:33:59PM -0400, Alberto Giannetti wrote:
> I need to log application-level messages on disk to trace my program
> activity. For better performances, one solution is to dedicate one
> processor to the actual I/O logging, while the other working
> processors would trace their activity through non-blocking, string
> message sends:

A few comments:

If you're using a cluster where each node has dedicated disk space, it
would probably be better to open a local file and log messages there.
After the application completes, then collate all the local files
together.

Even simpler is to open a file in the directory from which you started
the application. Using 16 older opterons and writing 2 lines per MPI
call per node, the overhead for doing this was small enough to be lost
in the noise. Call this after MPI_Init and after you've figured out
which node you are.

FILE *
initialize_logfile(int rank){

        char format[]="runtime.%02d.dat";
        char fname[64];
        sprintf(fname, format, rank);
        blr_logfile = fopen(fname, "w");
        assert(blr_logfile);
        return blr_logfile;
}

Then just fprintf(logfile, ...) as needed.

There are configurations where this won't work, of course, and it won't
scale to thousands of nodes. But I've found it to be rock-solid for my
work.

>
> /* LOGGER PROCESSOR MAIN LOOP */
> void logger(void)
> {
> MPI_Status status;
> char buf[LOGMSG_MAXSIZE];
>
> printf("Logger: Started\n");
>
> while( 1 ) {
> MPI_Recv(&buf, LOGMSG_MAXSIZE, MPI_CHAR, MPI_ANY_SOURCE,
> LOGMSG_TAG, MPI_COMM_WORLD, &status);
> buf[status.count] = '\0';
> /* ACTUAL I/O */
> printf("Processor %d ==> %s\n", status.MPI_SOURCE, buf);
> }
> }
>
>
> /* WORKER PROCESSOR LOGGING */
> void mylog(char* msg)
> {
> MPI_Request req;
> int msglen = strlen(msg);
>
> if( msglen > LOGMSG_MAXSIZE ) {
> /* Truncate */
> msg[LOGMSG_MAXSIZE-1] = '\0';
> msglen = LOGMSG_MAXSIZE;
> }
>
> /* Non-blocking send */
> MPI_Isend(msg, msglen, MPI_CHAR, LOGGER, LOGMSG_TAG,
> MPI_COMM_WORLD, &req);
> }
>
>
> I figured this must be a common problem in MPI applications and was
> wondering if there is any library available or related discussions.
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users