Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Message compression in OpenMPI
From: Tomas Ukkonen (tomas.ukkonen_at_[hidden])
Date: 2008-04-24 08:26:35


George Bosilca wrote:
> The paper you cited, while presenting a particular implementation
> doesn't bring present any new ideas. The compression of the data was
> studied for long time, and [unfortunately] it always came back to the
> same result. In the general case, not worth the effort !
>
> Now of course, if one limit itself to very regular applications (such
> as the one presented in the paper), where the matrices involved in the
> computation are well conditioned (such as in the paper), and if you
> only use MPI_DOUBLE (\cite{same_paper}), and finally if you only
> expect to run over slow Ethernet (1Gbs) (\cite{same_paper_again})...
> then yes one might get some benefit.
>
Yes, you are probably right that its not worth the effort in general and
especially not in HPC environments where you have very fast network.

But I can think of (rather important) special cases where it is important

- non HPC environments with slow network: beowulf clusters and/or
  internet + normal PCs where you use existing workstations and network
  for computations.
- communication/io-bound computations where you transfer
  large redundant datasets between nodes

So it would be nice to be able to turn on the compression (for spefic
communicators and/or data transfers) when you need it.

-- 
Tomas
>   george.
>
> On Apr 22, 2008, at 9:03 AM, Tomas Ukkonen wrote:
>
>> Hello
>>
>> I read from somewhere that OpenMPI supports
>> some kind of data compression but I couldn't find
>> any information about it.
>>
>> Is this true and how it can be used?
>>
>> Does anyone have any experiences about using it?
>>
>> Is it possible to use compression in just some
>> subset of communications (communicator
>> specific compression settings)?
>>
>> In our MPI application we are transferring large
>> amounts of sparse/redundant data that compresses
>> very well. Also my initial tests showed significant
>> improvements in performance.
>>
>> There are also articles that suggest that compression
>> should be used [1].
>>
>> [1] J. Ke, M. Burtcher and E. Speight.
>> Runtime Compression of MPI Messages to Improve the
>> Performance and Scalability of Parallel Applications.
>>
>>
>> Thanks in advance,
>> Tomas
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users