Argonne National Laboratory

Large-Scale Lossy Data Compression Based On A Priori Error Estimator In A Spectral Element Code

TitleLarge-Scale Lossy Data Compression Based On A Priori Error Estimator In A Spectral Element Code
Publication TypeReport
Year of Publication2016
AuthorsMarin, O, Schanen, M, Fischer, PF
Abstract

I/O is becoming an increasing bottleneck on large-scale computer systems. We present a highly scalable lossy data compression algorithm with controllable committed error based on an a priori estimator. The scientific data field is compacted via the discrete Chebyshev transform (DCT),
truncated at a level within a user-specified tolerance, and subsequently compressed before writing to disk by using a version of Huffman encoding. The flexibility of the DCT transform and computational efficiency doubled by the a priori error estimator allows a dynamic compression ratio which can
achieve as high as 97% compression for data visualization even in cases as complex as fully developed turbulent flow. In resilience problems however, the compression ratio is problem dependent. The algorithm is implemented in the spectral-element code Nek5000 and tested on highly turbulent large-
scale simulation data of up to 10 billion degrees of freedom. The implementation via tensor products is highly efficient, leading to a decrease of flops in matrix-vector multiplications in three dimensions from O(N6) down to O(N4).
 

PDFhttp://www.mcs.anl.gov/papers/P6024-0616.pdf