Data Compression of numerical data sets with the BigWhoop library

Description

Description

Over the years, a steady increase in computing power has enabled scientists and engineers to develop increasingly complex applications for machine learning and scientific computing. However, while these applications hold the promise of solving some of the most challenging problems we face today, their never-ending data hunger has revealed an ever-increasing I/O bottleneck. It is therefore imperative that we develop I/O strategies that will help us to more effectively use the raw power of our high-performance computing systems and improve the usability and efficiency of our tools.

To achieve this goal, the BigWhoop compression library, based on the JPEG 200 standard, was developed. It enables the efficient and lossy compression of numerical data while minimizing the loss of information and introduction of compression artefacts. 

Presenters

Patrick Vogler (HLRS)

URL

http://hlrs.de/training/2020/DL1/