Data Compression of numerical data sets with the BigWhoop library in From Machine Learning to Deep Learning: a concise introduction

Description

Both Machine and Deep Learning methods and examples will be presented, together with their implementation on HLRS systems. The first part will be an introduction to basic methods in Machine Learning, including pre-processing and supervised learning using Apache Spark. The course will then move on to elements of supervised Deep Learning on real data to classify annotated images of waste in the wild.

Data Compression on Day 3

Given the deluge of information needed to power machine and deep learning methods, it is imperative to think about effective data processing strategies. Therefore, the course will conclude with an introduction to data compression using the BigWhoop library (part of the EXCELLERAT Data Exchange and Workflow Portal). As an efficient data reduction tool, BigWhoop can be applied to generic numerical datasets to minimize I/O bottlenecks and optimize data storage.

The lectures are interleaved with many hands-on sessions using Jupyter Notebooks and scripts on HLRS systems.

Course Material

• In the section "Event material" you can find the slides about Data Compression and Big Whoop.
• Please contact  if you would like to access the whole course material.

Web-url

https://www.hlrs.de/training/2021/DL3/

Presenters

Khatuna Kakhiani (HLRS)Patrick Vogler (HLRS)Lorenzo Zanon (HLRS)Andrea Beck (USTUTT - IAG)

URL

https://www.hlrs.de/training/2021/DL3/