[1] Reis, Joe, and Matt Housley. Fundamentals of data engineering. " O'Reilly Media, Inc.", 2022.
[2] Haelen, B., & Davis, D. (2023). Delta Lake: Up and Running: Modern Data Lakehouse Architectures with Delta Lake. " O'Reilly Media, Inc.".
Data Engineering
| Type of study | Follow-up Master |
|---|---|
| Language of instruction | Czech |
| Code | 460-4178/01 |
| Abbreviation | DI |
| Course title | Data Engineering |
| Credits | 4 |
| Coordinating department | Department of Computer Science |
| Course coordinator | doc. Ing. Radim Bača, Ph.D. |
Subject syllabus
Lectures:
1. History of OLAP, data products, and the role and tools of data engineers and their place in data-oriented companies, ETL, ELT, and data backfilling. Separating data storage from data processing. Delta Lake – Parquet, ORC. Lakehouse – Delta Lake, Iceberg, Hudi
2. Data warehouse modelling. Star, Snowflake. Data Vault
3. Medallion architecture and its practical use. Data cleaning, validation, and quality.
4. Automation using the Data Engineering as Code (dbt) approach.
5. Orchestration. Data catalogues and metadata management.
6. Loading strategies. Batch vs. stream data processing. Schema versioning and Change Data Capture (CDC).
7. Introduction to data analysis. Dimensional querying. Introduction to DAX.
8. Basic data analysis in Power BI.
Exercises:
1. Introduction to development environments (logging into services). Working with Parquet and Lakehouse files using DuckDB
2. Converting the Sakila database to a star schema (DBDiagrams, SQL Server)
3. Integrating two Sakila databases using Data Vault
4. Cleaning and integrating movie ratings (find dataset on Kaggle) – PowerM, SQL
Transferring work from previous lessons to DBT
5. Trying to write ETL orchestration using Airflow. The input data is stored in an S3 bucket.
6. Insert metadata into the prepared system (Open metadata, DuckLake) — Automate metadata collection. Prepare specifications for your own project.
7. Try multiple loading strategies for one example: CDC, full-load, batch, streaming. Get acquainted with Power BI and write your first DAX.
8. Power BI. Project consultation.
9. Project submission
Translated with DeepL.com (free version)
1. History of OLAP, data products, and the role and tools of data engineers and their place in data-oriented companies, ETL, ELT, and data backfilling. Separating data storage from data processing. Delta Lake – Parquet, ORC. Lakehouse – Delta Lake, Iceberg, Hudi
2. Data warehouse modelling. Star, Snowflake. Data Vault
3. Medallion architecture and its practical use. Data cleaning, validation, and quality.
4. Automation using the Data Engineering as Code (dbt) approach.
5. Orchestration. Data catalogues and metadata management.
6. Loading strategies. Batch vs. stream data processing. Schema versioning and Change Data Capture (CDC).
7. Introduction to data analysis. Dimensional querying. Introduction to DAX.
8. Basic data analysis in Power BI.
Exercises:
1. Introduction to development environments (logging into services). Working with Parquet and Lakehouse files using DuckDB
2. Converting the Sakila database to a star schema (DBDiagrams, SQL Server)
3. Integrating two Sakila databases using Data Vault
4. Cleaning and integrating movie ratings (find dataset on Kaggle) – PowerM, SQL
Transferring work from previous lessons to DBT
5. Trying to write ETL orchestration using Airflow. The input data is stored in an S3 bucket.
6. Insert metadata into the prepared system (Open metadata, DuckLake) — Automate metadata collection. Prepare specifications for your own project.
7. Try multiple loading strategies for one example: CDC, full-load, batch, streaming. Get acquainted with Power BI and write your first DAX.
8. Power BI. Project consultation.
9. Project submission
Translated with DeepL.com (free version)
Literature
Advised literature
[1] Zagni, Roberto. "Data Engineering with dbt." Birmingham: Packt Publishing Ltd (2023).