* DENSMORE, James. Data pipelines pocket reference. O'Reilly Media, 2021
* Intorf, Dylan, Dylan Storey, and Kendrick van Doorn. Apache Airflow Best Practices: A practical guide to orchestrating data workflow with Apache Airflow. 1st ed. Birmingham, UK: Packt Publishing, October 31, 2024
* Apache Airflow documentation: https://airflow.apache.org/docs/
* Dask documentation: https://docs.dask.org/en/stable/
Complex Computational Workflows
| Type of study | Follow-up Master |
|---|---|
| Language of instruction | English |
| Code | 9600-1033/01 |
| Abbreviation | KVW |
| Course title | Complex Computational Workflows |
| Credits | 4 |
| Coordinating department | IT4Innovations |
| Course coordinator | Ing. Tomáš Martinovič, Ph.D. |
Subject syllabus
1. Overview of Scientific Workflow Systems (e.g. Nextflow, Apache Airflow, HyperQueue, Dask)
2. Directed Acyclic Graphs (DAGs) and Dependency Modeling and communication
3. Job Scheduling and Resource Management (e.g. SLURM)
4. Parallelism in Workflow Execution
5. Fault Tolerance and Recovery Mechanisms
6. Workflow Profiling, Bottleneck Analysis, and Load Balancing
7. Scientific Use Cases including AI/ML Workflows in HPC
8. Security and Access Control in HPC Workflows
9. Workflow Sharing, Reusability, and Publication
10. Workflows as a Service and interoperability
2. Directed Acyclic Graphs (DAGs) and Dependency Modeling and communication
3. Job Scheduling and Resource Management (e.g. SLURM)
4. Parallelism in Workflow Execution
5. Fault Tolerance and Recovery Mechanisms
6. Workflow Profiling, Bottleneck Analysis, and Load Balancing
7. Scientific Use Cases including AI/ML Workflows in HPC
8. Security and Access Control in HPC Workflows
9. Workflow Sharing, Reusability, and Publication
10. Workflows as a Service and interoperability
Literature
Advised literature
* HyperQueue documentation: https://it4innovations.github.io/hyperqueue/stable/
* LEXIS Platform documentation: https://docs.lexis.tech
* TERZO, Olivier; MARTINOVIČ, Jan (ed.). HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision. CRC Press, 2022
* LEXIS Platform documentation: https://docs.lexis.tech
* TERZO, Olivier; MARTINOVIČ, Jan (ed.). HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision. CRC Press, 2022