* DENSMORE, James. Data pipelines pocket reference. O'Reilly Media, 2021
* Intorf, Dylan, Dylan Storey, and Kendrick van Doorn. Apache Airflow Best Practices: A practical guide to orchestrating data workflow with Apache Airflow. 1st ed. Birmingham, UK: Packt Publishing, October 31, 2024
* Apache Airflow documentation: https://airflow.apache.org/docs/
* Dask documentation: https://docs.dask.org/en/stable/
Complex Computational Workflows
| Language of instruction | angličtina |
|---|---|
| Code | 9600-1033 |
| Abbreviation | KVW |
| Course title | Complex Computational Workflows |
| Coordinating department | IT4Innovations |
| Course coordinator | Ing. Tomáš Martinovič, Ph.D. |
Summary
This course introduces students to the design and application of complex computational workflows, considering their deployment and execution within different types of infrastructure (e.g. high-performance computing systems, data environments, AI-specific systems). The course also covers techniques for parallelism, fault tolerance and performance optimisation. The course emphasises the importance of reproducibility, data provenance and secure workflow sharing. Real-world use cases, including AI/ML, are presented to demonstrate practical implementation and scalability.
Literature
Advised literature
* HyperQueue documentation: https://it4innovations.github.io/hyperqueue/stable/
* LEXIS Platform documentation: https://docs.lexis.tech
* TERZO, Olivier; MARTINOVIČ, Jan (ed.). HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision. CRC Press, 2022
* LEXIS Platform documentation: https://docs.lexis.tech
* TERZO, Olivier; MARTINOVIČ, Jan (ed.). HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision. CRC Press, 2022