Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

DataCamp

Building Data Pipelines with Airflow

via DataCamp

Overview

Want to take your Airflow skills further?

Want to take your Airflow skills further? This course uses Airflow 3.2, so you'll learn the latest way of doing things. You'll author Dags with the TaskFlow API, schedule them based on data using Assets and the new Asset Partitions, and make them reliable with retries, callbacks, and tests. In the final chapter, you'll build an end-to-end SQL ETL pipeline on DuckDB and add data quality checks directly in Airflow, without any third-party libraries, so the data your pipeline produces stays trustworthy. By the end, you'll know how to take a pipeline from a prototype to something you can actually run in production.

Syllabus

  • Authoring Dags with TaskFlow and XCom
    • You'll start by meeting the Airflow components, writing your first Dags with the TaskFlow API, and passing data between tasks with XCom.
  • Dynamic and Data-Aware Pipelines
    • From there, you'll run tasks in parallel with dynamic task mapping, schedule Dags by data with Assets, and add human approval steps.
  • Preparing Dags for Production
    • In this chapter, you'll handle failures with retries and callbacks, save resources with deferrable sensors, and test your Dags at three levels.
  • Building a Production SQL ETL Pipeline
    • In this final chapter, you'll build a SQL ETL pipeline on DuckDB, add partition-aware scheduling with Asset Partitions, and embed data quality checks.

Taught by

Volker Janz

Reviews

4.7 rating at DataCamp based on 15 ratings

Start your review of Building Data Pipelines with Airflow

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.