Learning Apache Airflow (2023)

Introduction

• An overview of Apache Airflow

1. Introducing Apache Airflow

• Apache Airflow concepts and architecture
• Initial setup: macOS and Windows
• Understanding Airflow configuration
• Running Airflow

2. Running Workflows on Airflow

• Navigating and exploring the Airflow UI
• Creating your first DAG
• Running your first DAG
• Creating a DAG using the with keyword

3. Defining Tasks and Dependencies

• Specifying task dependencies
• Using the Bitshift operators for dependency specification
• Using bash scripts in the DAG

4. Using the Python Operator and XCom

• Introducing the Python operator
• Python operators and dependencies
• Passing parameters to Python callables
• Introducing XCom
• Using XCom to pass values
• Implementing a data transformation DAG pipeline, part 1
• Implementing a data transformation DAG pipeline, part 2

5. Using the SQLite Operator for SQL Operations

• Setting up the SQLite DB and connection
• Introducing the SQLite operator
• Implementing a SQL-based DAG pipeline, part 1
• Implementing a SQL-based DAG pipeline, part 2

6. Conditional Branching and TaskGroups

• Introducing branching
• Conditional branching with variables
• Task groups and edge labels

7. Catch Up, Backfill, and CRON Expressions

• Catch up with all previously scheduled runs
• Backfill scheduled runs

Conclusion

• Summary and next steps