Realtime Change Data Capture Streaming - End-to-End Data Engineering Project

Dive deep into the world of Change Data Capture (CDC) and learn how to implement real-time data streaming using a powerful tech stack in this comprehensive video tutorial. Explore the integration of Docker, Postgres, Debezium, Kafka, Apache Spark, and Slack to create an efficient and responsive data pipeline. Follow along as the instructor guides you through the system architecture, setting up live data in a Postgres database, connecting to Postgres with Debezium and Kafka, previewing data on Kafka, and handling various aspects of data capture. Gain practical knowledge on setting up a Debezium connector, managing decimal values, tracking user changes with timestamps, and creating a robust data capture system in Postgres. By the end of this tutorial, you'll have a solid understanding of implementing an end-to-end data engineering project for real-time change data capture streaming.

Syllabus

Introduction
The system architecture
Getting live data into postgres db
Connecting to Postgres with Debezium and Kafka from the UI
Previewing Debezium data on Kafka
Getting full data from Postgres with Debezium
Setting up debezium connector from the terminal
Handling decimal values on debezium
Getting the user that changed data on postgres with time
Creating a more robust data capture on postgres
Outro