Overview

Google, IBM & Meta Certificates – 40% Off

One plan covers every Professional Certificate on Coursera.

This course expands your data engineering skills by focusing on Delta Lake and Medallion Architecture for building dependable, scalable, and analytics-ready data platforms. You will learn how modern pipelines maintain data quality, support version control, and prepare trusted datasets for AI/ML workloads. You will start with Delta Lake fundamentals, including ACID transactions, schema enforcement, schema evolution, transaction logs, and Time Travel. Through practical demonstrations, you will create Delta tables, manage updates and deletes, apply MERGE operations, and restore earlier versions of data when needed. You will then apply PySpark transformation techniques to clean, reshape, join, and validate datasets. You will also explore performance-focused practices such as OPTIMIZE, VACUUM, and Z-Ordering to make Delta tables more efficient for large-scale processing. Next, you will design Medallion Architecture pipelines using Bronze, Silver, and Gold layers. This helps convert raw data into clean, validated, and business-ready datasets for reporting, analytics, and machine learning. The course also introduces structured streaming, Change Data Feed, and Delta constraints for improving pipeline reliability. By the end of this course, you will be able to: - Use Delta Lake for reliable and versioned data management. - Transform and validate datasets using PySpark. - Optimise Delta tables for performance and storage efficiency. - Build Bronze, Silver, and Gold data layers. - Develop dependable batch and streaming pipelines for AI/ML use cases. Designed for data engineers, analytics engineers, software developers, and AI/ML professionals, this course prepares you to build modern data pipelines that are reliable, scalable, and ready for production workloads.

Syllabus

Delta Lake Fundamentals

Master the fundamentals of Delta Lake and learn how it enables reliable, scalable, and high-performance data engineering on Databricks. This module covers ACID transactions, schema enforcement and evolution, transaction logs, and versioned data management. Through hands-on exercises, you'll create and manage Delta tables, perform CRUD and MERGE operations, and use Time Travel to track and recover data changes, building the skills needed to develop robust and trustworthy data pipelines.

Data Transformation with Pyspark

Develop practical PySpark skills to transform, clean, and optimize data for analytics and AI/ML workloads. This module covers essential data transformation techniques, including filtering, aggregations, joins, duplicate handling, null value management, and working with complex data types. Through hands-on exercises, you'll also learn to optimize Delta tables using OPTIMIZE, VACUUM, and Z-Ordering, enabling the creation of efficient, high-quality, and performance-optimized data pipelines.

Medallion Architecture for AI/ML

Understand how Medallion Architecture structures data into progressive layers for building scalable and reliable analytics and AI/ML pipelines. This module explains the role of Bronze, Silver, and Gold layers, how data is refined at each stage, and how modern workflows are designed for quality and performance. Through hands-on activities, you will ingest raw data, clean and validate it, and prepare business-ready datasets for AI and machine learning use cases.

Streaming and Pipeline reliability

Learn the fundamentals of streaming data processing and building reliable data pipelines with Delta Lake. This module introduces Structured Streaming, incremental data processing using Delta Change Data Feed (CDF), and techniques for enforcing data quality with Delta constraints. Through hands-on exercises, you'll build resilient, scalable pipelines capable of processing continuously changing data while maintaining data integrity and reliability.