Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Delta Live Tables: Modern Software Engineering for ETL Pipelines

Databricks via YouTube

Start learning Write review
Coursera Ad

Google Data Analytics, IBM AI & Meta Marketing — All in One Subscription

Learn More → CodeSignal Ad

Learn the Skills Netflix, Meta, and Capital One Actually Hire For

Learn More →

Overview

Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Explore a comprehensive talk on Delta Live Tables (DLT), a revolutionary ETL framework that simplifies data transformation and pipeline management. Learn how DLT incorporates modern software engineering practices to deliver reliable and trusted data pipelines at scale. Discover techniques for rapid innovation in pipeline development and maintenance, automation of administrative tasks, and improved visibility into pipeline operations. Gain insights into built-in quality controls and monitoring for accurate BI, data science, and ML. Understand how to implement simplified batch and streaming with self-optimizing and auto-scaling data pipelines. Delve into topics such as live table dependencies, pipeline boundaries, SQL expectations, Python integration, metaprogramming, event log integration, failure recovery automation, Spark Structured Streaming for ingestion, and Delta for infinite retention.

Syllabus

Intro
Life as a data professional.
What is a Live Table?
Development vs Production
Declare LIVE Dependencies
Choosing pipeline boundaries
Pitfall: hard-code sources & destinations
Ensure correctness with Expectations
Expectations using the power of SQL
Using Python
Installing libraries with pip
Metaprogramming in python
Best Practice: Integrate using the event log
DLT Automates Failure Recovery
What is SparkTM Structured Streaming?
Using Spark Structured Streaming for ingestion
Use Delta for infinite retention
Partition recomputation

Taught by

Databricks

Reviews

4.0 rating, based on 1 Class Central review

Start your review of Delta Live Tables: Modern Software Engineering for ETL Pipelines

  • Profile image for Suneel Katta
    Suneel Katta
    I recently completed the ETL course and found it very useful for understanding data engineering concepts. The course clearly explained the ETL process including data extraction, transformation, and loading with practical examples. I especially liked the hands-on exercises, which helped me improve my SQL and data handling skills. The instructor explained concepts in a simple and easy-to-understand way, which made learning smooth for beginners like me. This course has increased my confidence to work on real-time data projects and prepared me for entry-level data engineering roles.
Start learning

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free
Someone learning on their laptop while sitting on the floor.