Harnessing Databricks Asset Bundles - Transforming Pipeline Management at Scale at Stack Overflow
Databricks via YouTube
Power BI Fundamentals - Create visualizations and dashboards from scratch
Launch a New Career with Certificates from Google, IBM & Microsoft
Overview
Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Explore how Stack Overflow revolutionized its data engineering workflows through Databricks Asset Bundles (DABs) in this 40-minute conference talk. Learn about implementing structured pipeline architecture that emphasizes code reusability, modular design, and bundle variables to ensure clarity and data isolation across multiple projects. Discover how Stack Overflow's data team leverages enterprise infrastructure to streamline deployment processes across various environments while maintaining scalability and efficiency. Dive into key concepts including DRY-principled modular design, essential DAB features for automation, and comprehensive data security strategies using Unity Catalog. Gain actionable insights on optimizing data pipelines with Databricks' evolving toolset, presented by Chelsea Zhang, Staff Data Engineer at Stack Overflow, making this session particularly valuable for data engineers and teams managing complex multi-project workflows seeking to transform their pipeline management at scale.
Syllabus
Harnessing Databricks Asset Bundles: Transforming Pipeline Management at Scale at Stack Overflow
Taught by
Databricks