From Imperative to Declarative Paradigm - Rebuilding a CI/CD Infrastructure Using Hatch and DABs

Learn how to transform a complex, imperative CI/CD infrastructure into a streamlined, declarative system for PySpark pipeline deployment on Databricks. Discover the challenges faced by FreeWheel's engineering team with their legacy infrastructure that required custom scripts for each project, resulting in redundant boilerplate code and difficult maintenance. Explore the complete rebuild process that eliminated thousands of lines of code by adopting a declarative paradigm using Hatch and Databricks Asset Bundles (DABs) as primary tools. Understand how this transformation created a fully configuration-driven infrastructure that simplifies project onboarding and improves codebase quality. See practical demonstrations of how DABs streamline deployment across multiple environments and enable rapid adoption of new Databricks features. Gain insights from Luigi Di Tacchio, Senior Software Engineer, and Saswati Bhoi, Senior Site Reliability Engineer, both from FreeWheel, a Comcast Company, as they share their real-world experience rebuilding production CI/CD systems for data engineering workflows.