Lakeflow in Production - CI/CD, Testing and Monitoring at Scale

Learn how to build production-grade data pipelines using Lakeflow and Databricks Asset Bundles (DABs) in this 37-minute conference talk. Discover how to implement Git-based workflows, automated deployments, and comprehensive testing strategies that go beyond basic transformation logic. Explore rigorous testing methodologies including unit testing approaches, automated CI/CD workflows, and data quality monitoring techniques. Master environment-specific configurations and version control practices that ensure clear separation between development and production environments. Gain insights into observability techniques and performance tuning strategies to make your data pipelines scalable, maintainable, and production-ready. The presentation covers best practices for data engineering projects at scale, demonstrating practical implementation of robust testing frameworks and monitoring solutions for enterprise-level data infrastructure.