Kernel, Catalog, Action! Reimagining Delta-Spark Connector with DSv2

Explore how Delta Lake is revolutionizing its Spark connector architecture through three transformative technologies in this 20-minute conference talk. Discover the strategic shift to DSv2 APIs for enhanced catalog integration and improved Spark optimizer performance, while learning about the full integration with Delta Kernel that simplifies Delta protocol complexities and accelerates feature adoption. Understand how Delta is evolving into a catalog-aware lakehouse format through Catalog Commits, enabling more efficient metadata management, enhanced governance, and superior query performance. Gain insights into the advanced architectural changes pushing the boundaries of metadata management and creating a more intelligent, high-performance data lakehouse platform. Presented by Scott Sandre, Senior Software Engineer at Databricks, this technical session provides deep insights into the future of Delta Lake's connector technology and its impact on modern data architecture.