Iceberg Table Format Adoption and Unified Metadata Catalog Implementation in Lakehouse Platform

Learn how DoorDash's Data organization successfully migrated from classic Data Warehouse and Data Lake platforms to a unified LakeHouse solution in this 40-minute conference talk. Discover the comprehensive methodology that eliminates excessive data movement while seamlessly integrating and consolidating query engine layers across Snowflake, Databricks, EMR, and Trino. Explore strategic approaches to query performance optimization and understand how to abstract away the complexity of underlying storage layers and table formats. Gain insights into the decision-making process for selecting a unified metadata catalog that works effectively across various compute platforms. The presentation covers practical implementation strategies for adopting the Iceberg table format and establishing a cohesive metadata management system that supports the LakeHouse paradigm. Presented by Ruotian Wang, Software Engineer at DoorDash, and Sergey Zavgorodni, Lead Data Engineer at DoorDash, this talk provides real-world experience and lessons learned from a large-scale platform migration at one of the leading food delivery companies.