Iceberg Table Format Adoption and Unified Metadata Catalog Implementation in Lakehouse Platform
Databricks via YouTube
Learn Generative AI, Prompt Engineering, and LLMs for Free
AI Engineer - Learn how to integrate AI into software applications
Overview
AI, Data Science & Cloud Certificates from Google, IBM & Meta — 40% Off
One plan covers every Professional Certificate on Coursera. 40% off Coursera Plus Annual.
Unlock All Certificates
Learn how DoorDash's Data organization successfully migrated from classic Data Warehouse and Data Lake platforms to a unified LakeHouse solution in this 40-minute conference talk. Discover the comprehensive methodology that eliminates excessive data movement while seamlessly integrating and consolidating query engine layers across Snowflake, Databricks, EMR, and Trino. Explore strategic approaches to query performance optimization and understand how to abstract away the complexity of underlying storage layers and table formats. Gain insights into the decision-making process for selecting a unified metadata catalog that works effectively across various compute platforms. The presentation covers practical implementation strategies for adopting the Iceberg table format and establishing a cohesive metadata management system that supports the LakeHouse paradigm. Presented by Ruotian Wang, Software Engineer at DoorDash, and Sergey Zavgorodni, Lead Data Engineer at DoorDash, this talk provides real-world experience and lessons learned from a large-scale platform migration at one of the leading food delivery companies.
Syllabus
Iceberg Table Format Adoption and Unified Metadata Catalog Implementation in Lakehouse Platform
Taught by
Databricks