Optimizing Iceberg Table Layouts at Scale - A Multi-Objective Approach

This conference talk from Data Council features Sumedh Sakdeo and Jesus Camacho Rodríguez sharing their expertise on managing data layout for over 100,000 Iceberg tables. Discover how automated reorganization can be approached as a multi-objective optimization problem, balancing competing priorities in large-scale data environments. Learn practical strategies for improving storage efficiency, minimizing operational costs, and maximizing performance impact through intelligent table scoring and ranking systems. Gain valuable insights into establishing effective reorganization budgets and prioritization frameworks that can be applied to scale table management or enhance query performance in your own data infrastructure. The 34-minute presentation offers concrete approaches from experienced engineers at LinkedIn and Microsoft Grey Systems for tackling the challenges of Iceberg table optimization at enterprise scale.