Efficient Incremental Processing with Apache Iceberg at Netflix

Discover in this AWS re:Invent 2024 conference talk how Netflix implements efficient incremental processing using Apache Iceberg and Netflix Maestro. Explore how the combination of Iceberg's metadata capabilities and Maestro's workflow orchestration creates a powerful Incremental Processing Solution (IPS) that significantly reduces compute costs and execution time. Dive into the technical challenges of implementing incremental processing at scale, including handling late-arriving data and managing backfills beyond append-only scenarios. Learn about the architectural decisions and solutions that enable Netflix to process only new or changed data while maintaining data accuracy and freshness. Gain insights into how this innovative approach minimizes manual intervention, reduces system failures, and optimizes large-scale data workflows for better performance and cost efficiency.