Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Learn to build self-adapting data ingestion pipelines in this 30-minute conference talk that demonstrates how to seamlessly ingest thousands of evolving Apache Kafka topics into a lakehouse architecture using the Apache Flink Dynamic Iceberg Sink. Explore how this innovative connector enables dynamic writing of streaming data into multiple Apache Iceberg tables with full schema evolution support, automatically creating and writing to new tables based on record instructions while adapting to schema changes and partitioning scheme modifications in real-time without requiring job restarts. Discover practical implementation strategies for ingesting Kafka data with dynamic Avro schemas sourced from Apache Schema Registry, and gain a comprehensive blueprint for constructing scalable, self-adapting ingestion layers that eliminate operational overhead while effectively bridging streaming data sources with lakehouse storage systems.
