Coursera Spring Sale
40% Off Coursera Plus Annual!
Grab it
Learn how to build large-scale streaming ingestion pipelines using Apache Flink and Apache Hudi for real-time data processing in this 39-minute conference talk from StreamNative's Data Streaming Summit. Discover Uber's production-proven architecture for managing thousands of Flink ingestion pipelines that power real-time machine learning and analytics across pricing, matching, logistics, and safety systems in a hybrid cloud environment. Explore the comprehensive system design including built-in deployment safety mechanisms, failover protocols, and disaster recovery strategies that ensure reliable data flow at scale. Examine runtime considerations for cross-cloud data security, implementing column-level access control, and maintaining data privacy across distributed systems. Understand operational efficiency techniques including Flink Autoscaler as a Service for dynamic workload parallelism adjustment to optimize costs, and partial sort optimization strategies for enhanced performance. Gain deep insights into Apache Hudi's streaming integration capabilities with Flink, focusing on the non-blocking concurrency control (NBCC) design that enables seamless real-time data ingestion. Access practical architectural patterns and implementation strategies that have been battle-tested in Uber's production environment, making this essential viewing for engineers working with large-scale streaming systems or exploring real-time ingestion solutions for machine learning and analytics workloads.
