Managing Kubernetes Watch Events End-to-End Flow in Extremely Large Clusters

Explore the critical yet often overlooked Kubernetes watching mechanism and its impact on cluster stability and performance in this conference talk. Dive into the measurement of watch events latency, definition of watch SLI and SLO metrics, and the process of identifying bottlenecks in watching. Learn about optimizations made to various components including apiserver, etcd, kubelet, controller-runtime, and clients such as controllers and schedulers. Discover how these improvements have significantly enhanced watch latency, pod provisioning time, bandwidth usage, and overall cluster stability in large-scale environments with approximately 20,000 nodes. Gain insights into the importance of watch latency as an indicator of cluster health and how addressing this aspect can lead to substantial performance gains in Kubernetes deployments.