How We Scaled Metrics Monitoring for 10,000 Applications With VictoriaMetrics

Discover how Bilibili scaled metrics monitoring for over 10,000 applications serving 104 million+ daily active users in this conference talk from the Linux Foundation. Learn about the journey from Prometheus to VictoriaMetrics for monitoring one of Asia's largest video-sharing platforms, operating across 2 data centers with 1 million Kubernetes pods. Explore the technical challenges of hyper-scale operations and the strategic decision to separate storage and computation for optimal resource efficiency. Understand the implementation of advanced capabilities including data pre-aggregation with Flink, PromQL query rewriting for performance optimization, and automated replacement of performance-impacting subqueries with pre-aggregated alternatives. Gain insights into achieving 50 million samples per second ingestion with minimal resources while reducing anomalies like missing samples and out-of-memory issues by 90% and accelerating query speeds by 10x. See how the migration was made transparent to engineers who continue working as if using a single Prometheus instance, demonstrating effective large-scale infrastructure transformation without disrupting developer workflows.