Reverse-Engineering PromQL Usage - A Proxy's Tale

Learn how to gain visibility into Prometheus query usage patterns through the development of a custom analytics proxy in this conference talk from KubeCon + CloudNativeCon. Discover the challenges of understanding PromQL utilization in production environments where traditional monitoring falls short of revealing which queries are slow, who executes them, and which dashboards create excessive load on backend systems. Explore the technical implementation of Prom Analytics Proxy, a solution designed to monitor Prometheus usage at scale without requiring modifications to existing Prometheus servers. Examine the architecture decisions behind creating a low-overhead query proxy that captures essential metrics including latency, cardinality, and error rates while organizing data for efficient querying. Understand the practical strategies that proved successful, learn from implementation challenges and unexpected discoveries, and see how this approach enables teams to better comprehend and optimize their Prometheus deployments in production environments.