Beyond the Ephemeral: Mastering Serverless Metrics at Scale With Shopify

This conference talk explores a cutting-edge approach to enhancing observability for serverless applications on solutions like Google Cloud Run and Cloud Functions. Dive into Shopify's method for creating a scalable metrics pipeline using their internal app platform for seamless container configuration and an ingestion system capable of handling millions of datapoints per minute. Learn about the architecture featuring OpenTelemetry collectors as sidecars and regional workloads to ingest and manage metrics with varying temporality models. Discover how Shopify integrated OTLP with their ingestion layer, transforming exponential histograms into DD Sketches for optimal performance and accuracy. Gain insights into the challenges and solutions in building a comprehensive observability pipeline. Take away valuable lessons for enhancing serverless monitoring in Kubernetes environments, leveraging Shopify's philosophy of efficient, resilient, and cost-effective cloud utilization.