Coursera Spring Sale
40% Off Coursera Plus Annual!
Grab it
Learn how to scale telemetry systems for massive data volumes in this conference talk from NDC Melbourne 2025. Discover the evolution of Honeycomb's platform engineering approach to handling a 10x increase in incoming telemetry volume over four years using streaming and serverless technologies. Explore the architectural decisions behind scaling Kafka clusters, including broker and server-side optimizations for instance types, sizing, and network traffic management. Master chaos engineering techniques to ensure system resilience during instance failures and understand effective patterns for data distribution across partitions. Examine consumer scheduling strategies and methods for observing the impact of cluster configuration changes. Dive into post-Kafka consumption scaling techniques using tens of thousands of map workers running on AWS Lambda and S3 for efficient read operations. Gain practical insights into building robust, large-scale data analysis systems that provide real-time insights into telemetry data through streaming and map-reduce architectures.
