High-Volume Logging Without High Cost - Flight Recorder for OpenTelemetry Logs

Learn about the Flight Recorder pattern for OpenTelemetry Logs, a cost-effective solution to high-volume application logging challenges presented by Cijo Thomas from Microsoft at a CNCF conference. Discover how organizations can reduce escalating logging costs while maintaining observability by implementing ring buffers that store logs locally instead of continuously exporting them via OTLP. Explore the technical design where logs are kept in local buffers—backed by OS-native mechanisms like Windows ETW or Linux Tracepoints, or implemented in user space—and are only exported to OTel Collector when triggered by critical errors or operator requests. Understand how the system enters temporary streaming mode after triggers, exporting all new logs continuously for brief periods to maintain existing workflows. Examine a working prototype demonstration that shows substantial cost savings without sacrificing observability, addressing the common problem where teams aggressively filter logs to control costs, potentially creating incident blind spots.