Building Composable OTel Pipelines - CI/CD, Testing, Team-First, and Scalable Design

Learn how to build scalable, vendor-agnostic OpenTelemetry (OTel) pipelines through this conference talk by DoorDash engineers who share their experience processing 100TB of log data daily and over 15 million spans per second. Discover how the DoorDash observability team developed 28 in-house OTel components to power their logs, traces, and metrics pipelines at massive scale. Explore strategies for leveraging your organization's Git repository structure, CI/CD pipelines, and testing frameworks while adopting industry-standard engineering best practices. Understand how to promote collaborative contributions from observability team members to a unified codebase and implement composable pipeline architectures that incorporate security tooling and cost-efficient designs. Gain insights into running multiple pipeline configurations and applying continuous learnings to improve overall performance, enabling you to build maintainable OTel pipelines that scale with your organization's growing observability needs.