Scaling Validation and Quality of Streaming Data Products at Twitter

Explore the intricacies of scaling validation and quality assurance for streaming data products at Twitter in this 33-minute conference talk from Strange Loop. Dive into the challenges faced by the Data Products Quality Engineering team as they build and maintain services, tools, and processes to support feature teams in providing a robust distributed streaming architecture. Learn about the evolution from simple QA to sophisticated tools for identifying issues in pre-production and live microservices. Discover the team's specialized approaches, including end-to-end stress testing, measuring pipeline data loss against product SLAs, and per-message data quality assessment. Gain insights into the value these tools and processes bring to engineers and Twitter's data customers. Follow along as the speaker covers topics such as staging environments, operating constraints, testing methodologies, and the development of better testing tools to meet the growing demands of Twitter's data business.