Resilient Kafka - DNS Traffic Management and Client Wrappers for Ensuring Availability

Explore how to enhance Kafka client resilience through DNS traffic management and client wrappers in this 28-minute talk. Dive into Shopify's Kafka infrastructure and client connection methods. Learn about implementing failovers using DNS and discover real-world scenarios where this system prevented major outages. Gain insights into making Kafka clusters more resilient to infrastructure issues by combining DNS traffic management with opinionated client wrappers. Understand the importance of these techniques in ensuring message production continuity during incidents or maintenance, ultimately improving system availability and reducing the need for manual interventions.