Robinhood's Kafkaproxy - Decoupling Kafka Consumer Logic from Application Business Logic

Explore Robinhood's innovative approach to managing Apache Kafka consumers through their Consumer Proxy in this 34-minute conference talk. Discover how Robinhood decouples Kafka consumer logic from application business logic, addressing challenges faced by infrastructure engineers in managing diverse requirements across multiple microservices. Learn about the benefits of the Consumer Proxy, including stateless processing, commit and timeout management, and easy rebalancing of consumer groups during scaling events. Gain insights into how this sidecar container solution allows for efficient resource utilization, failure isolation, and simplified management of Kafka clients across the organization. Understand the advantages of centralizing Kafka consumption logic for infrastructure teams and how this approach can be applied to other organizations looking to streamline their Kafka client management.