How LinkedIn Navigates Streams Infrastructure Using Cruise Control - Adem Efe Gencer, PhD

This course presents Cruise Control for managing large-scale Apache Kafka clusters. It explains the architecture and operational challenges, then demonstrates dynamic load balancing, cluster maintenance, anomaly detection, self-healing, and real-cluster management.

Syllabus

Introduction
Kafka Concepts
Need for Management
Cruise Control
Dynamic Load Balancing
Demotion for Cluster Maintenance
Anomaly Detection
Replica Move
Cruise Control Architecture
Metrics Reporter
Cluster Model
Metric Sampler
Sample Store
Capacity Resolver
Analyzer
Proposal Generation
Execution Limits
Anomaly Detector
Goal Violation
Broker Failure Detector
Metric Anomaly Detector
Metric Anomaly Finder
Mitigation
GMX Metrics
Summary
Future work
Demo
Kafka support
Preferred leader election
Preferred leader election results
Selfhealing actions
Second Data Center
Third Message
Kafka Consumer
Kafka Audit
Kafka Mirroring
What happened
Questions
Example