Efficient Kafka Topic Compaction at Scale on Ursa

Learn advanced topic compaction techniques for large-scale data streaming environments through this 25-minute conference talk that explores Ursa's innovative approach to managing stateful applications while controlling storage costs. Discover how this leaderless, S3-based Kafka-compatible server implements sophisticated compaction strategies that leverage S3 object storage for durability and cost efficiency while maintaining essential Kafka guarantees. Explore the architecture and design principles behind Kafka topic compaction on Ursa, including techniques that intelligently balance storage costs and compaction efficiency by minimizing S3 requests through strategic minor and major compaction approaches. Gain practical insights into building scalable, cost-effective topic compaction solutions in cloud-native architectures, understanding how Ursa handles high volumes of data and numerous keys while ensuring the latest value per key is retained and never removed until manually deleted. Master the implementation of compaction services and indexing systems that enable robust performance in distributed streaming environments, with real-world examples of how this approach addresses the constant challenge of maintaining stateful applications efficiently in large-scale data streaming scenarios.