Support Tiered Storage in Strimzi-Operated Kafka

Explore how Apple engineers integrated tiered storage in Strimzi-operated Kafka to create a cost-effective solution for real-time streaming applications. Learn about the implementation of KIP-405's tiered storage feature and a custom remote plugin that enables data storage in cheaper storage options, extending retention periods and facilitating efficient backfilling of past data. Discover the process of integrating tiered storage with Strimzi Kafka operator and operated clusters, and gain insights into the proposed native support for tiered storage in Strimzi. Delve into the journey of optimizing performance for remote storage manager implementation and the valuable lessons learned along the way. This 30-minute conference talk, presented by Apple's Staff Software Engineer Lixin Yao and Software Engineers Vrishali Bhor and Bo Gao, offers a comprehensive look at how to leverage Strimzi-operated Kafka for building end-to-end streaming applications while significantly reducing storage costs.