Dancing With the Pods: Live Migration of a Database Fleet While Serving Millions of Queries

This conference talk explores the complex process of live migrating a database fleet while maintaining continuous service for millions of queries. Jayme Bird and Manish Gill from ClickHouse share their six-month journey of changing how they orchestrate customer-provisioned databases, specifically their StatefulSet implementation. Discover the numerous challenges they faced while ensuring zero downtime during the migration of thousands of services from legacy to new orchestration code-paths. Learn about their custom migration controller, durable execution workflows, and solutions to database synchronization problems. Gain insights into StatefulSet operations and how the ClickHouse Operator was utilized throughout this challenging migration process that allowed queries to continue running uninterrupted.