Five Great Ways to Lose Data on Kubernetes - And How to Avoid Them

Explore five common ways to lose data on Kubernetes and learn effective strategies to prevent data loss in this informative conference talk. Delve into the challenges of running databases and stateful applications on Kubernetes, drawing from the speaker's extensive experience as a database engineer and developer of the ClickHouse Kubernetes operator. Examine traditional high availability and disaster recovery models, and discover how they apply to Kubernetes environments. Investigate Kubernetes-specific pitfalls, including persistent volume issues, affinity problems, and human error. Gain valuable insights on implementing robust data protection measures, such as proper replication strategies, affinity and anti-affinity configurations, and blast radius management. Learn the importance of thorough testing, resource definition checks, and automation in maintaining data integrity. Develop a proactive approach to data management in cloud-native environments and acquire practical tools to ensure successful experiences with Kubernetes-based data systems.

Syllabus

Intro
Presenter and Company Bio
What do we mean by "data"?
Starting a database is easy in Kubernetes (Install helm)
A delicate flower is our database
Traditional database solution: replicas
Simplest K8s path to replicas: use an operator
Complex systems made simple[r]
What is "blast radius?"
Affinity vs. anti-affinity
Pod anti-affinity covers host failures
Node affinity + failure domain covers AZ failure
Protect replicas using distance
Blast protection across regions or K8s
Where are my pods running?
Ephemeral storage is a feature, not a bug!
PVs are not enough if you don't use them!
Testing to the point of abuse is the solution Check Kubernetes resource definitions Inspect file system mounts Kill pods Kill nodes
The best way to lose data: do it yourself
Steps to reclaim -- Automation anyone?