Spark on Kubernetes - A Practical Guide

Learn how to implement Apache Spark on Kubernetes through a practical conference talk that addresses the key decisions and challenges faced when transitioning from Spark on YARN to a Kubernetes-based deployment. Explore the critical architectural choices including job submission methods with two main operators, shuffle service selection options, scheduler and queue management systems, storage optimization strategies, and logging configurations. Discover real-world benchmarking approaches using TPC-DS and simulated workloads to evaluate Spark on Kubernetes performance against existing systems. Gain insights from Airbnb's production experience and learn practical solutions to common implementation challenges through field-tested strategies and hard-earned lessons from actual deployment scenarios.