Apache Spark on Kubernetes - Lessons Learned from Launching Millions of Spark Executors

Explore lessons learned from launching millions of Spark executors on Kubernetes in this 35-minute conference talk by Databricks. Dive into Apple's approach to supporting enormous Spark workloads for cloud services, covering orchestration techniques across Mesos and Kubernetes, private and on-premise infrastructure. Learn about effective monitoring systems, resource requirement tuning, and execution analysis. Gain insights on optimizing Kubernetes for Spark workloads, implementing granular concurrency checks, mitigating cluster storage stress, and utilizing dynamic allocation. Discover strategies for push-button cloud management and scaling up Spark on Kubernetes to support varying workload patterns across multi-cloud environments.