Challenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters to Kubernetes

This conference talk explores the challenges and solutions involved in migrating Apache Spark workloads from traditional bare metal Hadoop clusters to Kubernetes-based cloud environments. Discover how Apple's engineering team navigated technical hurdles including query performance parity, scheduling differences, resource management, multi-tenant security concerns, and diagnostic capabilities during their large-scale migration. Learn practical approaches for overcoming both technical and organizational obstacles when transitioning Spark workloads to cloud-native infrastructure, gaining insights into how Kubernetes offers superior scalability, flexibility, and cost-effectiveness for variable workloads compared to legacy Hadoop deployments. The presenters, Neha Singla and Rasik Pandey, share their real-world case study of implementing solutions in a large corporate environment, providing valuable lessons for organizations considering similar migrations.