Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events

Discover how to build and manage large-scale Spark clusters on Kubernetes for powerful data processing in this informative conference talk. Learn best practices for constructing scalable Spark clusters on Kubernetes, with a focus on leveraging Argo Workflows and Argo Events. Explore the challenges of configuring storage, compute, networking, and optimizing job scheduling, whether starting from scratch or migrating Spark workloads from existing Hadoop clusters. Gain insights into harnessing the potential of Argo Workflows and Argo Events for event-driven job scheduling, enabling efficient resource utilization and seamless scalability. Understand how integrating these powerful open-source tools can provide better control and flexibility for executing Spark jobs on Kubernetes, empowering you to build highly scalable and efficient data processing environments.