GPU Accelerated Spark Connect - End-to-End GPU Acceleration for ETL and ML Applications

Learn how to leverage GPU acceleration with Spark Connect in this 38-minute conference talk from Databricks. Explore the evolution of Spark Connect from its initial SQL/DataFrame API introduction in Apache Spark 3.4 to its recent MLlib extension in version 4.0, which enables running Spark applications over a gRPC protocol. Discover the key benefits of this architecture including simplified adoption for non-JVM clients, version independence between applications and clusters, and enhanced stability and security. Examine the new plugin interface introduced with the Spark Connect ML extension that allows configuration of enhanced server-side MLlib algorithm implementations. See practical demonstrations of how this interface, combined with Spark SQL's existing plugin capabilities, integrates with NVIDIA GPU-accelerated plugins for both ML and SQL operations. Understand how to achieve no-code change, end-to-end GPU acceleration of Spark ETL and ML applications over Spark Connect, with performance improvements up to 9x faster while reducing costs by 80% compared to CPU baselines. Gain insights from NVIDIA experts Erik Ordentlich and Gera Shegalov on implementing production-ready GPU-accelerated data pipelines for analytics and AI workloads.