Gazelle-JNI: Offloading Spark SQL to Native Engines for Execution Acceleration

Explore Gazelle-Jni, a middle layer designed to offload Spark SQL to native engines for execution acceleration, in this 41-minute conference talk from Databricks. Learn how Gazelle-Jni implements a shared JVM and JNI middle layer to better integrate various native SQL engines as Spark SQL's backend. Discover the process of passing Substrait transformed physical plans to native engines for improved performance. Gain insights into integrating native engines with Spark SQL through practical examples. Delve into topics such as basic operator performance, CPU bottlenecks, Spark's scalability, SQL engine development, and the evolution of data processing frameworks. Examine Gluten's layout, components, plan conversion, buffer passing and sharing, fallback processing, shuffle mechanism, and memory management. Compare performance metrics for Gluten with Velox and ClickHouse. Explore future steps, learn how to take Gluten for a spin, and understand the call to action for leveraging this technology in your data processing workflows.