DataFusion and Apache Arrow: Supercharging Data Analytics with a Rust-Based Query Engine

Discover how Rust, Apache Arrow, and the Data Fusion Query Engine are revolutionizing modern data stacks in this 29-minute video from Databricks. Explore the implementation timeline for new database systems, LLVM-like infrastructure for databases, and DataFusion's growth and milestones. Learn about DataFusion's features, including SQL support and extensibility, and examine real-world applications like VegaFusion and Cube.js. Gain insights into future directions for distributed computing with Ballista and understand how these technologies can supercharge your data analytical tools.

Syllabus

Introduction
What is going on?
Apache Arrow
Implementation timeline for a new Database system
LLVM Compiler Infrastructure
LLVM-like Infrastructure for Databases
DataFusion Project Growth
DataFusion Milestones: Time to Mature
Initial Logical Plan
Let's Optimize!
The Initial Execution Plan
DataFusion Features
SQL Support
Extensibility
VegaFusion
Cube.js / Cube Store
Coralogix
blaze-rs
Ballista Distributed Compute
Future Directions