Koalas: Scaling Pandas APIs on Apache Spark - Performance and Comparison with Dask
Databricks via YouTube
MIT Sloan: Lead AI Adoption Across Your Organization — Not Just Pilot It
Live Online Classes in Design, Coding & AI — Small Classes, Free Retakes
Overview
Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Explore the capabilities and performance of Koalas, an open-source project providing pandas APIs on top of Apache Spark, in this 24-minute talk from Databricks. Learn how Koalas bridges the gap between pandas' data science functionality and Apache Spark's scalability for big data. Compare Koalas with other pandas-scaling libraries, particularly Dask, through benchmarking and performance analysis. Discover the internal framework, execution time improvements, influence of Catalyst, and code generation techniques. Gain insights into recent updates and main changes in Koalas, equipping you with knowledge to effectively handle large-scale data manipulation and analysis.
Syllabus
Introduction
What is Koalas
Internal Frame
Benchmark
Results
Execution Time
Influence of Catalyst
Code Generation
Benchmark Results
Whats New
Main Changes
Taught by
Databricks