Databricks Courses

Understanding and Improving Code Generation in Spark

Explore Spark's code generation techniques, challenges with large queries, and Workday's improvements for handling whole-stage codegen, enhancing performance in production workloads.

• YouTube
• 24 minutes
• Self-Paced
• Free Video

Presto on Apache Spark - A Tale of Two Computation Engines

Explore the integration of Presto and Spark, combining low-latency evaluation with robust execution for enhanced SQL experience in both interactive and batch use cases at Facebook scale.

• YouTube
• 25 minutes
• Self-Paced
• Free Video

From Idea to Model: Productionizing Data Pipelines with Apache Airflow

Learn to productionize data pipelines using Apache Airflow, bridging data science and engineering. Explore collaborative workflows, custom operators, and scalable solutions for efficient model deployment and iteration.

• YouTube
• 22 minutes
• Self-Paced
• Free Video

DataSource V2 and Cassandra - A New Era of Speed and Flexibility

Explore DataSource V2 in Spark Cassandra Connector: enhanced speed, flexibility, and usability. Learn about Spark's understanding of Cassandra clustering and direct catalogue manipulation for improved data integration and analysis.

• YouTube
• 39 minutes
• Self-Paced
• Free Video

Designing the Next Generation of Data Pipelines with Apache Spark - Zillow's Approach

Zillow's data engineering team shares their journey in redesigning data pipelines using Apache Spark, focusing on scalability, maintainability, and robustness while reducing code complexity.

• YouTube
• 27 minutes
• Self-Paced
• Free Video

Apache Pulsar - Next Generation Messaging and Queuing System

Overview of Apache Pulsar's architecture, features, and benefits as a scalable messaging system with unified storage, processing, and serverless capabilities for building end-to-end streaming applications.

• YouTube
• 38 minutes
• Self-Paced
• Free Video

Simplify and Boost Spark 3 Deployments with Hypervisor-Native Kubernetes

Explore integrating Spark 3 with VMware Pacific for scalable, reliable, and high-performance big data analytics deployments using hypervisor-native Kubernetes on vSphere 7.

• YouTube
• 33 minutes
• Self-Paced
• Free Video

Scaling Up AI Research to Production with PyTorch and MLFlow

Explore PyTorch's latest advancements in AI research and production, including distributed training, model optimization, and deployment using MLFlow, with insights on scaling and efficiency.

• YouTube
• 44 minutes
• Self-Paced
• Free Video

Model Explanation and Prediction Exploration Using Spark ML

Explore techniques for explaining non-linear models, showing feature contributions, and performing what-if analysis using Spark ML. Implement these methods to enhance model interpretability and enable interactive prediction exploration.

• YouTube
• 23 minutes
• Self-Paced
• Free Video

Migrating Pinterest Apache Spark Clusters from HDFS to S3

Explore Pinterest's migration of Apache Spark clusters from HDFS to S3, addressing technical challenges, optimizing performance, and achieving a smooth transition process.

• YouTube
• 30 minutes
• Self-Paced
• Free Video

Deploying Apache Spark Jobs on Kubernetes with Helm and Spark Operator

Learn to deploy Scala Spark jobs on Kubernetes using Helm and Spark Operator. Live coding demo shows scalable, flexible deployments with minimal custom configurations for stress-free implementation.

• YouTube
• 31 minutes
• Self-Paced
• Free Video

User Defined Aggregation in Apache Spark - From Challenges to Improvements

Explore Apache Spark's User Defined Aggregate Functions (UDAFs), their evolution, and improvements in Spark 3.0. Learn to create custom aggregations, enhance performance, and contribute to the Spark community.

• YouTube
• 21 minutes
• Self-Paced
• Free Video

Best Practices for Building Robust Data Platforms with Apache Spark and Delta

Learn best practices for building robust data platforms using Apache Spark and Delta. Gain insights on optimizing performance, scaling, security, and cost-effectiveness in big data architectures from real-world experiences.

• YouTube
• 27 minutes
• Self-Paced
• Free Video

Patterns and Operational Insights for Large-Scale Delta Lake Workloads

Explore effective Delta Lake patterns, optimization techniques, and operational insights for managing large-scale data workloads on Databricks, including streaming ETL, data enrichment, and analytics.

• YouTube
• 42 minutes
• Self-Paced
• Free Video

Delta Lake 0.7.0 and Spark 3.0 - Ask Me Anything Session

Illuminating AMA session with Apache Spark and Delta Lake experts. Explore Spark 3.0 and Delta Lake's latest features, use cases, and development insights. Perfect for data engineers seeking cutting-edge knowledge.

• YouTube
• 54 minutes
• Self-Paced
• Free Video