Databricks Courses

DataSource V2 and Cassandra - A New Era of Speed and Flexibility

Explore DataSource V2 in Spark Cassandra Connector: enhanced speed, flexibility, and usability. Learn about Spark's understanding of Cassandra clustering and direct catalogue manipulation for improved data integration and analysis.

• YouTube
• 39 minutes
• On-Demand
• Free Video

Designing the Next Generation of Data Pipelines with Apache Spark - Zillow's Approach

Zillow's data engineering team shares their journey in redesigning data pipelines using Apache Spark, focusing on scalability, maintainability, and robustness while reducing code complexity.

• YouTube
• 27 minutes
• On-Demand
• Free Video

Apache Pulsar - Next Generation Messaging and Queuing System

Overview of Apache Pulsar's architecture, features, and benefits as a scalable messaging system with unified storage, processing, and serverless capabilities for building end-to-end streaming applications.

• YouTube
• 38 minutes
• On-Demand
• Free Video

Simplify and Boost Spark 3 Deployments with Hypervisor-Native Kubernetes

Explore integrating Spark 3 with VMware Pacific for scalable, reliable, and high-performance big data analytics deployments using hypervisor-native Kubernetes on vSphere 7.

• YouTube
• 33 minutes
• On-Demand
• Free Video

Scaling Up AI Research to Production with PyTorch and MLFlow

Explore PyTorch's latest advancements in AI research and production, including distributed training, model optimization, and deployment using MLFlow, with insights on scaling and efficiency.

• YouTube
• 44 minutes
• On-Demand
• Free Video

Model Explanation and Prediction Exploration Using Spark ML

Explore techniques for explaining non-linear models, showing feature contributions, and performing what-if analysis using Spark ML. Implement these methods to enhance model interpretability and enable interactive prediction exploration.

• YouTube
• 23 minutes
• On-Demand
• Free Video

Migrating Pinterest Apache Spark Clusters from HDFS to S3

Explore Pinterest's migration of Apache Spark clusters from HDFS to S3, addressing technical challenges, optimizing performance, and achieving a smooth transition process.

• YouTube
• 30 minutes
• On-Demand
• Free Video

Deploying Apache Spark Jobs on Kubernetes with Helm and Spark Operator

Learn to deploy Scala Spark jobs on Kubernetes using Helm and Spark Operator. Live coding demo shows scalable, flexible deployments with minimal custom configurations for stress-free implementation.

• YouTube
• 31 minutes
• On-Demand
• Free Video

User Defined Aggregation in Apache Spark - From Challenges to Improvements

Explore Apache Spark's User Defined Aggregate Functions (UDAFs), their evolution, and improvements in Spark 3.0. Learn to create custom aggregations, enhance performance, and contribute to the Spark community.

• YouTube
• 21 minutes
• On-Demand
• Free Video

Best Practices for Building Robust Data Platforms with Apache Spark and Delta

Learn best practices for building robust data platforms using Apache Spark and Delta. Gain insights on optimizing performance, scaling, security, and cost-effectiveness in big data architectures from real-world experiences.

• YouTube
• 27 minutes
• On-Demand
• Free Video

Patterns and Operational Insights for Large-Scale Delta Lake Workloads

Explore effective Delta Lake patterns, optimization techniques, and operational insights for managing large-scale data workloads on Databricks, including streaming ETL, data enrichment, and analytics.

• YouTube
• 42 minutes
• On-Demand
• Free Video

Delta Lake 0.7.0 and Spark 3.0 - Ask Me Anything Session

Illuminating AMA session with Apache Spark and Delta Lake experts. Explore Spark 3.0 and Delta Lake's latest features, use cases, and development insights. Perfect for data engineers seeking cutting-edge knowledge.

• YouTube
• 54 minutes
• On-Demand
• Free Video

Koalas - Scaling Pandas API on Apache Spark

Learn to scale pandas operations to big data using Koalas, an open-source project implementing the pandas API on Apache Spark. Transition seamlessly from single-machine to distributed environments for large-scale data science.

• YouTube
• 58 minutes
• On-Demand
• Free Video

Healthcare Claim Reimbursement Using Apache Spark - Optum's System Transformation

Optimizing healthcare claim processing with Apache Spark: Improved performance, reduced costs, and enhanced capabilities for hospitals' revenue recovery through efficient data processing and analysis.

• YouTube
• 49 minutes
• On-Demand
• Free Video

From Python to PySpark and Back Again - Unifying Single-Host and Distributed Deep Learning with Maggy

Unifying single-host and distributed deep learning with Maggy framework, enabling seamless transition from laptop to cluster for efficient model development and training using TensorFlow and PySpark.

• YouTube
• 28 minutes
• On-Demand
• Free Video