Databricks Courses

Improving Broadcast Joins in Apache Spark SQL

Discover techniques to enhance broadcast joins in Apache Spark SQL, including executor-side broadcasting and memory optimization, for improved performance in large-scale ETL pipelines and complex data processing tasks.

• YouTube
• 28 minutes
• Self-Paced
• Free Video

Fine-Tuning and Enhancing Performance of Apache Spark Jobs

Explore best practices for optimizing Apache Spark jobs, including resource allocation, parameter tuning, and performance enhancement techniques. Learn to identify bottlenecks and implement solutions for improved efficiency.

• YouTube
• 25 minutes
• Self-Paced
• Free Video

Real-Time Forecasting at Scale Using Delta Lake and Delta Caching

Explore real-time forecasting at scale using Delta Lake and Delta Caching. Learn efficient data sampling, storage, and caching techniques for handling massive datasets and achieving rapid forecast response times.

• YouTube
• 25 minutes
• Self-Paced
• Free Video

Building a Streaming Microservice Architecture with Apache Spark Structured Streaming

Explore building a scalable, fault-tolerant streaming microservice architecture using Apache Spark, Kafka, gRPC, and Delta Lake for real-time data processing and reliable insights.

• YouTube
• 39 minutes
• Self-Paced
• Free Video

Productionizing Machine Learning with Apache Spark, MLflow and ONNX - Cloud Deployment Using SQL Server

Explore productionizing machine learning using Apache Spark, MLflow, and ONNX with SQL Server. Learn model management, lifecycle orchestration, and leveraging SQL Server for ML model storage and deployment.

• YouTube
• 26 minutes
• Self-Paced
• Free Video

Building the Petcare Data Platform with Delta Lake and Spark ETL Pipeline

Explore Mars Petcare's data platform using Delta Lake and Spark ETL pipeline 'Kyte'. Learn about advantages over Azure Data Factory and leveraging Delta Lake for ETL configurations and data science.

• YouTube
• 27 minutes
• Self-Paced
• Free Video

Productionizing Deep Reinforcement Learning with Spark and MLflow

Aprenda a aplicar Deep Reinforcement Learning em escala usando Spark e MLflow para personalização de jogos e aumento de engajamento, com dicas práticas para superar desafios de produção.

• YouTube
• 26 minutes
• Self-Paced
• Free Video

Zeus - Uber's Highly Scalable Distributed Shuffle as a Service

Explore Uber's Zeus: a highly scalable, distributed shuffle service powering data processing. Learn its architecture, integration with Spark, and performance advantages over traditional external shuffle methods.

• YouTube
• 31 minutes
• Self-Paced
• Free Video

Using Apache Spark for Predicting Degrading and Failing Parts in Aviation

Leveraging Apache Spark for predictive maintenance in aviation, enabling efficient data processing, anomaly detection, and component health scoring to improve fleet readiness and prevent failures.

• YouTube
• 28 minutes
• Self-Paced
• Free Video

Care and Feeding of Catalyst Optimizer - Practical Troubleshooting for Spark SQL

Explore Spark's Catalyst Optimizer challenges, including UDF issues, codegen limitations, and JVM tuning. Learn to diagnose and solve complex query problems for improved Spark SQL performance.

• YouTube
• 42 minutes
• Self-Paced
• Free Video

Faster Data Integration Pipeline Execution Using Spark-Jobserver

Optimize data integration pipelines using Spark-Jobserver for faster execution, context reuse, and efficient resource management. Learn configuration, security, and API integration for improved performance.

• YouTube
• 32 minutes
• Self-Paced
• Free Video

Encryption and Masking for Sensitive Apache Spark Analytics Addressing CCPA and Governance

Practical methods for balancing data encryption and analytics in Apache Spark, addressing CCPA and governance requirements while maintaining analytical value.

• YouTube
• 27 minutes
• Self-Paced
• Free Video

Data Mesh in Practice - From Data Lake to Distributed Architecture at Zalando

Explore Zalando's journey from centralized Data Lake to distributed Data Mesh architecture, focusing on data ownership, quality, and accessibility in large-scale e-commerce environments.

• YouTube
• 30 minutes
• Self-Paced
• Free Video

Running Apache Spark Jobs Using Kubernetes

Learn to run Apache Spark jobs on Kubernetes, unifying analytics and data science on a cloud-native architecture. Discover how this approach simplifies infrastructure, enables on-demand deployment, and eliminates big data cluster overhead.

• YouTube
• 52 minutes
• Self-Paced
• Free Video

Tackling Scaling Challenges of Apache Spark at LinkedIn - Infrastructure Optimization and User Productivity

Explore LinkedIn's journey scaling Apache Spark, addressing infrastructure challenges, optimizing resource management, and enhancing user productivity through automated analysis and efficiency improvements.

• YouTube
• 26 minutes
• Self-Paced
• Free Video