Databricks Courses

Deep Learning Pipelines for High Energy Physics Using Apache Spark and Distributed Keras

Explore CERN's Apache Spark-based data pipeline for deep learning in High Energy Physics. Learn about distributed training of neural network classifiers using BigDL and Analytics Zoo for improved event filtering at LHC experiments.

• YouTube
• 39 minutes
• Self-Paced
• Free Video

Downscaling Apache Spark Clusters - Challenges and Solutions

Explore techniques to improve downscaling in Spark-on-YARN clusters, addressing container fragmentation and shuffle data challenges for efficient resource management and faster autoscaling.

• YouTube
• 36 minutes
• Self-Paced
• Free Video

Modern ETL Pipelines with Change Data Capture - Building Resilient Data Streams

Discover how GetYourGuide built a resilient ETL pipeline using Debezium, Kafka, Spark, and Airflow. Learn to handle schema changes automatically, stream data from SQL databases, and refresh data lakes multiple times daily for fresher insights.

• YouTube
• 43 minutes
• Self-Paced
• Free Video

Managing the Complete Machine Learning Lifecycle with MLflow - Part 2

Explore MLflow's capabilities for managing machine learning lifecycles, from data preparation to model deployment, with practical insights on hyperparameter optimization and production workflows.

• YouTube
• 41 minutes
• Self-Paced
• Free Video

Best Practices for Building and Deploying Data Pipelines in Apache Spark

Learn best practices for building and deploying efficient, reproducible data pipelines in Apache Spark. Discover a toolkit that simplifies pipeline creation for non-engineers and explore deployment strategies.

• YouTube
• 41 minutes
• Self-Paced
• Free Video

The Parquet Format and Performance Optimization Opportunities

Explore Parquet format intricacies, columnar storage basics, and optimization techniques for Spark workloads. Learn about encoding, compression, predicate pushdown, and Delta Lake to enhance analytical performance.

• YouTube
• 41 minutes
• Self-Paced
• Free Video

Apache Spark Core - Practical Optimization Techniques - Partition Shaping and Job Optimization

Explore Apache Spark optimization techniques, including partition shaping, join and aggregate optimizations, and multi-dimensional parallelism to maximize cluster utilization and eliminate data skew.

• YouTube
• 1 hour 32 minutes
• Self-Paced
• Free Video

How to Automate Performance Tuning for Apache Spark

Discover techniques to automate Apache Spark performance tuning, addressing common issues and exploring tools for efficient, scalable data pipeline maintenance in production environments.

• YouTube
• 41 minutes
• Self-Paced
• Free Video

Building a Knowledge Graph with Spark and NLP for Novel Drug Recommendations

Explore drug discovery using knowledge graphs, NLP, and Spark. Learn techniques for efficient data processing, entity recognition, and relationship extraction to enhance pharmaceutical research.

• YouTube
• 41 minutes
• Self-Paced
• Free Video

Accelerating Apache Spark with GPUs Using RAPIDS

Explore GPU acceleration for data science with RAPIDS, offering significant performance gains over CPU-only solutions. Learn its functionality, architecture, and integration with Apache Spark for large-scale computation.

• YouTube
• 36 minutes
• Self-Paced
• Free Video

Performance Troubleshooting Using Apache Spark Metrics - Databricks Talk

Explore Apache Spark metrics for performance troubleshooting in distributed data processing. Learn to deploy dashboards, use sparkMeasure, and leverage Spark 3.0 improvements for optimizing workloads.

• YouTube
• 40 minutes
• Self-Paced
• Free Video

Listening at the Cocktail Party - Deep Neural Networks for Speech Separation

Explore deep neural networks for speech separation, focusing on data preparation, network architectures, and real-time implementation on embedded devices for smart audio applications.

• YouTube
• 40 minutes
• Self-Paced
• Free Video

Augmenting Machine Learning with Databricks Labs AutoML Toolkit

Explore how Databricks Labs AutoML Toolkit simplifies and optimizes machine learning processes, from data preparation to model optimization, using financial loan risk data examples.

• YouTube
• 30 minutes
• Self-Paced
• Free Video

Improving the Life of Data Scientists - Automating ML Lifecycle through MLflow

Explore Flock, an end-to-end platform leveraging MLflow to automate and simplify enterprise-grade machine learning, enhancing data scientist productivity and addressing regulatory challenges in ML adoption.

• YouTube
• 36 minutes
• Self-Paced
• Free Video

Performance Analysis of Apache Spark and Presto in Cloud Environments

Comparative analysis of Apache Spark and Presto on Amazon EMR, focusing on SQL performance using TPC-DS benchmark. Explores execution times, costs, and optimization techniques to guide big data analytics platform selection.

• YouTube
• 37 minutes
• Self-Paced
• Conference Talk