Databricks Courses

Observability for Data Pipelines with OpenLineage

Explore data pipeline observability using OpenLineage, enhancing reliability and auditability. Learn how metadata collection enables understanding of data flow and dependencies across teams and technologies.

• YouTube
• 24 minutes
• On-Demand
• Free Video

Natural Language Query and Conversational Interface to Apache Spark

Explore conversational analytics with Apache Spark, enabling natural language queries for data insights. Learn about NLP-to-SQL translation and its integration with dialog management and visualization.

• YouTube
• 25 minutes
• On-Demand
• Free Video

Modernizing Apache Spark 3.0 Applications - Best Practices and Optimization Techniques

Optimize legacy Spark applications for 3.0: leverage Adaptive Query Execution, columnar processing, GPU acceleration, and ML integration. Concrete examples and lessons from modernizing analytics workloads.

• YouTube
• 25 minutes
• On-Demand
• Free Video

Dumb-Proofing Data Pipelines: Techniques for Configurable and Maintainable ETL - Databricks

Techniques to create configurable, maintainable data pipelines. Learn to externalize configurations, validate inputs, and leverage Scala features for robust, easily deployable ETL processes.

• YouTube
• 22 minutes
• On-Demand
• Free Video

Learn to Use Databricks for Data Science

Discover how Databricks simplifies data science with a unified platform for analytics workloads, from data preparation to predictive analytics, enhancing collaboration and productivity at scale.

• YouTube
• 39 minutes
• On-Demand
• Free Video

Identifying Sensitive Data and Mitigating Risk in Apache Spark and Databricks

Discover sensitive data in Apache Spark and Databricks using BigID's platform. Learn to scale discovery, label data, and apply necessary guardrails for compliance and risk mitigation in data analytics.

• YouTube
• 27 minutes
• On-Demand
• Free Video

Hybrid Apache Spark Architecture: Optimizing YARN and Kubernetes for Lyft's Workloads

Explore Lyft's hybrid Apache Spark architecture using YARN and Kubernetes, addressing scalability challenges and optimizing workload management for improved reliability and efficiency.

• YouTube
• 44 minutes
• On-Demand
• Free Video

Optimizing Spark SQL Jobs with Parallel and Asynchronous IO

Optimizing Spark SQL jobs through parallel and async IO techniques, including file-level and row group-level parallel read, asynchronous spill, and parquet column family, resulting in 5-30% performance improvement.

• YouTube
• 21 minutes
• On-Demand
• Free Video

Getting Started with Databricks SQL Analytics

Explore Databricks SQL Analytics: set up SQL endpoints, query workspaces, create dashboards, and connect BI tools. Learn how this tool enhances the Lakehouse architecture for SQL-savvy data analysts.

• YouTube
• 27 minutes
• On-Demand
• Free Video

Fully Utilizing Spark for Data Validation with Fugue and Pandera

Explore lightweight data validation for Spark using Fugue and Pandera, enabling partition-specific rules and efficient big data processing without compromising on functionality.

• YouTube
• 22 minutes
• On-Demand
• Free Video

Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSA

Designing an intelligent, event-driven data platform for EFSA's digital transformation, enhancing transparency in food safety risk assessment through Azure and Databricks technologies.

• YouTube
• 27 minutes
• On-Demand
• Free Video

Efficient Large-Scale Language Model Training on GPU Clusters

Explores advanced techniques for efficient large-scale language model training on GPU clusters, discussing parallelism methods, novel scheduling, and performance optimization to achieve unprecedented model sizes and training speeds.

• YouTube
• 23 minutes
• On-Demand
• Free Video

Drug and Vaccine Discovery Using Knowledge Graphs and Apache Spark

Explore how GSK uses knowledge graphs and Apache Spark to build a massive medical database, enabling advanced drug and vaccine discovery through innovative data querying and machine learning techniques.

• YouTube
• 31 minutes
• On-Demand
• Free Video

Delight - Improved Apache Spark UI for Performance Troubleshooting

Explore Delight, a free monitoring dashboard for Apache Spark. Learn to troubleshoot and optimize data engineering pipelines using system metrics and Spark information for improved performance and cost-effectiveness.

• YouTube
• 21 minutes
• On-Demand
• Free Video

Deep Dive into New Features of Apache Spark 3.1

Explore Apache Spark 3.1's new features: ANSI SQL compliance, streaming enhancements, Python improvements, and performance optimizations for faster, easier, and smarter data processing.

• YouTube
• 49 minutes
• On-Demand
• Free Video