Apache Spark Courses and Certifications

Databricks

Fine-Tuning and Enhancing Performance of Apache Spark Jobs

Explore best practices for optimizing Apache Spark jobs, including resource allocation, parameter tuning, and performance enhancement techniques. Learn to identify bottlenecks and implement solutions for improved efficiency.

• YouTube
• 25 minutes
• Self-Paced
• Free Video

Databricks

Running Apache Spark on Kubernetes - Best Practices and Pitfalls

Learn best practices and pitfalls for running Apache Spark on Kubernetes, covering setup, performance optimization, resource management, security, and future developments in this comprehensive overview.

• YouTube
• 25 minutes
• Self-Paced
• Free Video

Databricks

Introducing Apache Spark 3.0 - A Decade of Progress and Future Outlook

Apache Spark 3.0: Reflecting on a decade of progress and unveiling new features to enhance data processing accessibility, speed, and compatibility across various sources and environments.

• YouTube
• 29 minutes
• Self-Paced
• Free Video

Databricks

Apache Spark Core - Practical Optimization Techniques - Partition Shaping and Job Optimization

Explore Apache Spark optimization techniques, including partition shaping, join and aggregate optimizations, and multi-dimensional parallelism to maximize cluster utilization and eliminate data skew.

• YouTube
• 1 hour 32 minutes
• Self-Paced
• Free Video

Bryan Cafferky

Master Databricks and Apache Spark - Introduction

Learn the fundamentals of Databricks and Apache Spark, exploring scale-up vs. scale-out concepts and laying the groundwork for mastering these powerful data processing tools.

• YouTube
• 32 minutes
• Self-Paced
• Free Video

Bryan Cafferky

Create a Spark Cluster - Master Databricks and Apache Spark Step by Step - Lesson 4

Learn to create and use Apache Spark clusters, understand their differences from Databricks, and explore data querying with Zeppelin notebooks in this step-by-step guide.

• YouTube
• 21 minutes
• Self-Paced
• Free Video

Great Learning

Spark Streaming Tutorial - Twitter Real Time Streaming - Apache Spark for Beginners

Apache Spark Streaming is a scalable fault-tolerant streaming processing system that natively supports both batch and streaming workloads. This video will help you to understand every aspect of Spark Streaming along with the examples and visuals for bett…

• YouTube
• 2 hours 35 minutes
• Self-Paced
• Free Video

CodeWithYu

Realtime Socket Streaming with Apache Spark - End-to-End Data Engineering Project

Build a real-time data streaming pipeline using TCP/IP Socket, Apache Spark, OpenAI LLM, Kafka, and Elasticsearch. Process 7 million records for sentiment analysis and efficient data indexing.

• YouTube
• 1 hour 42 minutes
• Self-Paced
• Free Video

Introduction to Spark with sparklyr in R

Learn how to run big data analysis using Spark and the sparklyr package in R, and explore Spark MLIb in just 4 hours.

• DataCamp
• 4 hours
• Self-Paced
• Free Trial Available

Introduction to Spark SQL in Python

Learn how to manipulate data and create machine learning feature sets in Spark using SQL in Python.

• DataCamp
• 4 hours
• Self-Paced
• Free Trial Available

Databricks

Recent Parquet Improvements in Apache Spark - Vectorized Complex Types and Column Index Support

Explore recent improvements in Apache Spark's Parquet performance, including vectorized read support for complex types and column index support, leading to significant speed enhancements and efficient data processing.

• YouTube
• 37 minutes
• Self-Paced
• Free Video

Databricks

Natural Language Query and Conversational Interface to Apache Spark

Explore conversational analytics with Apache Spark, enabling natural language queries for data insights. Learn about NLP-to-SQL translation and its integration with dialog management and visualization.

• YouTube
• 25 minutes
• Self-Paced
• Free Video

Databricks

Modernizing Apache Spark 3.0 Applications - Best Practices and Optimization Techniques

Optimize legacy Spark applications for 3.0: leverage Adaptive Query Execution, columnar processing, GPU acceleration, and ML integration. Concrete examples and lessons from modernizing analytics workloads.

• YouTube
• 25 minutes
• Self-Paced
• Free Video

Databricks

Hybrid Apache Spark Architecture: Optimizing YARN and Kubernetes for Lyft's Workloads

Explore Lyft's hybrid Apache Spark architecture using YARN and Kubernetes, addressing scalability challenges and optimizing workload management for improved reliability and efficiency.

• YouTube
• 44 minutes
• Self-Paced
• Free Video

Databricks

Delight - Improved Apache Spark UI for Performance Troubleshooting

Explore Delight, a free monitoring dashboard for Apache Spark. Learn to troubleshoot and optimize data engineering pipelines using system metrics and Spark information for improved performance and cost-effectiveness.

• YouTube
• 21 minutes
• Self-Paced
• Free Video