Azure Databricks Cookbook

Packt via Coursera

Go to class Write review

Overview

Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off

One annual plan covers every course and certificate on Coursera. 40% off for a limited time.

Get Full Access

The Azure Databricks Cookbook shows you how to work with the latest as well as older versions of Apache Spark and integrate with various Azure resources for orchestrating, deploying, and monitoring big data solutions. You'll use Azure Databricks to build end-to-end solutions and address challenges in securing, productionizing, and monitoring them. This course provides a hands-on approach to mastering Azure Databricks for scalable analytics and data pipeline development. It covers essential skills for working with data in cloud environments, including integration with Azure services and real-time data processing. Designed for professionals looking to enhance their data engineering and analytics capabilities, it offers practical insights and actionable techniques. With a problem solution approach, this book teaches how to create end-end big data solution using data from various sources like batch and streaming and how to version control, deploy the solution to production and monitor the solution. This course is ideal for data engineers, data scientists, and big data professionals with prior experience in Apache Spark and Azure. It helps learners develop advanced skills in cloud-based data analytics and pipeline development.

Syllabus

Creating an Azure Databricks Service

This module guides learners through the process of setting up and managing an Azure Databricks workspace using the Azure CLI. You will explore how to add users and groups, configure permissions, and work with notebooks and jobs to streamline data application development. By the end, you'll be equipped to automate deployments and foster collaboration within Azure Databricks.

Reading and Writing Data from and to Various Azure Services and File Formats

This module explores how to efficiently read and write data between Azure Databricks and a variety of Azure services and file formats. Learners will gain hands-on experience connecting to Azure Blob Storage, ADLS Gen2, Azure SQL Database, Azure Synapse Analytics, and Azure Cosmos DB, as well as working with CSV, Parquet, and JSON files. By the end, you will be able to manage data ingestion and storage across multiple Azure platforms.

Understanding Spark Query Execution

This module explores how Spark executes queries, including how to inspect execution details, understand schema inference, and interpret query execution plans. Learners will also examine how joins and partitions impact performance and how to optimize query execution using Spark's tools and parameters. By the end, you'll be equipped to analyze and improve the efficiency of Spark applications.

Working with Streaming Data

This module introduces the fundamentals of processing streaming data using tools like Azure Event Hubs, Apache Kafka, and Spark Structured Streaming. Learners will explore how to connect to real-time data sources, manage log file streams, and implement advanced features such as window aggregation, trigger options, and checkpointing for fault tolerance.

Integrating with Azure Key-Vault, App Configuration and Log Analytics

This module guides learners through securely managing credentials and configuration settings in Azure by leveraging Key Vault, App Configuration, and Log Analytics. You will gain hands-on experience creating and deploying these resources using ARM templates and the Azure CLI, as well as learn how to centralize monitoring for Azure services.

Exploring Delta Lake in Azure Databricks

This module introduces the core concepts and practical applications of Delta Lake on Azure Databricks, including streaming data integration, data versioning, and ACID transaction support. Learners will explore how to optimize Delta tables, enforce data integrity with constraints, and manage concurrent operations for reliable big data solutions.

Implementing Near-Real-Time Analytics and Building Modern Data Warehouse

This module guides learners through building a modern data warehouse solution on Azure, integrating both batch and real-time analytics. You will create and configure Azure resources, simulate streaming data, process and transform data using Structured Streaming, and visualize near-real-time analytics with Databricks and Power BI.

Databricks SQL

This module introduces learners to the core features of Databricks SQL, including managing user access, utilizing query parameters and filters, and creating visualizations. Participants will gain practical skills for querying large datasets and presenting insights effectively within the Databricks environment.

DevOps Integrations and Implementing CI/CD for Azure Databricks

This module explores how to integrate DevOps practices with Azure Databricks, focusing on version control, CI/CD pipelines, and automated deployment strategies. Learners will gain hands-on experience connecting GitHub and Azure DevOps for notebook management and deploying resources across multiple environments. By the end, you'll be able to implement and manage CI/CD workflows tailored for Azure Databricks projects.

Understanding Security and Monitoring in Azure Databricks

This module explores essential security and monitoring practices in Azure Databricks, including configuring access controls, credential passthrough, and network security. Learners will also discover how to monitor cluster health using Ganglia and implement granular data access restrictions. By the end, you will be able to secure data and resources effectively within Azure Databricks environments.