In this comprehensive course, students will learn about data engineering patterns and best practices as applied to batch and real-time analytical solutions using Microsoft Azure data platform technologies. Participants will start by mastering the core compute and storage technologies that form the foundation of analytical solutions. They will then examine how to architect analytical serving layers and explore critical data engineering considerations for managing source files. Students will develop skills in interactively exploring and analyzing data stored within a data lake environment. The course covers various ingestion techniques used to load data, including Apache Spark capabilities available in Azure Synapse Analytics or Azure Databricks, as well as data ingestion using Azure Data Factory or Azure Synapse pipelines.
Attendees will also explore the different approaches available for transforming data using the same technologies employed for data ingestion. Students will dedicate substantial time to learning how to monitor and evaluate the performance of analytical systems, enabling them to optimize data load operations and query execution against those systems. The course emphasizes the importance of implementing comprehensive security measures to protect data both at rest and in transit. Participants will learn how data within an analytical system can be leveraged to create business intelligence dashboards and build predictive machine learning models within Azure Synapse Analytics.
Audience Profile:
The primary audience includes data professionals, data architects, and business intelligence professionals who wish to develop expertise in data engineering and build analytical solutions utilizing data platform technologies available on Microsoft Azure.
Secondary audience members include data analysts and data scientists who work with analytical solutions built on the Microsoft Azure platform.
Job Role: Data Engineer
Preparation for Exam: DP-700
Skills Gained:
- Explore compute and storage options for data engineering workloads in Azure
- Design and implement the analytical serving layer
- Understand data engineering considerations and best practices
Prerequisites:
- Successful students begin this course with foundational knowledge of cloud computing, core data concepts, and professional experience with data solutions, specifically including completion of: AZ-900: Azure Fundamentals and DP-900: Microsoft Azure Data Fundamentals
Course Outline
- Module 1: Explore compute and storage options for data engineering workloads
- Module 2: Design and implement the serving layer
- Module 3: Data engineering considerations for source files
- Module 4: Run interactive queries using Azure Synapse Analytics serverless SQL pools
- Module 5: Explore, transform, and load data into the Data Warehouse using Apache Spark
- Module 6: Data exploration and transformation in Azure Databricks
- Module 7: Ingest and load data into the data warehouse
- Module 8: Transform data with Azure Data Factory or Azure Synapse Pipelines
- Module 9: Orchestrate data movement and transformation in Azure Synapse Pipelines
- Module 10: Optimize query performance with dedicated SQL pools in Azure Synapse
- Module 11: Analyze and optimize Data Warehouse storage
- Module 12: Support Hybrid Transactional Analytical Processing (HTAP) with Azure Synapse Link
- Module 13: End-to-end security with Azure Synapse Analytics
- Module 14: Real-time stream processing with Stream Analytics
- Module 15: Create a stream processing solution with Event Hubs and Azure Databricks
- Module 16: Build reports using Power BI integration with Azure Synapse Analytics
- Module 17: Perform integrated machine learning processes in Azure Synapse Analytics
