Data Engineer

This course focuses on testing knowledge of building data analytics solutions with Azure Synapse Serverless SQL. It contains multiple knowledge checks, suggesting a focus on assessing and reinforcing understanding of various concepts related to this Azure service.

The primary learning objective is to test and solidify your understanding of how to build data analytics solutions using Azure Synapse Serverless SQL. The multiple knowledge check files indicate that the folder covers a range of topics within this domain. You will likely encounter questions and scenarios designed to evaluate your comprehension of key concepts, best practices, and practical application of Azure Synapse Serverless SQL for data analysis. The content will help you gauge your proficiency and identify areas where further study may be needed.

The course "Implement a Data Streaming Solution with Azure Stream Analytics" focuses on learning how to build and deploy a data streaming pipeline using Azure Stream Analytics. Learners will learn the fundamentals of real-time data processing within the Azure cloud environment. This includes understanding how to ingest, process, and analyze data streams as they are generated. Key aspects likely covered involve configuring Azure Stream Analytics jobs, defining input and output sources for data streams, and writing queries to perform real-time analysis and transformations. The content will likely provide insights into integrating Azure Stream Analytics with other Azure services such as Event Hubs or IoT Hub for data ingestion and Power BI for visualization of processed data. Overall, the folder's content will guide someone through the process of creating a complete data streaming solution on Azure.

In this course, you will learn data engineering with Apache Spark in Azure Synapse Analytics, including using Delta Lake and data visualization.

You will learn how to master the core features and capabilities of Apache Spark for large-scale data processing and analytics within the Azure Synapse Analytics environment, including configuring Spark pools and using notebooks to run code for loading, analyzing, and visualizing data from a data lake, as well as understanding how Spark works in a distributed environment and how to use dataframes and Spark SQL for data manipulation.

You will be introduced to Delta Lake, an open-source storage layer that brings ACID transactions to Apache Spark. You will learn to create and use Delta Lake tables, including updating, querying previous versions (time travel), and using them for streaming data. You will explore how to define tables in the Spark metastore and query them using SQL, as well as how to use Delta Lake tables as sources and sinks for streaming data.

Begin transforming data using Spark, including loading data into dataframes, restructuring data, and saving it in formats like Parquet. This course covers partitioning data for optimization and filtering partitioned data in queries. The use of SQL for querying and transforming data is also covered, along with visualizing data within Spark notebooks using built-in charts and Python libraries like Matplotlib.