Data Factory & Orchestration

Advance your automation capabilities by building reliable, production-ready data pipelines in Microsoft Fabric. This hands-on course focuses on pipeline engineering using Fabric Data Factory and Spark orchestration to help you transform raw data into trusted, analytics-ready assets. You’ll create reusable transformations in Dataflows Gen2 using Power Query and schedule refreshes for business users. You’ll design Fabric Data Pipelines with parameters, triggers, and structured error-handling logic, and configure Copy Jobs to ingest incremental CSV and Parquet files from cloud storage. Moving into code-based processing, you’ll author PySpark notebooks for multi-step transformations with optimized Delta outputs, implement data quality validation rules to prevent bad loads, and monitor pipeline, notebook, and job logs to diagnose and resolve failures efficiently. By the end of the course, you’ll be equipped to automate complex data workflows, enforce quality controls, and orchestrate Spark jobs with performance monitoring—developing the engineering discipline required to deliver scalable, enterprise-grade data pipelines.