Overview

This course provides a comprehensive exploration of the entire data lifecycle in generative AI systems. You will learn to plan and build effective data pipelines, manage multimodal data processing, and implement data versioning and lineage. The course covers essential concepts in access control, data cataloging, and synthetic data creation and evaluation. A key project involves developing an Ethical Multi-Agent Data Orchestrator. By integrating theory with practical applications, this course prepares you to proficiently manage data in AI contexts, ensuring ethical and effective use of generative technologies.

Syllabus

Course Introduction

Get oriented to GenAI data management: course overview, learning goals, prerequisites, Azure setup, tooling, workspaces, and essential skills for the capstone project.

The Generative AI Data Lifecycle

Learn how generative AI systems manage data across the entire lifecycle—collection to inference—requiring continuous governance, user data ownership, and bias monitoring beyond training.

AI Data Lifecycle Planning
Understanding Data Management for RAG

Explore enterprise RAG: its layered data architecture, chunking strategies, retrieval design, embeddings, and the governance of prompts and conversational memory.

Building a RAG Data Pipeline

Learn to refactor an enterprise RAG pipeline for privacy, separating user and enterprise data, supporting per-session storage, and enabling right-to-erasure on user data.

Understanding Multimodal Data Processing

Learn to build pipelines that process and align text and images, ensuring accurate retrieval from multimodal documents using modality-specific preprocessing and structured records.

Implementing Multimodal Data Pipelines

Build a multimodal data pipeline to extract, describe, and index every image from a PDF, enabling fine-grained product retrieval in an apparel catalog RAG chatbot.

Understanding Data Versioning and Lineage

Learn why and how to version data, code, and pipelines in generative AI to enable traceability, reproducibility, auditability, and responsible AI governance.

Implementing Data Versioning and Lineage

Learn to version data, prompts, and artifacts in a RAG pipeline using DVC and Git, enabling full lineage tracing, reproducible rollbacks, and defensible audit trails in generative systems.

Understanding Access Control for AI Data Systems

Explore essential access control for AI systems: authentication, authorisation, pre-retrieval filtering, identity propagation, auditability, and enforcing security throughout data pipelines.

Implementing Access Management for AI Data Systems

Learn to enforce access management in AI data systems by configuring role-based policies, controlling data access pre-retrieval, and maintaining auditable query logs.

Understanding Data Cataloging and Metadata Management

Learn how data catalogs and metadata ensure generative AI uses approved, reliable, and governed data, preventing unfiltered or unauthorized information from influencing AI outputs.

Implementing Data Catalogs

Automate data catalog classification to control, filter, and trace RAG chatbot sources by metadata, enabling category-based exports and defensible, audit-ready retrieval.

Understanding Synthetic Data

Learn what synthetic data is, when to use it, key risks like bias amplification, quality validation steps, and best practices for governance and provenance in generative AI workflows.

Generating and Evaluating Synthetic Data

Build a synthetic Q&A pipeline that redacts PII before LLM use, ensuring privacy and auditability while generating clean, reviewable datasets for fine-tuning.

Project: Ethical Multi-Agent Data Orchestrator

Build a multi-agent AI assistant that securely queries distributed data sources, applies ethical safeguards, and delivers compliant responses without centralizing sensitive data.