Overview

Google, IBM & Meta Certificates – 40% Off

One Coursera Plus subscription covers most Professional Certificates on Coursera.

Production generative AI systems require more than good prompts; they need systematic experimentation, rigorous evaluation, and managed model customization. This course teaches the core operational practices that make GenAI solutions reliable and continuously improvable. You'll design prompt versioning and variant experimentation frameworks, including A/B testing methodology, hyperparameter tracking, and rollback procedures using Microsoft Foundry. You'll run structured RAG optimization experiments across chunking strategies, embedding model variants, and retrieval configurations, evaluating each using groundedness, relevance, coherence, and fluency metrics via the Azure OpenAI Evaluation SDK. From there, you'll build automated evaluation frameworks with quality and safety thresholds, human-in-the-loop review triggers, and integration with CI/CD release decisions. You'll also design the fine-tuning operations lifecycle, including dataset versioning, LoRA configuration, model registry management, and evaluation against baseline models.

Syllabus

PromptOps: Versioning and Variant Experimentation
- This section teaches you to design prompt versioning schemas and configure variant experiments using the current Microsoft Foundry tooling, enabling systematic prompt optimization with full reproducibility.
PromptOps: Governance and Release Control
- This section teaches you to establish governance processes for production prompt management, including authoring standards, change control workflows, A/B testing methodology, and rollback procedures.
RAG Experiments: Design Experiment Matrices
- This module teaches you to design comprehensive RAG experiment matrices that systematically vary chunking strategies, embedding models, and retrieval configurations while maintaining reproducibility.
RAG Experiments: Evaluate Retrieval Performance
- This module teaches you to evaluate RAG pipeline experiments using the Azure OpenAI Evaluation SDK, interpret results across quality metrics, and make data-driven configuration decisions.
Evaluation Frameworks: Metrics by Use Case
- This module teaches learners to select and combine evaluation metrics by GenAI use case. Learners distinguish reference-based metrics, RAG retrieval/response metrics, classification metrics, rubric/custom evaluators and safety/security evaluators so they can design evaluation frameworks that avoid false confidence and production risk.
Evaluation Frameworks: Automated Gates and Human-in-the-Loop
- This module teaches you to build automated evaluation gates that integrate into CI/CD pipelines and design human-in-the-loop processes to handle cases where automation is insufficient.
Evaluation Frameworks: Limitations and Mitigations
- This module teaches you to understand the fundamental limitations of automated evaluation gates and design mitigations that prevent false confidence in quality controls.
Fine-Tuning Ops: Lifecycle and Configuration
- This section teaches you to design the complete operational lifecycle for fine-tuning generative AI models, from Azure ML data asset preparation through Azure OpenAI Foundry model catalog registration, with emphasis on reproducibility via seed/hyperparameter manifesting, Azure Monitor governance logging, and Azure Service Health base model retirement planning, with emphasis on reproducibility and governance.
Fine-Tuning Ops: Approach Selection and Model Promotion
- This section teaches you to evaluate different fine-tuning approaches against use case requirements and design model versioning and promotion strategies that enable safe progression from development to production.
GenAI Module: AI-Assisted Prompt Engineering
- This module teaches you to use AI assistance to draft, vary, review, and document prompts while preserving human ownership, safety review, version control, and evaluation-before-release discipline.
Project Module: Prompt & RAG Experimentation Framework
- Learners design a comprehensive Prompt & RAG Experimentation Framework for an enterprise scenario. Learners will create a PromptOps governance policy, design a RAG experiment matrix, build an evaluation framework with automated gates and human-in-the-loop processes, and document the complete experimentation workflow, producing portfolio-ready artifacts that demonstrate your ability to establish systematic optimization practices.