MLOps: Automated Pipelines and Model Monitoring

Overview

This course provides the practical skills needed to deploy and manage machine learning models in real-world production environments. You will learn to design robust data pipelines that ingest, version, and validate data, track experiments, and implement automated continuous training and deployment (CI/CD) pipelines. The course also covers containerizing model APIs, building scalable serving endpoints on AWS, and adding crucial components like feature stores, model registries, monitoring, data drift detection, explainability, and fairness assessments. By the end, you will be able to build, deploy, and maintain reliable and scalable machine learning systems effectively.

Syllabus

Welcome to ML Ops

Get oriented with the ML Ops course, its goals, prerequisites, and the tools you will use throughout.
Introduce the big picture and the course roadmap

Understand ML Data Ingestion & Validation

Collect, store, and version datasets for reproducible ML pipelines with traceability from raw data to training-ready sets, and validate them to detect ML-specific quality concerns.

Implement a Reproducible Data Pipeline with DVC and Great Expectations

Ingest a dataset from a public API, store it locally, track dataset versions using DVC and Git
Validate the dataset using Great Expectations

Understand Centralized Feature Stores

Learn how feature stores solve train-serve skew by providing consistent feature definitions and serving.

Define and Serve Features with Feast

Define feature views in Feast, generate point-in-time training datasets, and retrieve features for inference.

Understand Experiment Tracking with MLflow

Learn the principles of experiment tracking and how MLflow organizes parameters, metrics, and artifacts.

Implement Experiment Tracking with MLflow

Instrument a training script to log parameters, metrics, and models to MLflow, then compare runs in the UI.

Understand Continuous Training Pipelines

Learn CI/CD principles applied to ML, including triggers, orchestration, and automated retraining strategies.

Build a Continuous Training Pipeline with Prefect

Create a CI/CD pipeline that detects new data, retrains a model and validates it

Understand Model Lifecycle Management

Learn how model registries provide versioning, stage transitions, and governance for production ML models.

Manage Model Versions in the MLflow Model Registry

Understand Model API Containerization

Learn what containers are, why they solve environment inconsistency, and how Docker packages ML model APIs.

Containerize a Model API with Docker

Package a trained model and its FastAPI service into a Docker container, then build, run, and push the image.

Understand Automated Model Quality Testing

Learn how to design model test suites that validate performance, robustness, and fairness before deployment.

Implement Model Quality Testing with Deepchecks

Create automated test suites that evaluate model performance on subpopulations and block flawed deployments.

Understand Model Serving & Scalable Infrastructure

Learn how to deploy containerized models as APIs in production

Deploy and Scale a Model Serving Endpoint on AWS

Deploy a containerised model API to AWS, verify endpoint responses, and apply a cost optimization strategy to reduce serving costs.

Understand CI/CD for deployment

Learn how to automate deployment pipelines for ML models, including CI/CD principles, canary release patterns, and automated rollback strategies

Implement CI/CD using GitHub Actions

Build CI/CD pipeline that automatically deploys a containerized ML model from registry

Understand Model Serving Monitoring

Learn to monitor deployed ML model APIs for latency, error rates, resource usage and trigger alerts

Implement Monitoring for a Deployed ML Model API

Implement monitoring for a deployed ML model API, including metrics collection, dashboards, and alerting.

Understand Data Drift Detection

Learn the statistical foundations of data drift, how it differs from concept drift, and why monitoring matters.

Implement Data Drift Detection with Evidently AI

Generate drift reports comparing training and production data, identify drifted features, and set up alerts.

Understand Model Explainability

Learn explainability methods like SHAP and LIME, the difference between local and global explanations, and their limits.

Build a Model Explainability Service with SHAP

Create an API endpoint that returns SHAP-based feature importance explanations for individual predictions.

Understand Model Fairness Assessment

Learn fairness metrics like demographic parity and equalized odds, and how to evaluate bias across groups.

Implement Model Fairness Monitoring with Fairlearn

Assess model fairness across subgroups, generate fairness reports, and set up automated regression alerts.

Understand End-to-End ML Ops Architecture

Learn how managed cloud ML services integrate into a cohesive ML Ops lifecycle from training to monitoring.

Build an End-to-End ML Ops Workflow on AWS SageMaker

Create a full ML Ops pipeline on SageMaker with automated training, deployment, auto-scaling, and monitoring.

ML Ops Course Review

Review the skills gained throughout the ML Ops course and explore next steps for applying them in production.

Project: Building a Real-Time Financial News Sentiment Service

Learners will operationalize a fine-tuned sentiment analysis model for financial news by building a complete ML Ops workflow, replacing a brittle rules-based system.