Candidate Generation & Retrieval Architectures

Microsoft via Coursera

Go to class Write review

Overview

Google, IBM & Meta Certificates – 40% Off

One Coursera Plus subscription covers most Professional Certificates on Coursera.

Unlock All Certificates

Large-scale recommendation systems depend on fast, accurate candidate generation before any ranking takes place. This course teaches you to architect and optimize production-grade retrieval infrastructures capable of surfacing relevant candidates from multi-million item catalogs within strict sub-50ms latency constraints. You will implement distributed matrix factorization models and Bayesian Personalized Ranking for implicit feedback datasets, build hybrid semantic embedding pipelines to resolve cold-start bottlenecks, and construct two-tower retrieval models optimized for online serving. You will also implement graph-based multi-hop networks using PyTorch Geometric and sequence-based causal transformers for next-item prediction, all executed programmatically within Azure Databricks and Azure ML SDK v2 environments. This course is designed for Advanced Machine Learning Engineers, Infrastructure Engineers, and Data Platform Architects who are responsible for scaling retrieval services across large item catalogs. You should be comfortable with Python, PyTorch, and core ML evaluation metrics such as Recall@K and NDCG@K. You will also need access to an active Azure subscription with a configured Azure Databricks workspace and a running compute cluster, as hands-on activities throughout the course are performed in Databricks notebooks. Familiarity with FAISS for vector indexing and similarity search is also expected before beginning.

Syllabus

RecSys Landscape: Formulate Architectural Designs
- Learn the core components of the multi-stage recommendation pipeline and how to scale from hundreds of millions of candidates to a final curated list.
RecSys Landscape: Evaluate Domain Constraints
- Analyze how different business models (social feeds, streaming media, and multi-tenant enterprise B2B SaaS applications) completely shift caching strategies, data isolation requirements, and graph topologies.
Collaborative Filtering: Implement Matrix Factorization
- Dive into the mathematics and distributed implementation of SVD and PySpark ALS for collaborative filtering on massive datasets.
Collaborative Filtering: Apply Bayesian Personalized Ranking
- Shift to ranking loss frameworks by implementing BPR and negative sampling strategies natively in PyTorch and tracking experiment metrics via Azure ML SDK v2.
Content & Hybrid: Content Embedding Extraction
- Learn to extract and leverage structured features and unstructured embeddings to build a robust content-based representation layer for your items, utilizing both scalable enterprise API endpoints and open-source foundation models.
Content & Hybrid: Hybrid Routing & Cold Start
- Architect routing logic to blend content-based and collaborative signals via deep learning layers, building fallback mechanisms to solve the cold-start problem.
Two-Tower: Neural Retrieval Architecture
- Architect the structural components of a deep neural retrieval network using dual-encoder designs and contrastive learning methodologies.
Two-Tower: Model Serialization & Serving
- Prepare your trained two-tower model for production by pre-computing item catalogs and serializing the user tower for lightning-fast inference.
ANN Indexing: Scaling Search with FAISS
- Understand the algorithms behind Approximate Nearest Neighbor search and implement highly optimized local indexes using FAISS.
ANN Indexing: Managed Enterprise Vector Search
- Move from local FAISS indexes to scalable, managed cloud infrastructure using Azure AI Search to implement hybrid search and semantic reranking.
Sequential Retrieval: SASRec, BERT4Rec & Data Leakage
- Compare leading sequence-aware candidate generation architectures and build robust verification steps to prevent inference-time data leakage.
Sequential Retrieval: Causal Transformers
- Architect advanced retrieval models by leveraging fine-tuned causal large language models (LLMs) as dual encoders to process complex interaction streams.
Graph Retrieval: LightGCN & Bipartite Graphs
- Traverse highly sparse networks using Graph Neural Networks (GNNs) to capture higher-order collaborative signals through multi-hop neighbor aggregation.
Graph Retrieval: Multi-Source Generation
- Architect the aggregation layer that merges graph-derived candidates with dense embeddings to formulate a robust, multi-source retrieval pool.
GenAI Module: AI-Augmented Candidate Generation
- Explore how modern Large Language Models can synthetically augment user profiles and act as direct semantic retrievers. You will analyze the critical trade-offs in system engineering between zero-shot prompt retrieval and indexed MRL vector search architectures.
Project Module: Multi-Source Retrieval Pipeline
- Synthesize your candidate-generation engineering skills by constructing a unified, high-scale multi-source retrieval pipeline within Azure Databricks. You will programmatically orchestrate this multi-source pool and execute global deduplication within Azure Databricks to maximize candidate diversity and recall, compiling a portfolio-ready technical architecture artifact.