Generative AI and LLMs on Databricks

Overview

Build production GenAI systems on Databricks by mastering prompt engineering, RAG pipelines, model governance, and code intelligence. You will apply chain-of-thought, ReAct, and few-shot prompting patterns to decompose complex tasks, then construct retrieval-augmented generation pipelines that fuse vector search with BM25 using Reciprocal Rank Fusion. The course progresses from foundational techniques through production deployment across four weeks. Week one covers tokenization mechanics, sampling parameters, system prompts, and the Databricks Playground. Week two builds RAG systems using embeddings, MLflow experiment tracking, Feature Store, and PMAT code intelligence with TDG scoring and PageRank on call graphs. Week three addresses the fine-tuning vs RAG decision matrix, cryptographic model signing with SHA-256 chain-of-trust verification, AI Gateway configuration, model registry governance via Unity Catalog, and Databricks compute infrastructure. Week four integrates all concepts into a capstone project: a quality-aware code retrieval pipeline using trueno-rag and pmat. You will evaluate RAG quality using faithfulness-relevance diagnostic quadrants and six standard retrieval metrics including MRR, NDCG, recall, precision, hit rate, and MAP.

Syllabus

GenAI Foundations

Covers the four composable GenAI approaches (prompt engineering, RAG, fine-tuning, agents), tokenization mechanics (BPE, vocabulary tradeoffs), advanced prompting patterns (CoT, ReAct, few-shot), sampling parameters, and Databricks Playground for interactive model exploration.

RAG Systems

Covers embeddings and vector space semantics, MLflow experiment tracking for GenAI runs, Feature Store integration, code intelligence architecture (PMAT), hybrid RAG pipelines with RRF fusion, production RAG evaluation, and interactive notebook-based retrieval.

Advanced GenAI

Covers the fine-tuning vs RAG decision matrix, model security through cryptographic signing and chain-of-trust verification, AI Gateway for unified multi-provider access, model registry governance via Unity Catalog, and Databricks compute infrastructure for GenAI workloads.

Capstone

Integrate all course concepts into a single Rust project: a quality-aware code retrieval pipeline using trueno-rag for RAG infrastructure (chunking, embedding, hybrid retrieval, RRF fusion) and pmat for code quality signals (TDG grades, complexity, fault patterns). The capstone demonstrates end-to-end RAG: ingest, enrich, index, query, evaluate.