Scale up your RAG system by building and querying a vector database. Learn to preprocess documents, store chunk embeddings in ChromaDB, retrieve relevant chunks, and construct prompts that can handle multiple context chunks. Additionally, see how to manage updates to your collection and how to approach large-scale ingestion using batch strategies.
Overview
Syllabus
- Unit 1: Chunking Text for Retrieval-Augmented Generation Systems
- Mastering Document Chunking Basics
- Preserve Sentence Boundaries in Chunks
- Enhance Text Chunking Skills
- Organize Chunks with Metadata
- Enhance Text with Keyword Detection
- Unit 2: Storing and Managing Text Chunks in Vector Databases
- Loading and Chunking text
- Building a ChromaDB Collection
- Dynamic Chunk Management in ChromaDB
- Dynamic Document Management in ChromaDB
- Unit 3: Retrieving and Prompt Building in RAG Systems
- Adding Query Options
- Complete the Retrieval
- Crafting Context-Rich LLM Prompts
- Metadata-Based Retrieval Enhancement
- Adding Distance-Based Filtering
- Unit 4: Metadata-Based Filtering in RAG Systems
- Where Clauses
- Crafting Metadata Enhanced Search
- Adding Date-Based Filtering
- Adding a simple Fallback