How to Improve Quality of Multi-Agent Systems with Agent Bricks

Learn to build, evaluate, and deploy multi-agent systems on Databricks using natural language through this comprehensive 16-minute tutorial. Discover how to create a "Novel Ideas Bookworm" multi-agent system that combines an AI Genie for structured inventory data, a RAG chatbot for book recommendations, and an MCP server for web search to provide intelligent bookstore assistance. Master key agent use cases including information extraction, knowledge assistants, AI/ML genies for structured data, multi-agent supervisors, and custom LLM implementations. Explore the architecture of a vibe-coding stack featuring an AI Genie that utilizes real-time inventory and sales data, a Knowledge Assistant (Bookstock Bot) that serves as a RAG chatbot for personalized book recommendations, and a Multi-Agent Supervisor (Bookworm) that orchestrates multiple components including Tavly MCP for broader queries. Understand how to continuously improve multi-agent system output quality using natural language feedback from labeling sessions, built-in evaluation metrics, and SME feedback integration. Practice using the "Improve Quality" feature to label sessions, add expectations, and merge expert feedback effectively. Learn to evaluate system performance using the "Experiment" tab for detailed interaction traces, the "Scores" section for custom evaluations, and the "Evaluations" tab for built-in metrics such as correctness and relevance. Gain hands-on experience with MCP server implementation for routing general questions and discover how to build and evaluate complete multi-agent systems in minutes using Databricks' natural language interface.

Syllabus

Agent Bricks Overview – Build, evaluate, and deploy multi-agent systems on Databricks with natural language.
Architecture: Vibe-Coding Stack – Overview of the Novel Ideas multi-agent system:
AI Genie Inventory Space – Uses real-time inventory and sales data for bookstore assistance.
Knowledge Assistant Bookstock Bot – RAG chatbot for personalized book recommendations.
Multi-Agent Supervisor Bookworm – Orchestrates Bookstock Bot, Inventory Space, and Tavly MCP for broader queries.
Improving Quality – Use “Improve Quality” to label sessions, add expectations, and merge expert feedback.
Wrap-Up & Next Steps
Using the MCP Server – Routes general questions to Tavly MCP when not in references.
Evaluating Performance – Use “Experiment” for traces, “Scores” for custom evals, and “Evaluations” for metrics like correctness and relevance.
Conclusion – Build and evaluate a multi-agent system in minutes with natural language on Databricks.