The Illusion of Done - Why the Real Work for AI Starts in Production

Learn why AI deployment is just the beginning of building successful production systems in this 16-minute conference talk from Databricks' Data + AI Summit. Discover the critical challenges that emerge after model training and pilot launches through real-world insights from Monte Carlo's Field CTO Shane Murray, who shares lessons learned from developing Monte Carlo's Troubleshooting Agent - an AI assistant designed to help users diagnose and fix data issues in production environments. Explore the concept of "The Illusion of Done" and understand why most teams face their toughest challenges after deployment, examining what commonly breaks in production AI systems. Gain behind-the-scenes insights into the architecture, integration strategies, and user experience design considerations that went into building Monte Carlo's production AI agent. Master the fundamentals of operationalizing AI reliability, including proven methods for evaluating AI performance in real-world scenarios, building the right team structure to support production AI systems, and establishing effective feedback loops between end users and model performance. Whether you're working with RAG pipelines or running large language models in production, acquire a comprehensive playbook for building trustworthy data and AI systems that both you and your users can rely on in mission-critical environments.