Coursera Spring Sale
40% Off Coursera Plus Annual!
Grab it
Learn to implement comprehensive monitoring solutions for production AI agents through this 36-minute conference talk from Databricks. Explore the Mosaic Agent Monitoring solution and discover how to track critical operational metrics and quality indicators using an intuitive dashboard designed for deployed AI agents. Follow a practical demonstration showing how to iteratively improve a production customer support agent while reducing operational costs. Master techniques for monitoring essential operational metrics including request volume, latency, errors, and cost efficiency across AI agent deployments. Understand strategies for implementing continuous quality evaluation using AI judges that assess correctness, guideline adherence, and safety without requiring ground truth labels. Discover best practices for setting up effective monitoring dashboards that enable dimension-based analysis across time periods, user feedback, and topic categories. Learn methods for collecting and integrating end-user feedback to create closed-loop systems that drive iterative improvement of AI agents. Gain insights into identifying and proactively fixing quality problems with GenAI agent responses before they become major issues, understanding user usage patterns, and implementing feature improvements to enhance agent performance.
