Building Trust in Generative AI: Accuracy and Automation

In this 18-minute conference talk from Conf42 DevOps 2025, Nirhoshan Sivaroopan explores the critical aspects of building trust in generative AI through accuracy evaluation and automation. Learn about the importance of accuracy in AI systems and how to implement automated evaluation processes. Discover the differences between generative AI with and without context, and gain insights into the Retrieval-Augmented Generation (RAG) pattern. Explore essential metrics for evaluating RAG applications, including context recall, precision, faithfulness, and semantic similarity. Follow the key stages of accuracy evaluation from dataset preparation to metrics computation, and understand how to integrate these evaluations into CI/CD pipelines. The talk also addresses challenges in accuracy evaluation and provides practical guidance for implementing robust evaluation frameworks to ensure trustworthy AI systems.

Syllabus

00:00 Introduction to the Speaker and Topic
00:12 Importance of Accuracy in Generative AI
01:01 Automating Accuracy Evaluation
01:33 Generative AI with and without Context
02:47 Retrieval-Augmented Generation RAG Pattern
03:52 Metrics for Evaluating RAG Applications
05:21 Context Recall and Precision
08:25 Evaluating the Generator: Faithfulness and Semantic Similarity
09:48 Key Stages of Accuracy Evaluation
10:29 Dataset Preparation and Metrics Computation
12:48 Integrating Accuracy Evaluation into CI/CD Pipeline
15:58 Challenges in Accuracy Evaluation
17:29 Conclusion and Final Thoughts