LLM Monitoring & Tracing: AI Observability with Datadog

Packt via Coursera

Go to class Write review

Details

Go to class

Provider

Coursera
Pricing

Paid Course
Languages

English
Certificate

Certificate Available
Effort

7 weeks, 1 hour a week
Sessions

Self-Paced
Level

Intermediate
Subtitles

English

Found in

Overview

Google, IBM & Meta Certificates – 40% Off

One Coursera Plus subscription covers most Professional Certificates on Coursera.

Unlock All Certificates

Learn to optimize LLM observability with Datadog. This course covers setting up Datadog, instrumenting AI workflows, debugging multi-agent systems, and monitoring performance, security, and costs in enterprise environments. Gain hands-on experience in ensuring production-grade AI applications. This course provides an in-depth look at LLM observability using Datadog, equipping you with essential skills for monitoring and tracing AI applications. You will begin with Datadog setup, understanding span types, SDK integrations, and the process of tracing LLM calls in local environments. As the course progresses, you’ll dive into instrumenting multi-step AI workflows, using annotations and tags to track performance, and debugging complex agentic AI systems. You’ll also explore LangChain integration, learning how to instrument RAG pipelines and use Datadog for full observability. The course goes further by focusing on evaluations, quality monitoring, and A/B testing to optimize LLM performance. You’ll learn how to create custom evaluations and integrate them with your code for automated monitoring. In addition to performance and debugging, you’ll tackle cost optimization, ensuring your LLM applications run efficiently and affordably. You’ll also gain knowledge in security and compliance practices, such as PII redaction and deployment best practices. By the end of the course, you will have a comprehensive understanding of deploying and maintaining observability in production-grade LLM applications, ensuring they are secure, efficient, and cost-effective. This course is designed for AI engineers, DevOps professionals, data scientists, and anyone working with AI systems, especially LLM applications. It is ideal for those focused on AI observability, monitoring, and optimization in production environments. A basic understanding of cloud-based systems and software development is recommended. Familiarity with Datadog or LLM technologies is beneficial but not required. This course takes a hands-on approach, offering practical exercises to set up observability tools, debug complex AI workflows, and optimize performance. Participants will learn through interactive sessions and real-world scenarios, ensuring they gain valuable skills to deploy and monitor LLM applications effectively in production. This course is based on LLM Monitoring & Tracing: AI Observability with Datadog, by Paulo Dichone. This course is licensed and distributed by Packt. All rights reserved. Packt is one of the world's most prolific publishers of cutting-edge technical content. For over two decades we've made it our mission to curate and publish the knowledge of only the very best technical experts. We focus on real-world courses that help our customers get the job done, with coverage that extends across a wide range of established and cutting-edge technical topics. If you're an individual or an organisation that embraces learning by doing, Packt is the perfect fit for you.

Syllabus

Introduction & Enterprise Value
- This module introduces the importance of LLM observability in enterprise settings, covering key concepts, practical skills, and real-world applications using Datadog. Learners will understand why monitoring LLMs is critical and gain hands-on insight into Datadog's dashboard capabilities.
Setting up LLM Observability
- This module provides hands-on guidance on setting up and testing LLM observability using Datadog. Learners will explore key concepts like traces, spans, and SDK integrations, and gain practical skills in monitoring and analyzing LLM applications in local environments.
Instrumenting LLM Applications
- This module covers the techniques for instrumenting LLM applications, including creating spans with annotations, tracking multi-step workflows, and setting up observability in RAG pipelines. Learners will gain hands-on experience with tools like LangChain and Datadog to monitor and optimize LLM-based systems effectively.
Tracing Agentic AI Workflows
- This module equips learners with the skills to trace, monitor, and debug agentic AI workflows using tools like Datadog. It covers multi-agent interactions, observability techniques, and strategies for addressing common issues in AI systems. By the end, learners will be able to analyze and optimize AI-driven processes effectively.
Evaluations & Quality Monitoring
- This module provides hands-on training on setting up and managing evaluations and quality monitoring for large language models. Learners will explore dataset creation, golden evaluation sets, A/B testing, and integration of monitoring tools like Datadog. The content emphasizes practical skills for assessing and optimizing LLM performance.
Cost Monitoring & Optimization
- This module equips learners with strategies and tools for effectively monitoring and optimizing costs associated with large language models. It covers automated cost tracking using Datadog, practical cost optimization techniques, and methods for analyzing and managing expenses in LLM applications.
Security, Compliance & Production Patterns
- This module covers essential security, compliance, and production practices for deploying large language models (LLMs) in regulated environments. Learners will gain hands-on experience with tools like Datadog for PII redaction, data scanning, and production deployment. The focus is on implementing safeguards and best practices to ensure privacy, security, and operational readiness.