LLM-as-a-judge Courses and Certifications

Weights & Biases

From 0 to Automated Evals - Hands-on Workshop

1 rating

Master LLM-as-a-Judge evaluation frameworks from scratch using Weave, tackling real-world challenges and practical considerations for generative AI applications.

• YouTube
• 2 hours 39 minutes
• Self-Paced
• Free Video

UofU Data Science

LLM-as-Judge - Instruction Finetuning and Supervised Fine-Tuning

Explore LLM-as-judge techniques and instruction finetuning methods for supervised fine-tuning in this comprehensive data science presentation.

• YouTube
• 1 hour 20 minutes
• Self-Paced
• Free Video

OpenSource Connections

From LLM-as-a-Judge to Human-in-the-Loop - Rethinking Evaluation in RAG and Search

Explore RAGElo, an ELO-style ranking framework for evaluating LLM judges in RAG systems, plus human-in-the-loop tools for better search relevance assessment.

• YouTube
• 46 minutes
• Self-Paced
• Free Video

Design, Develop, and Deploy Multi-Agent Systems with CrewAI

10 ratings at Coursera

Master multi-agent AI systems with CrewAI to automate complex workflows, build collaborative agent teams, and deploy production-ready solutions for real-world applications.

• Coursera
• 21 hours 55 minutes
• Self-Paced
• Paid Course

DeepLearning.AI;  Snowflake

Building and Evaluating Data Agents

Master multi-agent workflows with LangGraph and Snowflake to create trustworthy data agents that plan, search, retrieve, and evaluate responses with proper goal-plan-action alignment.

• Coursera
• 2 hours 30 minutes
• Self-Paced
• Paid Course

UiPath

UiPath Agentic Automation Associate

Master intelligent agent development in UiPath Studio Web through no-code tools, prompt engineering, context configuration, and evaluation strategies for enterprise-ready AI automation.

• Coursera
• 13 hours 1 minute
• Self-Paced
• Paid Course

Edureka

CrewAI Flows and Monitoring

Master CrewAI Flows to build scalable multi-agent systems with event-driven workflows, guardrails, human-in-the-loop design, and observability tools like LangSmith and AgentOps.

• Coursera
• 9 hours 35 minutes
• Self-Paced
• Paid Course

Qdrant - Vector Database & Search Engine

Challenges in Using LLM-as-a-Judge for Production Evaluations

Explore key techniques and challenges in using LLMs as evaluation tools for AI applications, with insights on improving RAG-based systems and implementing production-grade assessments.

• YouTube
• 43 minutes
• Self-Paced
• Free Video

AutoML Seminars

Tuning LLM Judge Design Decisions for 1/1000 of the Cost

Discover how to systematically tune LLM judge hyperparameters using multi-objective multi-fidelity optimization to achieve better accuracy at 1/1000th the cost while ensuring reproducibility.

• YouTube
• 42 minutes
• Self-Paced
• Free Video

OpenSource Connections

Judge Moody's - Automating Semantic Search Relevance Evaluation with LLM Judges

Discover how Moody's automated semantic search evaluation using LLM judges, achieving 80% agreement with experts while cutting experiment time from days to minutes.

• YouTube
• 48 minutes
• Self-Paced
• Free Video

OpenSource Connections

Beyond LLM-as-Judge Towards LLM Relevance Engineering

Explore advanced LLM ensemble techniques for search relevance engineering, moving beyond single judges to multi-strategy systems that guide feature development and optimization.

• YouTube
• 42 minutes
• Self-Paced
• Free Video

LLM AI Agent Evaluations and Observability with Galileo AI

11 ratings at Udemy

Build Robust AI Agents | Monitor Production AI Agents | Build Custom Evals | Master Galileo AI | For Engineers

• Udemy
• 7 hours 29 minutes
• Self-Paced
• Paid Course

Peter Yang

Complete Beginner's Course on AI Evaluations in 50 Minutes - 2025

Master AI evaluation fundamentals through hands-on demos building evals for customer support agents, from creating golden datasets to aligning LLM judges with human judgment.

• YouTube
• 52 minutes
• Self-Paced
• Free Video

Google Cloud Tech

The Agent Factory - Agent Evaluation with ADK and Vertex AI - Episode 9

Master AI agent evaluation from local testing with ADK to enterprise-scale assessment using Vertex AI's GenAI Evaluation service for production-ready agents.

• YouTube
• 21 minutes
• Self-Paced
• Free Video

Yacine Mahdid

Understanding R1-Zero-Like Training with Dr. GRPO Algorithm

Explore R1-Zero-like training mysteries with Dr. GRPO algorithm's first author, covering LLM post-training, self-reflection detection, and algorithmic improvements in this deep-dive interview.

• YouTube
• 1 hour 9 minutes
• Self-Paced
• Free Video