MLCommons and MLPerf: Understanding AI Performance Benchmarks and Standards

Watch a comprehensive 30-minute presentation exploring MLCommons, a global non-profit industry consortium, and its flagship project MLPerf, delivered by Executive Director David Kanter at AI Field Day 6. Gain deep insights into how this organization, with over 125 members across six continents, works to advance AI technology through standardized benchmarking and collaborative innovation. Learn about the intricate relationship between big data, models, and computing power in AI development, illustrated through performance charts showing how model capabilities scale with increased resources. Discover MLPerf's diverse benchmark suite, covering training, inference, storage, and client systems, designed to provide transparent, reproducible measurements of AI system performance across hardware, algorithms, and software optimizations. Understand how these benchmarks help navigate the growing demands of AI computing while ensuring accountability through open-source results, peer review, and audits. Presented live in San Jose, California on January 29, 2025, this talk illuminates MLCommons' mission to improve AI accessibility, accuracy, safety, speed, and power efficiency through standardized industry benchmarks and collaborative development.