Benchmarks Are Memes - How What We Measure Shapes AI and Us

Explore how benchmarks function as powerful cultural forces that shape both AI development and society's future in this 16-minute conference talk from the AI Engineer World's Fair. Discover why the metrics we choose to measure AI performance become self-fulfilling prophecies that guide technological progress toward specific abilities and ultimately define humanity's evolving relationship with artificial intelligence. Learn from practical examples across finance, journalism, education, and gaming industries, including insights from AI consulting work at Every Inc. and an engaging AI Diplomacy benchmark demonstration. Examine what makes benchmarks impactful, approachable, and inspiring while understanding their role as cultural memes that influence not just technical outcomes but societal direction. Gain the knowledge and inspiration needed to design benchmarks intentionally around what genuinely matters—cooperation, creativity, education, and meaningful human experiences—helping steer AI development toward outcomes that truly benefit humanity.