Building Super Intelligent Tools - AI Agent Development and Experimentation

Explore the development of super intelligent programming agents in this 28-minute conference talk by Weights & Biases Co-Founder Shawn Lewis. Learn about the process behind building a programming agent that dominated the SWE-Bench leaderboard for months and discover key insights from agent experimentation loops. Understand AI agent nomenclature, evaluation methods, and experimentation frameworks while examining different types of experimentation processes. Dive into optimization strategies for each step of the experiment loop and discover how W&B Launch addresses complex experimentation challenges. Examine research phase optimization in Weave and explore the potential for AI to automatically improve experimental loops. Investigate the impact of reinforcement learning's resurgence on agent development, including insights from OpenPipe's Kyle Corbit on building reliable agents with RL. Address evaluation limitations through researcher agents and consider the current proximity to achieving self-improving AI systems.