Coursera Spring Sale
40% Off Coursera Plus Annual!
Grab it
Explore the engineering challenges and strategic decisions behind training frontier AI models like Claude in this comprehensive interview featuring Nick Joseph, Anthropic's Head of Pre-training, in conversation with Y Combinator General Partner Ankit Gupta. Discover the technical realities of managing thousands of GPUs, debugging complex infrastructure issues, and balancing computational resources between pre-training and reinforcement learning processes. Learn about scaling laws and how they create feedback loops between compute power, model capabilities, and revenue generation, while examining why infrastructure problems often present greater challenges than machine learning problems in AI development. Understand the evolution from various AI approaches to why next-word prediction became the dominant paradigm, and gain insights into building early infrastructure at Anthropic. Examine efficiency optimization techniques, debugging strategies at massive scale, and the composition of pre-training teams, including the balance between generalists and specialists. Delve into the complexities of distributed training across thousands of GPUs, compare working with different chip architectures including GPUs versus TPUs, and understand the relationship between pre-training and post-training processes like RLHF and reasoning models. Explore future considerations around data quality and availability challenges, and gain perspective on where pre-training technology is headed next, all while learning from Joseph's career journey from Vicarious to OpenAI to Anthropic.
