Completed
27:35 – Efficiency hacks and debugging at scale
Class Central Classrooms beta
YouTube videos curated by Class Central.
Classroom Contents
Scaling Laws, Compute, and the Future of AI - Engineering Challenges Behind Training Frontier Models
Automatically move to the next video in the Classroom when playback concludes
- 1 00:00 – Introduction
- 2 01:05 – From Vicarious to OpenAI to Anthropic
- 3 06:40 – What pretraining is
- 4 11:20 – Why next-word prediction won out
- 5 16:05 – Scaling laws and the feedback loop of compute → models → revenue
- 6 21:50 – Building Anthropic’s early infrastructure
- 7 27:35 – Efficiency hacks and debugging at scale
- 8 33:10 – Generalists vs. specialists on the pretraining team
- 9 38:45 – Challenges of training across thousands of GPUs
- 10 44:15 – Working with new chips: GPUs vs. TPUs
- 11 49:00 – Pretraining vs. post-training RLHF and reasoning models
- 12 54:25 – The future of data quality and availability
- 13 59:10 – Where pretraining goes next
- 14 1:03:00 – Closing reflections
