Completed
0:00 Train an LLM from scratch with $100 Nanochat by Karpathy
Class Central Classrooms beta
YouTube videos curated by Class Central.
Classroom Contents
Train an LLM from Scratch with Karpathy's Nanochat
Automatically move to the next video in the Classroom when playback concludes
- 1 0:00 Train an LLM from scratch with $100 Nanochat by Karpathy
- 2 1:44 One-click Runpod Template: https://console.runpod.io/deploy?template=ikas3s2cii&ref=jmfkcdio
- 3 4:46 Running speedrun.sh script with wandb
- 4 7:55 Tokenizer Training Byte Pair Encoding, BPE
- 5 10:52 Pretraining on 10B tokens of fine-web edu
- 6 15:15 Pre-training evaluation and Mid-training on Smoltalk+
- 7 17:33 Instructions on pushing and pulling checkpoints to / from HuggingFace Hub
- 8 18:49 Instructions for running the chat interface / UI
- 9 21:15 Post-training on benchmark training/aux splits
- 10 23:11 Demo of running the chat interface
- 11 25:29 Pushing the model, optimizer, report and tokenizer to hub
- 12 28:20 Resources: Model and Repo
