How Replit Trained Their Own LLMs - LLM Bootcamp

Explore the comprehensive process of training custom Large Language Models (LLMs) in this 32-minute conference talk by Reza Shabani from Replit. Gain insights into the entire workflow, from data processing to deployment, including the modern LLM stack, data pipelines using Databricks and Hugging Face, preprocessing techniques, tokenizer training, and running training with MosaicML and Weights & Biases. Learn about testing and evaluation methods using HumanEval and Hugging Face, as well as deployment strategies involving FasterTransformer, Triton Server, and Kubernetes. Discover valuable lessons on data-centrism, evaluation, and collaboration, and understand the qualities that make an effective LLM engineer.