Building RedPajama - Open-Source LLM Development

Explore the process of building open-source large language models (LLMs) in this 28-minute conference talk by Vipul Ved Prakash, co-founder and CEO of Together. Delve into the challenges and key aspects of creating new LLMs, including broad training datasets, strong base models, well-aligned instruction datasets, moderation subsystems, and cost-effective infrastructure. Learn about RedPajama, an open-source initiative led by Together and collaborators, which aims to harness community power for building and improving LLMs. Gain insights from Prakash's extensive experience in tech innovation, including his work with social media search, anti-spam technology, and AI/ML systems at Apple.