Stanford Seminar - Mixture of Experts Paradigm and the Switch Transformer

This seminar explains the Mixture of Experts approach and the Switch Transformer, including changes to routing, communication costs, and training stability. It describes faster pretraining of T5-based and multilingual language models, including models scaled to a trillion parameters.