Efficient Inference of Extremely Large Transformer Models

Explore the challenges and solutions for efficient inference of massive transformer-based language models in this 28-minute Toronto Machine Learning Series (TMLS) talk. Dive into the world of multi-billion-parameter models and learn how they are optimized for production environments. Discover key techniques for making these behemoth models faster, smaller, and more cost-effective, including model compression, efficient attention mechanisms, and optimal model parallelism strategies. Gain insights from Bharat Venkitesh, Senior Machine Learning Engineer at Cohere, as he discusses the establishment of the inference tech stack and the latest advancements in handling extremely large transformer models.