Demystifying the Communication Characteristics for Distributed Transformer Models

Learn about the communication patterns and characteristics of distributed transformer models in this 34-minute technical conference presentation from the HOTI Hot Interconnects Symposium. Explore detailed insights from researchers Quentin Anthony, Benjamin Michalowicz, Jacob Hatef, Lang Xu, Mustafa Abduljabbar, Aamir Shafi, Hari Subramoni and Dhabaleswar Panda as they break down the complex communication requirements for running large-scale transformer architectures across distributed systems. Part of Technical Paper Session D focused on optimizing collective operations, this talk chaired by Craig Stunkel of Nvidia provides valuable understanding of the networking and communication challenges involved in training and deploying distributed transformer models.