The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

This conference talk explores the "Semantic Hub Hypothesis" which proposes that language models develop shared representation spaces across different languages and modalities. Learn how modern language models place semantically similar inputs near each other in their representation space, regardless of language or modality differences. Discover research showing that model representations for semantically equivalent inputs in different languages exhibit similarities in intermediate layers, and how this representation space can be interpreted using the model's dominant pretraining language. The presentation extends this concept to other data types including arithmetic expressions, code, and visual/audio inputs, demonstrating that interventions in the shared representation space of one data type predictably affect outputs in other data types. Presented by Zhaofeng Wu, a third-year NLP PhD student at MIT working with Professor Yoon Kim, whose research focuses on model evaluation, analysis, interpretability, and multilingualism.