Global Minimizers of Sigmoid Contrastive Loss

Explore the theoretical foundations of sigmoid contrastive loss in this 49-minute conference talk from Harvard CMSA's Workshop on Mathematical Foundations of AI. Delve into the mathematical analysis of contrastive pre-training methods, particularly focusing on the advantages of synchronizing with trainable inverse temperature and bias under sigmoid loss as implemented in Google DeepMind's SigLIP models. Learn about (m,b)-Constellations, a novel combinatorial object related to spherical codes that are parametrized by margin m and relative bias b, and discover how temperature and bias can drive the loss function to zero for this rich class of configurations. Understand the theoretical justification for SigLIP's success in retrieval tasks, gain insights into the modality gap present in SigLIP, and examine the necessary dimensions required for producing high-quality representations. The presentation also introduces a reparameterization of the sigmoid loss with explicit relative bias that appears to improve training dynamics, providing valuable insights for researchers working on representation learning and contrastive methods in machine learning.