Emergence and Scaling Laws for SGD Learning and Learning Compositional Functions with Transformers

This talk by Jason Lee from Princeton University explores two key topics in deep learning theory: the emergence and scaling laws for stochastic gradient descent (SGD) learning, and how transformers learn compositional functions. Presented at the Simons Institute as part of their Deep Learning Theory program, the lecture delves into mathematical frameworks that explain how neural networks learn and scale, with particular focus on transformer architectures that have revolutionized natural language processing and other domains.