Mitigating the Curse of Detail - A Heuristic for Feature Learning and Sample Complexity

Explore a theoretical framework for understanding deep learning through statistical mechanics in this Hebrew-language lecture by Prof. Zohar Ringel from the Hebrew University of Jerusalem. Learn about a novel heuristic approach that addresses the "curse of detail" in deep learning theory, where exact analytical predictions become computationally intractable due to their dependence on specific architectural and data characteristics. Discover how scaling arguments can replace exact solutions to predict the emergence of distinct feature learning patterns across different data and network width scales. Examine the application of this variational approach to reproduce known scaling exponents for two-layer networks and its extension to more complex architectures including three-layer non-linear networks and softmax attention blocks. Gain insights into how statistical mechanics and field theory principles can be applied to deep learning problems, drawing from Prof. Ringel's expertise in topological phases of matter, disordered systems, and computational physics. The lecture is presented as part of the HUJI Machine Learning Club series and provides a first-principles theoretical perspective on feature learning and sample complexity without requiring intensive numerical computations.