A Function Space View of Overparameterized Neural Networks - Rebecca Willet, University of Chicago

Explore a function space perspective on overparameterized neural networks in this 35-minute conference talk by Rebecca Willet from the University of Chicago. Delve into the intriguing phenomenon of vastly overparameterized neural networks generalizing well despite their capacity to fit any labels. Examine the role of weight magnitude control in complexity regulation and investigate the functions approximated by neural networks with bounded weight norms. Discover a precise characterization of functions realizable by two-layer ReLU networks with bounded Euclidean norm weights, drawing surprising connections to the Radon transform used in computational imaging. Learn how Radon transform analysis provides novel insights into learning with two and three-layer ReLU networks. Gain understanding of topics such as infinite-width ReLU nets, norm-controlled learning, convex nets, and depth separation results. Conclude with open questions in this cutting-edge area of machine learning research presented at the Alan Turing Institute.

Syllabus

Intro
Overparameterized models in machine learning
An experiment
Training overparameterized neural nets
Approximation theory perspective
Infinite-width two-layer ReLU nets
Learning with norm-controlled infinite-width ReLU networks
From Two-layer ReLU Nets to Convex Nets
Intuition in 1D
Intuition in Higher Dimensions
The Radon Transform in 2D
Radon Transform as Line Detector
Key Derivation
Example
Implications: Comparison to Kernel Learning
Implications: Depth Separation Result
Open Questions