How Neural Networks Learn Features from Data

Explore the fundamental mechanisms of feature learning in neural networks through this insightful lecture presented at IPAM's Theory and Practice of Deep Learning Workshop. Delve into the unifying concept of the average gradient outer product (AGOP) and its role in capturing relevant patterns across various network architectures, including convolutional networks and large language models. Discover the Recursive Feature Machine (RFM) algorithm and its ability to identify sparse subsets of features crucial for prediction. Gain a deeper understanding of how neural networks extract features from data, connecting this process to classical sparse recovery and low-rank matrix factorization algorithms. Uncover the implications of this research for developing more interpretable and effective models in scientific applications, advancing the reliable use of neural networks in technological and scientific fields.