Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally in Neural Architecture Design

Explore a technical talk that delves into a novel approach for optimizing neural network architectures during the training process. Learn how to identify and address expressivity bottlenecks in machine learning tasks through dynamic architecture adaptation, rather than relying on fixed architectures with pre-determined parameters. Discover a mathematical framework for detecting and quantifying these bottlenecks, enabling the strategic addition of neurons to improve network performance. Understand how this innovative method challenges the conventional wisdom of starting with large networks, instead demonstrating how to effectively grow networks from minimal initial configurations. The presentation, delivered at the Erwin Schrödinger International Institute's Thematic Programme on "Infinite-dimensional Geometry," offers valuable insights into more efficient and adaptable approaches to neural network development and optimization.