Mehler's Formula, Branching Processes, and Compositional Kernels of Deep Neural Networks

Explore the intricate connections between compositional kernels, branching processes, and deep neural networks in this 53-minute seminar presented by VinAI. Delve into Hai Tran-Bach's research, which utilizes Mehler's formula to provide novel insights into the mathematical role of activation functions in neural networks. Examine the unscaled and rescaled limits of compositional kernels, investigating their behavior as compositional depth increases. Analyze the memorization capacity of compositional kernels and neural networks, focusing on the interplay between compositional depth, sample size, dimensionality, and activation non-linearity. Discover explicit formulas for eigenvalues of compositional kernels, quantifying the complexity of corresponding Reproducing Kernel Hilbert Spaces (RKHS). Learn about a new random features algorithm that compresses compositional layers through an innovative activation function. Gain valuable insights into the mathematical foundations of deep learning and their implications for developing more principled machine learning algorithms.