Matrix Calculus for Machine Learning and Beyond - IAP 2023

Explore advanced mathematical techniques essential for modern machine learning and optimization through this comprehensive course that bridges the gap between traditional calculus and matrix-based computations. Master matrix calculus by learning to think of matrices holistically rather than as arrays of scalars, and discover how to generalize and compute derivatives of complex matrix factorizations and operations. Develop skills in vectorization of matrix functions, Kronecker products, and Jacobians while understanding how differentiation formulas must be reimagined for large-scale computing applications. Delve into gradients and inner products in various vector spaces, nonlinear root finding, and adjoint gradient methods essential for optimization problems. Study the derivatives of matrix determinants and inverses, forward automatic differentiation using dual numbers, and differentiation on computational graphs. Examine advanced topics including adjoint differentiation of ODE solutions, calculus of variations, derivatives of random functions, second derivatives and Hessian matrices, derivatives of eigenproblems, and automatic differentiation techniques. Gain practical understanding of how these mathematical foundations support machine learning algorithms, large-scale optimization, and other modern computational applications through lectures taught by MIT faculty Alan Edelman and Steven G. Johnson.