Weak-to-Strong Generalization Even in Random Feature Networks, Provably

A lecture by Marko Medvedev from the University of Chicago exploring the phenomenon of Weak-to-Strong Generalization in the context of random feature networks. Discover how a strong student model can learn from a weak teacher model and significantly outperform it, even in simple random feature networks rather than complex models like GPT-4. Learn about the mathematical proofs demonstrating how a student with a larger number of random features can exceed the performance of a teacher with fewer features, despite being trained only on data labeled by that teacher. Understand the role of early stopping in enabling this generalization phenomenon and examine the quantitative limitations of weak-to-strong generalization within this model. This one-hour talk, part of the Deep Learning Theory series at the Simons Institute, presents joint research work with Kaifeng Lyu, Dingli Yu, Sanjeev Arora, Zhiyuan Li, and Nathan Srebro.