Training on the Test Set and Other Heresies

Explore the controversial topic of training on test sets and other unconventional practices in machine learning through this thought-provoking lecture by Ben Recht from UC Berkeley. Delve into fundamental concepts of machine learning and generalization, challenging conventional wisdom in the field. Examine critical aspects such as regularization, random features, regression, boosting, and model size. Analyze the implications of diminishing returns, new holdout and test sets, and the variability in datasets like ImageNet. Gain insights into the complexities of data collection, including discussions on Mechanical Turk and Kaggle competitions. Question established norms and consider alternative approaches to improve machine learning practices and outcomes.