Explore the limitations of the training/test experimental paradigm in machine learning and delve into crucial aspects of building effective machine learning systems in this lecture by Kyunghyun Cho from New York University. Examine three key areas: model assumption and construction, optimization, and inference. Learn about generative multitask learning, incidental correlation in multimodal learning, and systematic approaches to studying learning trajectories. Discover the consistencies that large-scale language models must satisfy and why most current models fall short. Gain insights from a distinguished expert in computer science, data science, and machine learning as he challenges conventional thinking and proposes new directions for research beyond test accuracies.