How I Learned to Stop Worrying and Love Evaluations (and Keep Worrying)

Explore the impact of coordinated evaluations in natural language processing research through this insightful talk by Jonathan May. Delve into the world of bake-offs, shared tasks, and evaluations, examining their role in exposing algorithms and models to unseen data. Discover how these high-stress periods, often criticized for metrics, procedures, and score-chasing, can actually benefit NLP research and lead to significant accomplishments. Gain valuable insights into recent evaluation-grounded work, including rapid generation of translation and information extraction for low-resource surprise languages (DARPA LORELEI) and the organization of SemEval shared tasks in semantic parsing and generation. Learn from May's extensive experience as a Research Assistant Professor at USC's Information Sciences Institute and his previous roles at SDL Research and Raytheon BBN Technologies.