The Elusive Ground Truth - An Ongoing Journey in Data Labeling

Explore the critical yet underappreciated stage of data labeling in machine learning through a 39-minute conference talk that chronicles the evolution of building a data labeling process from scratch. Follow the journey from single-person annotation to outsourcing and ultimately to a hybrid model, discovering how each iteration unveiled new complexities around inherent subjectivity and hidden biases that challenge label consistency even among domain experts. Learn about the practical challenges of managing evolving label schemas and confronting the human subjectivity that fundamentally defines the labeling process. Gain insights into how acknowledging these inherent challenges and the potentially elusive nature of ground truth can lead to more robust machine learning systems, moving beyond purely technical considerations to understand the human elements that shape data quality and model performance.