Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Explore a 15-minute conference presentation that critically examines and advances flaky test classification in software development. Discover how researchers from the University of Texas at Austin and Cornell University identified significant flaws in prior approaches to classifying flaky tests—tests that pass and fail non-deterministically on the same code—and developed improved solutions. Learn about the experimental design issues and dataset misrepresentations that led previous fine-tuned large language models to overestimate their classification accuracy, with F1-scores dropping from 81.82% to 56.62% when proper methodology was applied. Understand the development of FlakeBench, a more realistic dataset for evaluating flaky test classifiers, and examine FlakyLens, a new training strategy that achieves 65.79% F1-score performance. Compare the effectiveness of specialized models versus general-purpose LLMs like CodeLlama and DeepSeekCoder on this specialized classification task. Investigate token-level attribution analysis that reveals which code elements influence model predictions and explore adversarial perturbation experiments that demonstrate how classification accuracy can change by up to 18.37 percentage points when important tokens are modified. Gain insights into the limitations of current models in generalizing beyond training data and their tendency to rely on category-specific tokens rather than semantic understanding, highlighting the need for more robust training methodologies in automated software testing.
