Divining Profiler Accuracy - An Approach to Approximate Profiler Accuracy Through Machine Code-Level Slowdown

Watch a 15-minute conference presentation from OOPSLA 2025 that introduces a groundbreaking methodology for assessing the accuracy of sampling-based profilers in just-in-time (JIT) compiled programs. Learn about the fundamental challenge developers face when optimizing performance on modern runtime systems, where sampling profilers often provide inaccurate and unreliable results for JIT-compiled applications ranging from mobile browser apps to large-scale server systems. Discover how researchers Humphrey Burchell and Stefan Marr from the University of Kent developed a novel technique to approximate ground truth by precisely slowing down Java programs at the machine-code level while preserving optimization decisions, compilation behavior, and execution patterns on modern CPUs. Explore their experimental findings demonstrating controlled slowdown of benchmarks despite CPU optimization complexities, verified through hardware profiling that confirms basic-block level accuracy for execution-dominating blocks. Examine their comprehensive evaluation of popular profilers including Async-profiler, JFR, JProfiler, and YourKit, revealing how these tools maintain original performance behavior and method runtime percentage assignments under controlled conditions. Understand the identification of accuracy issues caused by missing debug information that prevents correct source code mapping, and review their testing methodology that revealed significant accuracy differences between profilers when approximating ground truth through specific basic block slowdowns. Gain insights into what researchers believe is the first practical methodology for assessing sampling profiler accuracy in JIT-compiling systems, establishing a foundation for future profiler accuracy improvements.