Avoiding the Ordering Trap in Systems Performance Measurement

Explore the critical issue of ordering effects in systems performance measurement through this 17-minute conference talk from USENIX ATC '23. Learn why the common assumption of trial independence in performance studies can lead to biased results and incorrect conclusions. Discover the methodology developed for studying ordering effects, including statistical tests for order dependence. Examine findings from a large-scale empirical study involving nearly 2.3 million measurements from over 1,700 servers collected over 9 months. Gain insights into OrderSage, a tool designed to detect and mitigate ordering effects, and review case studies that reveal previously unknown ordering effects in published research. Understand the importance of considering experiment ordering in performance studies to avoid hidden traps and ensure more accurate results in systems research.