Benchmarking is Hard - A Hermetically Sealed Container for CMS Workload Benchmarking

Learn about a revolutionary approach to standardizing CMS workload benchmarking through hermetically sealed containers in this conference talk from the Open Compute Project. Discover how Grant Mackey from Jackrabbit Labs and Vikrant Soman from Uber address the critical challenges of ensuring fair and repeatable workload deployment in technology benchmarking, particularly for CXL memory systems. Explore the development of an open-source containerized solution that eliminates deployment complexity by packaging all necessary software and dependencies into a "look but don't touch" environment. Understand how this standardization approach enables consistent benchmark execution across different operating systems and hardware platforms while requiring minimal user support. Examine the container's configurable input parameters and standardized output formats that facilitate confident performance comparisons across various system configurations. See practical demonstrations of initial workload containers including STREAM and MLC benchmarks running on diverse CXL memory devices from multiple vendors and systems. Gain insights into parsing and analyzing the resulting JSON output data for conducting fair system comparisons, ultimately supporting better TCO and ROI analysis decisions in enterprise technology deployments.