Streamlining Competitive Data Science at CERN: Running ML Challenges With Kubeflow

This conference talk explores how CERN leverages Kubeflow on Kubernetes to streamline machine learning challenges for particle physics research. Learn how Raulian-Ionut Chiorescu and Hannes Hansen developed an infrastructure solution that overcomes the limitations of popular platforms like Kaggle, which often face resource constraints for scalable model training and aren't suitable for sensitive internal data. Discover their implementation of a pipeline-based framework where data loading, distributed training, and scoring are managed automatically, allowing participants to focus exclusively on model development. The presentation includes a detailed case study of particle physics challenges at CERN, highlighting the setup process and development challenges encountered. This solution demonstrates how cloud-native technologies can effectively support competitive data science while maintaining control over proprietary data and integrating with internal tooling.