How Not to Do ML - Common Pitfalls in Cybersecurity Threat Prediction Models

Learn about common pitfalls in applying machine learning to cybersecurity threat prediction through this 26-minute conference talk from NorthSec 2025. Explore the challenges of building predictive models for security vulnerability exploitation and exploit publication using publicly available data from online discussions and CVE databases. Discover why many published ML approaches that perform well on historical data fail dramatically when deployed in real-world environments. Examine four critical mistakes in feature encoding and evaluation that cause models to make incorrect predictions in live systems, including issues with temporal data evolution and feature availability timing. Understand how vulnerability and threat information changes over time and why data often isn't available on a vulnerability's publication day. See practical demonstrations of implementing and deploying various ML methods, comparing their historical performance against real-world effectiveness. Gain insights into how models with lower historical evaluation scores can actually outperform seemingly better models in live settings when features are encoded correctly for operational cybersecurity environments.