How We Can Power Real Security Machine Learning Progress Through Open Algorithms and Benchmarks

Explore a thought-provoking conference talk that challenges the current state of machine learning in cybersecurity. Delve into the speaker's argument for fostering a culture of research transparency and openness in security ML, similar to other subfields like cryptography. Examine the contrast between recent breakthroughs in language, vision, and robotics ML and the slower progress in security ML due to unverifiable product claims and misleading marketing. Learn about the importance of open algorithms and benchmarks in driving real progress in the field. Discover the potential benefits and challenges of implementing such openness, including concerns about weaponization of benchmark malware and protecting personal information. Gain insights into the need for comprehensive benchmark coverage across critical detection problems in cybersecurity and the ongoing efforts to address these issues within the industry.

Syllabus

Intro
What does science look like?
Security data sciences Are we doing science?
What you can do when you have openness and benchmarks
Outside of security, ML benchmarks have told the story and made the story
Cybersecurity Al vs. vision and language Al Obscurantism and hype vs. openness and progress
Security ML openness issues
Benchmark malware could be weaponized by adversaries
Protecting personal information
The risks of exposing too much to adversaries
We need benchmark coverage over the most important detection problems in cybersecurity
There are actors in the industry working on solutions, but we need more