François Labrèche - How not to do ML

Problem Statement

François Labrèche of Sophos presents pitfalls in ML-based exploit prediction for vulnerability management. With over 40,000 CVEs yearly and few actually exploited, a classifier could help prioritize patching—but improper feature encoding can silently destroy real-world performance.

Model Design

The team builds a random forest on NVD metadata: LDA topic vectors from CVE descriptions, description length, product and reference counts, CVSS features, publication dates, and online mention counts from Reddit, Pastebin, Telegram, and dark web forums via Flare. Labels indicate whether public exploits exist in Exploit-DB, Packet Storm, or GitHub PoCs.

Encouraging Offline Metrics

Ten-fold cross-validation reports 93% accuracy and 83% recall, suggesting production readiness. The speaker warns such metrics can mislead because encoding mistakes often surface only during live deployment on newly published CVEs.

Lessons on Feature Selection

The talk title frames the presentation as how not to do machine learning: even standard pipelines need careful feature design. Labrèche planned to demonstrate revised modeling and April live predictions, emphasizing validation beyond offline cross-validation scores.