Throwing Darts in the Dark? Detecting Bots with Limited Data using Neural Data Augmentation

### Overview Presented by Steve on behalf of a research collaboration between the University of Illinois, Virginia Tech, and Shape Security, this talk focuses on developing machine learning (ML) models for cybersecurity when labeled data is scarce. Using bot detection as a primary case study—covering threats like account takeover and web scraping—the presentation addresses three major challenges: the scarcity of ground-truth labels, concept drift caused by evolving attacker behavior, and the poor generalizability of highly customized domain features across different web services.

### Key Topics - **Hybrid Bot Detection Strategy:** The architecture prioritizes rule-based detection for high-precision, low-cost coverage of common bots, deploying machine learning specifically to catch advanced bots that evade static rules by mutating their behavior. - **Frequency-Based Feature Encoding:** Replacing raw application-specific features with generic network features (e.g., URL, cookie, browser), converted dynamically into frequency distributions using cumulative distribution functions (CDFs) over a sliding window. - **Neural Data Augmentation:** A specialized dual-generator Generative Adversarial Network (GAN) architecture tailored for security contexts where benign data is representative, but malicious data is scarce, biased, and rapidly shifting.

### Technical Details - **Dataset & Ground Truth:** The research evaluated 23 million network requests collected across three distinct websites (A, B, and C) spanning three months of data over a one-year period. Ground-truth labels were primarily established via user CAPTCHA interactions. - **Feature Processing:** To avoid raw string dependencies and generalize across services, features are encoded based on frequency percentiles in an empirical CDF. A daily sliding window updates the distribution dynamically to handle unseen items and concept drift in real time. - **Data Synthesis Architecture:** Standard GANs merely mimic existing samples, which fails against adapting bots. The proposed method splits malicious samples into clusters and outliers using DBSCAN: 1. *Cluster Data Synthesis:* Synthesizes data near existing bot clusters by expanding boundaries cautiously, becoming less aggressive near benign data regions. 2. *Outlier Data Synthesis:* Aggressively synthesizes data outward in the complementary space outside benign regions to anticipate future bot variants. 3. *Discriminator Role:* Functions as the downstream classifier trained to differentiate benign users from both real and synthetic malicious bots.

### Takeaways - **Rule and ML Complementarity:** Rule-based detection captured roughly 80% of bots, while adding the ML pipeline boosted detection to over 90%. - **Robustness Under Extreme Label Scarcity:** When trained on just 1% of labeled data, the proposed method achieved F1 scores comparable to an LSTM trained on 100% of labeled data across evaluated services. - **Mitigating Long-Term Model Decay:** Evaluating a model trained in August 2018 on data from January and September 2019 demonstrated that periodic model retraining with only 1% of new data per month effectively restores detection performance.