BSidesSF 2018 - Data Driven Bug Bounty (Arkadiy Tetelman)

### Overview In this BSidesSF 2018 presentation, Arkadiy Tetelman, an application security engineer with experience running bug bounty programs at Twitter and Airbnb, presents the case for a data-driven bug bounty approach. Tetelman argues that collecting and analyzing bug bounty metrics provides as much value as resolving the individual vulnerabilities themselves. Given limited security resources, organizations must use empirical data to prioritize application security initiatives, measure return on investment (ROI), manage internal remediation performance, and maintain healthy relationships with external security researchers.

### Key Topics - **Program Logistics and Setup**: Tetelman compares the structures used at Twitter and Airbnb. Twitter launched in May 2014 with an initial unpaid public soft launch to tune processes before switching to a paid public program, using NCC Group for frontline triage, an internal rotation of four to six AppSec engineers, and paying roughly $1M across 850 resolved reports over four years. Airbnb operated dual programs (a public unpaid and private paid program) before merging them into a single public paid program in March 2018, utilizing HackerOne managed triage, a two-week AppSec rotation, and paying $430,000 across 430 resolved reports over three years. - **Taxonomy and Tracking in JIRA**: Gaining actionable insights requires a meticulous tagging taxonomy covering vulnerability class, source, product component, owning team, and SLA status. Teams can track these via custom fields in JIRA and visualize them with tools like Apache Superset. - **Security Roadmap Planning & ROI**: Analyzing vulnerability classes (such as cross-site scripting, sensitive data exposure, and security misconfigurations) directly informs quarterly planning. When investments are made (such as static analysis, templating layer output encoding, or automated scanning), subsequent quarter metrics show whether vulnerabilities actually decreased. - **Driving Accountability and Culture**: Data on resolution times shows significant variance (up to a 10x difference between the fastest and slowest engineering teams). Rather than blaming slower teams, AppSec should use data constructively to identify root causes like resourcing or prioritization, while also celebrating successes (e.g., throwing a cupcake party for a team reaching zero issues past SLA).

### Technical Details - **Metrics Collected**: Metrics include vulnerability counts by class (e.g., CSRF, SSRF, XSS, open redirects), time to fix by team, open tickets past SLA, subteam breakdowns by severity (critical, high, medium, low), and report intake source distribution. - **Report Source Breakdown**: In a featured 90-day dataset, approximately 50% of incoming vulnerabilities came via HackerOne, ~35% from automated scanners, followed by third-party and internal penetration tests, with employee reports being the smallest volume. - **Researcher Experience Metrics**: Public metrics such as first response time and time to bounty are critical. Researchers prioritize prompt communication over payout sizes; delays cause negative word-of-mouth on researcher Slack channels, whereas good relationships yield early vulnerability heads-ups and collaboration on public disclosure timelines.

### Takeaways - **Start Small and Scale Slowly**: Do not jump straight into a public paid program. Begin with an internal assessment or penetration test to understand weak spots, establish scope, launch as a private program with a limited researcher pool, refine triage workflows, and only transition to public when processes are mature and report volume in private tests begins to plateau. - **Maintain High Visibility**: Share clear, high-level vulnerability trend graphs across the organization, in newsletters, and with executive leadership to demonstrate AppSec ROI and build social capital for future security initiatives.