AppSecCali 2019 - (in)Secure Development - Why some product teams are great and others … aren’t...

Overview Ken Hendrix, a Dutch development manager in Riot Games' security team, shares how his organization discovered why some product teams excel at security while others struggle. Riot Games, founded in 2006 and grown from 40–50 to over 3,500 employees, operates a highly autonomous engineering culture where every team maintains complete independence in how they build and organize their work. This autonomy, while powerful for product delivery, creates a core security challenge: traditional top-down mandates and gatekeeping fail because teams simply refuse to adopt them.

Key Topics Maturity Model: Through observation of 50+ teams across 2017–2018, Hendrix identified a four-level maturity spectrum for security practices. Level 1 teams have limited security awareness and don't discuss it in planning. Level 2 teams treat security as a reactive pre-release phase, dependent on a single champion (e.g., a tech lead) who advocates for it—making security unsustainable when that person leaves. Level 3 teams integrate security into nearly every conversation and formal agenda, becoming a sustainable practice across the whole team. Level 4 teams practice security "like breathing," automatically and without prompting, typically in security-sensitive domains like authentication or payment.

Research Findings: By end of 2017, most teams clustered in Levels 1–2, with significant gaps before Level 3–4. Bug bounty data proved impact: a Level 4 team paid 55 cents per bounty dollar compared to Level 1's $1.00, and fixed high-risk issues in one-third the time. Security review data confirmed the pattern: Level 4 issues had severity valued at 58 cents versus Level 1's $1.00.

The Lightweight Solution: Rather than imposing heavy threat modeling (e.g., STRIDE taking 30+ minutes per sprint) or lengthy security checklists, Hendrix's team introduced a single security question per sprint story: "How can a malicious user intentionally abuse this functionality and how can we prevent that?" A brief 2–3 minute conversation ensures the team thinks about security without the overhead. The approach prioritizes scalability (not requiring security staff in every meeting) over perfection.

Practical Example: The Battlegrounds tournament-organizing tool team (5 engineers, designer, QA, product owner) applied this during sprint planning. One story about mass CSV tournament uploads surfaced the need for file validation and logging. Another about a tournament-help button revealed a denial-of-service vector when a team member asked what happens if someone clicks rapidly—prompting the product owner (non-technical) to raise the security question. Both conversations led to secure design choices and helped the team move from Level 2 to Level 3.

Technical Details Implementation: The security question is integrated into the existing agile workflow (JIRA, Trello, etc.) rather than layered on top. Teams typically need 4–5 weeks to develop the "muscle" of thinking like an attacker. The process is intentionally imperfect—the security team acknowledges they could identify 20+ additional issues in a typical story but don't intervene, allowing the team's own conversation to drive lightweight solutions.

Challenges: Riot's decentralized model (20+ offices, 20+ programming languages, multiple frameworks) makes centralized code analysis difficult. Teams dropping in maturity occur, especially when a security champion or advocate leaves. Formal security training has struggled to be relevant and valuable across diverse engineering roles.

Metrics: By end of 2018, progress was observed with teams moving from Level 1→2 and Level 2→3. However, advancement from Level 3→4 is a multi-year journey with diminishing returns. The leadership in Level 4 teams consistently prioritized security, integrity, and trust—sometimes delaying releases for better security over additional player features.

Takeaways Riot's core lesson is that security interventions must align with how teams actually work and operate. Preaching to teams already good at security moves the needle nowhere; focusing on teams at Level 1–2 is where impact happens. Lightweight, conversation-based tools that fit into existing agile workflows are far more adoptable than heavyweight processes. Security should not be a pre-release gate but an ongoing part of team thinking. Understanding team maturity and culture is essential before imposing solutions; understanding why teams reject mandates enables better engagement. Finally, observable, data-driven metrics (bug payouts, fix times, severity) help persuade teams and stakeholders that security practices yield tangible value.