CONFidence 2014: Scaling Security

### Overview At CONFidence 2014, members of the Prezi security team—Mishi, Robi, and Attila—presented practical approaches to scaling security in a fast-growing startup environment. Operating with over 40 million users, more than 400 AWS instances, over 400 Git repositories, and 15+ programming languages, Prezi deploys software over 100 times per week with more than 80 engineers. With only a three-person security team, traditional blocking security controls (such as mandatory pre-deployment sign-offs or blocking code reviews) are impossible. Instead, the team prioritizes influence, culture, non-blocking monitoring, automated alerting, and iterative learning.

### Key Topics - **DevOps Culture & Agile Security:** Integrating security into full-stack engineering where developers own code and infrastructure. Critical incidents trigger a 'Prior 1' mode where developers rally to fix issues within 30 minutes. - **Static Code Analysis with RepoGuard:** A custom Python-based tool that fetches Git repositories and applies regular expressions to detect risky code changes without blocking deployments. - **Infrastructure Auditing with Red Alert:** Built around Netflix’s open-source tool Edda (and libraries like Boto), Red Alert tracks near-real-time changes across AWS environments and triggers alerts on risky infrastructure modifications. - **Logging and Fast Search:** Transitioning from slow Hadoop queries to Elasticsearch for near-real-time log indexing and correlation, complemented by host-level monitoring like auditd. - **Bug Bounty Program & Feedback Loops:** Leveraging external security researchers to identify real-world attack vectors, followed by blameless post-mortem investigations that produce new detection rules.

### Technical Details - **RepoGuard:** Because market static analysis tools did not support Prezi's 15+ diverse languages, the team developed RepoGuard. It runs pattern-matching regexes across code commits and chef repositories (e.g., detecting newly exposed virtual hosts or insecure ORM usage such as Django's `get_by_secondary_key`). Although initially generating overwhelming false positives, the rule set was refined to over 70 rules, yielding roughly 20 manageable alerts per day. - **Red Alert & AWS Tracking:** Netflix's Edda polls AWS APIs and stores environment state snapshots in a local database, allowing diffing without incurring heavy AWS API costs. Prezi built Red Alert as an extensible plugin framework on top of Edda and Boto. Red Alert monitors security groups (e.g., catching accidentally exposed ports like 9080/TCP), elastic load balancers, and user creation. The team open-sourced Red Alert at the conference. - **Vulnerability Insights & Bug Bounty:** Prezi instituted a bug bounty program that generated nearly 1,000 unique submissions and over 60 paid issues in its first eight months, requiring around five hours weekly just for email triage. Notable findings included an SSRF/local file read in an offline conversion service where an external researcher bypassed theoretical defenses by querying the AWS link-local metadata service to extract bootstrap secrets. Subsequent fixes were subjected to further community review, exposing subtle issues like DNS rebinding. - **Post-Mortem Process:** Following critical outages or security findings, teams produce detailed post-mortem documents and hold open review meetings to implement multi-layered prevention rules rather than one-off patches.

### Takeaways Defenders often theorize about attack vectors that differ from what attackers actually exploit. Rather than relying on generic out-of-the-box tools (such as unmodified Suricata), organizations should tailor detection systems to their specific architecture. By adopting non-blocking automated detectors like RepoGuard and Red Alert, fostering open communication with developers, and actively learning from external researchers via bug bounties, a small security team can effectively protect an agile, cloud-native enterprise.