BSidesSF 2017 - Make Alerts Great Again (Daniel Popescu)

### Overview At BSidesSF 2017, Daniel Popescu, a security engineer at Yelp, presented the strategies and internal tools Yelp developed to overhaul their security alerting pipeline. Operating at a scale of over 100 million monthly active mobile users and hundreds of microservices, Yelp collects logs across servers, laptops, and services into data stores such as Splunk and Elasticsearch (utilizing Yelp's open-sourced ElastAlert tool). Over time, the security team faced severe alerting pitfalls: lack of visibility into alert volume, non-actionable email notifications, lack of authoring standards, unverified alert functionality, and false positives. Popescu explains how Yelp addressed these issues through visibility reporting, ticketing automation, playbooks, and testing.

### Key Topics - **Alert Visibility**: Yelp created "Alert Reporter," a standalone Python service that queries systems like Splunk and ElastAlert weekly. It compiles metrics into a spreadsheet showing total alert counts (e.g., 173 alerts at the time), firing frequencies, and notification channels to identify noisy alerts and false positives. - **Actionability and Eliminating Email**: Alerts delivered by email suffered from a lack of ownership, filters, and no accountability. Even ticketing systems (Jira) failed if tickets languished unmonitored. Yelp migrated entirely away from email alerts to Jira Service Desk, leveraging queues, visible countdown timers, and configurable SLAs linked via JQL. - **Actionable Alerting Service**: Yelp engineered a stateless service to enforce action on Jira tickets without maintaining its own database. By inspecting ticket metadata, it automatically assigns owners and escalates unacknowledged or overdue tickets: - *Self-Service Alerts*: Automatically routes alerts triggered by routine administrative tasks (e.g., AWS infrastructure changes, Duo integrations) directly to the acting user via Active Directory lookup, removing the security team from the critical path while retroactively auditing acknowledgments. - *Service Accounts*: Automatically identifies non-human service accounts and pings the owning team across Jira, IRC, or Slack. - *On-Call and Alert Owners*: Assigns urgent issues (such as suspicious logins) by querying PagerDuty schedule names (e.g., malware on-point), falling back to the alert's original author if no specific owner is found. - *Automated Escalation*: Nagging users in IRC and Jira when tickets breach SLA, eventually CCing the user's manager automatically. - **Standardization and Alert Playbooks**: Yelp authored an alerting playbook/runbook that acts as a manifesto defining priority levels (P0 vs. lower), mandating required fields (like owner metadata), guiding appropriate granularity (e.g., higher priority when scope of access increases vs. decreases), and establishing when alerts should page or create Jira tickets. - **Functional Testing and Pipeline Health**: Alerts frequently suffer from broken queries, syntax typos, or missing expected fields. Yelp mandates end-to-end testing of every alert before considering it complete (e.g., triggering a mock reverse shell or data exfiltration to verify firing). Yelp also established "Flatline" alerts to detect sudden drops or complete cessations of log ingestion.

### Technical Details - **Data Stores & Alerting Engines**: Splunk (native alerting) and Elasticsearch with ElastAlert. - **Automation Layer**: Alert Reporter (Python service aggregating weekly stats), Jira Service Desk (SLA enforcement, queues), and the Actionable Alerting Service (stateless Python service inspecting metadata, querying Active Directory, PagerDuty, and notifying via Jira/IRC/Slack).

### Takeaways To eliminate alert fatigue and ensure incident handling: (1) Make alerts actionable by abandoning email in favor of trackable tickets with clear human or self-service owners and SLA-based escalation; (2) Maintain visibility over alerting metrics to prune noisy rules; (3) Mandate rigorous end-to-end testing and flatline monitoring to catch broken queries and dead log pipelines.