411: A Framework for Managing Security Alerts

Architecture and Background

Modern web applications generate a ton of logs. Suites like ELK (Elasticsearch, Logstash, Kibana) exist to help manage these logs, and more people are turning to them for their log analysis needs. These logs contain a treasure trove of information regarding bad actors on your site, but surfacing that information in a timely manner can be difficult. When Etsy moved over from Splunk to ELK in mid-2014, we realized that ELK lacked necessary functionality for real-time alerting. We needed a solution that would provide a robust means of querying ELK and enrich the data with additional context. We ended up creating our own framework to give us this functionality.

Vulnerability Analysis

We’ve named this open-source framework 411. We designed 411 as a solution for detecting and alerting on interesting anomalies and security events. The Security team at Etsy was interested in using this functionality to detect everything from XSS to monitoring for potential account compromises. First, we’ll start off with a discussion of what you should be logging into Elasticsearch. This is important to help you create useful, actionable alerts in 411. We’ll note a number of configuration tips and tricks to help you get the most out of your ELK cluster.

Exploitation and Demonstrations

From there, we’ll dive into 411’s features and how it allows the Etsy security team to work effectively. We’ll conclude with two demos of 411 in action. This presentation will show you several examples of useful searches you can build in 411 and how this data can be manipulated to generate clear, actionable alerts. We’ll demonstrate the built-in workflow for responding to alerts and how 411 allows you to pull up additional context as you work on an alert. Additionally, while much of our discussion will be centered around ELK, 411 can in fact be used with a variety of data sources (Several of these sources are built into 411). Whether you’re a newbie looking to learn more or a security veteran with an established system, 411 will help change the way you handle security alerts.

Remediation and Defensive Strategies

Kai is a security engineer at Etsy. At work, he fiddles around with security features, works on 411 and responds to the occasional bug bounty report. He went to NYU-Poly and got a degree in Computer Science, with a MS in Computer Security. In his free time, he enjoys reverse engineering, CTFs board games, starting yet another project that he’ll never finish and learning all the things. Kenneth Lee is a senior product security engineer at Etsy.com, working on everything from managing the bug bounty program to shattering the site with new vulnerabilities. Previously, Kenneth worked at FactSet Research Systems preventing The Hackers from stealing financial data. He went to Columbia and got an MS in computer science focusing on computer security. Between sweet hacks, Kenneth enjoys drinking tea and force feeding Etsy's operations team with Japanese chocolates.