JDD 2016: A needle in a logstack (Tomasz Gągor, Paweł Torbus)

### Overview In this talk from JDD 2016, Tomasz Gągor (system administrator at Interia) and Paweł Torbus (system architect) present practical lessons from implementing and maintaining the ELK stack (Elasticsearch, Logstash, Kibana) across their infrastructure. They walk through the real-world operational challenges encountered with each component, tuning recommendations, and concrete use cases where log analysis and visualization in Kibana uncovered production bugs, performance degradations, and configuration oversights.

### Key Topics - **Architecture of the ELK Stack:** Deploying Logstash (log preprocessor running on JRuby), Elasticsearch (distributed storage, indexing, and search engine based on Lucene), and Kibana (visualization front end), along with ecosystem tools like Filebeat, Logstash Forwarder, Shield (access protection), and Watcher (alerting). - **Log Ingestion & Logstash Pitfalls:** Handling multi-line logs (e.g., XML, Java stack traces), thread bottlenecks, event timestamp extraction, avoiding variable collisions in Logstash (e.g., overriding the `message` field), and moving from syslog UDP to reliable shippers like Logstash Forwarder and Filebeat. - **Elasticsearch Cluster Operation & Tuning:** Shard and replica allocation, handling disk space and node restarts, switching from auto-discovery (broadcast) to unicast discovery to avoid switch packet drops, disabling shard reallocation during maintenance, JVM heap limits (under 32 GB), and tuning the field data cache. - **Production Troubleshooting with Kibana:** Visualizing web front-end status codes, Varnish cache hit rates, REST API function performance, external integration bottlenecks (Apple iTunes proxy sniffing), unexpected cron job resource contention, and massive log volume spikes.

### Technical Details - **Logstash Ingestion & Parsing:** Logstash uses three internal queues: input, filter, and output. When processing multi-line logs such as Java stack traces or XML, multithreaded filtering cannot easily scale across lines, forcing single-thread processing or horizontal scaling via multiple Logstash instances. In contrast, Filebeat can perform multi-line handling on the source server and forward single grouped events. The speakers also stressed the importance of parsing native event timestamps from access logs (otherwise Elasticsearch records the log ingestion time) and converting numeric variables using Grok and `int` casts so Elasticsearch can perform aggregations, averages, and min/max statistics. - **Log Forwarding:** Relying on syslog over UDP caused dropped logs during traffic peaks and server failures. Interia moved to Logstash Forwarder (written in Go), which supports compression, encryption, backpressure/waiting, and automatic detection of new files. However, an unclosed file handle issue in Logstash Forwarder required configuring timeouts (e.g., 10 minutes of inactivity) to prevent disk space exhaustion from unlinked open log files. - **Elasticsearch Sizing and Maintenance:** Elasticsearch nodes should not exceed 32 GB of JVM heap memory to preserve compressed object pointers (avoiding switching to slower 64-bit pointers) and leave the remaining system memory (at least 50%) for OS filesystem caches. Disabling swap is mandatory. Interia configured nodes with a maximum of 200–250 GB of data per node to avoid multi-day shard synchronization times upon restarts. When updating nodes sequentially, administrators must disable shard allocation before restart to prevent the cluster from prematurely rebalancing shards across remaining nodes. - **Investigating Outages via Kibana:** In one case, a cluster experienced a surge from typical daily volumes of 15 GB (around 300,000 events per 10 minutes) up to 96 GB, peaking at over 30 million events per 10 minutes. Kibana visualizations isolated the spike to a single testing server looping over missing files and outputting endless Java stack traces. In another instance, Kibana graphs revealed an unexplained daily latency spike between 10:00 AM and 11:00 AM caused by a typo in a Tomcat cron task scheduled during peak daytime hours rather than at night.

### Takeaways - Never rely on default ingestion or cluster settings: enforce strict field types, configure unclosed file handle timeouts on log shippers, and explicitly disable shard allocation during node rolling restarts. - Limit Elasticsearch heap to under 32 GB and avoid large disks per node to ensure shard resynchronization remains manageable. - Centralized log visualization makes it possible to detect intermittent cross-service latency, misconfigured background jobs, and application regressions that are impossible to spot using conventional command-line tools like grep across multiple web servers.