"Hodor: Detecting and Addressing Overload in LinkedIn Microservices" by Bryan Barkley

Hodor (Holistic Overload Detection and Overload Remediation) is a system developed by LinkedIn to detect and address overload conditions in microservices. Deployed to approximately 1000 services in production for over a year, Hodor has engaged in hundreds of instances to protect services from becoming overloaded and unresponsive.

Barkley defines overload as when a process is asked to do more work than it can handle with available resources. Resources fall into two categories: physical (CPU, memory, disk IO, network bandwidth—hardware-limited) and virtual (thread pools, connection pools, file descriptors, semaphores—configurable thresholds that still cause system-wide overload when exceeded). Common scenarios include traffic spikes, back pressure from slow downstream services causing upstream accumulation, and noisy neighbors on the same host impacting each other.

Hodor uses three primary detectors with conservative thresholds. The Heartbeat/CPU Detector monitors JVM thread scheduling delays by running a background thread that sleeps for ~10ms; consistent violations (e.g., 28+ violations in 30 windows) indicate CPU starvation. The Garbage Collection Detector tracks time spent in GC and fires on thresholds like >11% GC time for >10 consecutive seconds, indicating potential GC death spirals. The Thread Pool Detector monitors Jetty thread pool queue wait times, flagging overload if requests wait >50ms for >10 consecutive seconds.

Hodor employs multiple load shedding strategies. Initial Adaptive Percentage approach dropped fixed traffic percentages but proved inefficient with varying patterns. Concurrency-Based strategy dynamically limits concurrent requests, better handling traffic fluctuations. Rate-Based with Traffic Tiering uses throughput as the metric with concurrency as the source of truth, enabling three-tier prioritization: optional (pre-fetches, offline systems—dropped first), degradable (user sees some impact), and non-degradable (critical paths—dropped last).

To prevent retry storms, Hodor maintains server-side histograms tracking retry counts and client-side budgets limiting retries. No retry storms have occurred with this system in place. Hodor deployed in batches with load shedding initially disabled, monitoring services for at least one week through production load tests before enabling. False positives arose from code behavior changes, fleet resizing, and poorly-tuned garbage collection; solutions included correlating detector firings with latency increases using moving average crossover algorithms.

Future roadmap includes additional detectors for Netty event loops and Parsec (LinkedIn's async execution engine); capacity engineering integration; intelligent auto-scaling accounting for cascading failures; and support for stateful and data systems. Hodor operates transparently at the framework level, requiring zero configuration from application teams.