"Zuul's Journey to Non-Blocking" by Arthur Gonigberg

### Overview Arthur Gonigberg presents Netflix's architectural migration of its edge proxy service, Zuul, from a blocking Java servlet model to a non-blocking, asynchronous architecture. Zuul operates as the front door to Netflix on AWS, managing tens of billions of requests per day and handling critical functions including routing, security (DDoS mitigation and blocking fraudulent traffic), monitoring, and resiliency. While initial micro-benchmarks showed dramatic throughput increases (up to 73,000 requests per second), real-world production performance between Zuul 1 and Zuul 2 showed only a modest 10% to 20% throughput gain. The talk explores how Netflix executed the rewrite, the severe engineering and operational pitfalls encountered, and the distinct resiliency and architectural benefits achieved.

### Key Topics - **Zuul 1 Architecture & Limitations**: Zuul 1 ran on Tomcat with standard Java servlets using one thread per request. Latency spikes at origins or connection spikes caused rapid thread pool exhaustion and severe cascading failures during traffic surges. - **Building Blocks of Non-Blocking Systems**: The shift required non-blocking I/O primitives (e.g., epoll on Linux, kqueue on BSD), an asynchronous programming paradigm (RxJava), and an event loop framework (Netty). - **Incremental Migration via Zuul 1.5**: To safely replace the engine while serving massive production traffic, Netflix first adopted Tomcat's APR connector to decouple connections from requests, then converted the filter chain to return observables (`applyAsync`), allowing filter execution in both Tomcat and Netty. - **Zuul 2 Architecture**: Netty server fronting incoming requests, Netty client managing outbound proxying, and an RxJava filter chain stitched in between on single-threaded event loops.

### Technical Details & Operational Pitfalls - **Blocking Shared Libraries**: Shared platform libraries often made blocking calls (such as `Thread.sleep`), which paused Netty event loop threads and stalled hundreds of concurrent requests. Netflix used `reactive-audit` to detect blocking calls at runtime. - **ThreadLocal State Corruption & The 'Geo Shuffle'**: Netty event loop threads multiplex many requests. Shared libraries relying on `ThreadLocal` storage leaked state between requests. A severe production bug dubbed the 'Geo Shuffle' occurred when geo-lookup IP data was stored in thread locals, causing San Francisco users to be assigned geo-data for Siberia, Brazil, or South Africa and routed to remote CDNs. The fix required explicitly copying and setting channel state onto thread locals upon entering Netty channel pipelines or RxJava threads. - **Direct Memory Leaks**: Netty utilizes off-heap direct memory via JNI and epoll. Incomplete request lifecycles caused gradual direct memory leaks that silently brought server traffic to zero RPS without throwing traditional heap errors. - **Debugging & The Request Passport**: Standard thread dumps and stack traces were ineffective in async code. Netflix truncated repetitive RxJava stack traces, disabled trace generation in high-CPU critical paths, and implemented the 'Request Passport' (inspired by Zach Hellman) to record timestamped channel state transitions to debug complex timing issues and ELB timeouts.

### Takeaways & Guidance - **Workload Profiles Matter**: Zuul is fundamentally CPU-bound due to authentication, encryption, decryption, and hashing. Non-blocking models do not reduce core CPU processing requirements; hence, CPU-bound workloads see minimal throughput improvements (10–20%). In contrast, I/O-bound services, large file streams, chat systems, or push notification architectures benefit heavily. - **Resiliency Benefits**: Zuul 2 excelled during 'retry storms' (when failing clients rapidly retry, outstripping auto-scaling capacity). Instead of aggressively throttling legitimate traffic, Zuul 2 handled significantly higher RPS at max CPU and safely dropped retries when origin error thresholds were breached. - **Expanded Capabilities**: Zuul 2 comfortably handles tens of thousands of concurrent connections per instance, enabling WebSocket support for push notifications and removing the need for strict timeouts and Hystrix circuit breakers.