"When the OS gets in the way" by Mark Price

Mark Price from Elmax Exchange discusses optimizing the Linux operating system to minimize latency and jitter for ultra-low-latency applications, particularly relevant to high-frequency financial trading systems. The talk emphasizes that while Linux is an excellent general-purpose OS, specific tuning is necessary for sub-millisecond latency requirements.

The speaker defines system jitter as variations in response time and identifies that the underlying causes are the operating system's task scheduling decisions, CPU housekeeping tasks, and hardware interrupts. Most systems exhibit a "long tail" where worst-case latency significantly exceeds typical performance.

The technical foundation covers modern multi-socket, multi-core hardware: NUMA nodes with local RAM per socket, physical cores with shared L1/L2 caches, logical hyperthreads sharing physical core resources, and L3 caches spanning multiple cores. The completely fair scheduler (CFS) distributes CPU time fairly across all runnable tasks but lacks application-specific awareness.

The core solution involves CPU isolation and affinity techniques: using the isolcpus kernel parameter to reserve CPUs, creating separate CPU sets via cgroups to compartmentalize OS and application tasks, pinning latency-critical threads to specific cores using sched_setaffinity(), and restricting runtime threads using taskset. The speaker demonstrates that cpusets are superior to isolcpus alone because they maintain proper load-balancing within isolated domains.

Verification requires Linux tracing tools: lstopo for hardware topology discovery, perf for sampling scheduler events, and ftrace for function-level tracing. Through tracing, specific jitter sources are identified including cache_reap (slab allocator maintenance), vmstat_update (kernel statistics), kworker threads, hardware interrupts, and timer ticks (1000/second default).

Measurements of inter-thread latency on a 64-CPU system show tuned latency approximately 2x better than untuned on quiet systems. Under realistic workloads with deliberate thread contention every 30 milliseconds, untuned latency spikes above 800 microseconds while properly tuned systems remain consistently low.

The methodology emphasizes measurement discipline: end-to-end testing with production traffic replay, isolated one-change-at-a-time testing, and rigorous verification through tracing. Key principles include identifying latency-critical threads, reserving CPU resources, pinning threads to isolated cores, carefully handling interrupts, and continuously validating assumptions rather than relying on intuition.