What Every Driver Developer Should Know about RT

Julia Cartwright, a Linux stable RT (real-time) maintainer at National Instruments, presents common pitfalls in device drivers running the preempt-RT patch and provides practical guidance for driver authors. The talk focuses on "mixed criticality systems"—embedded systems that share a kernel between real-time and non-real-time tasks—and how driver misbehavior can degrade real-time latency guarantees.

Cartwright defines the critical performance metric as "Delta": the time from an external event (interrupt, timer, I/O) firing to the real-time task executing. This breaks into two phases: (1) IRQ dispatch latency—time from hardware event to thread scheduler notification, and (2) scheduling latency—time from scheduler awareness to CPU execution.

**IRQ Dispatch Latency Issues:** In mainline kernels, interrupt handlers run in hard-irq context with interrupts disabled, so high-priority events must wait for low-priority handlers to complete. Preempt-RT addresses this via IRQ threading: only minimal shim code runs in hard-irq context; handlers execute in preemptable threads. However, drivers must avoid threading if involved in interrupt dispatch (IRQ chips, CPU frequency control) by using the `IRQF_NO_THREAD` flag.

Another problem: explicit interrupt disabling via `local_irq_disable()` (~80 instances in the kernel drivers tree). When external events fire during disabled intervals, interrupt handlers cannot execute until re-enabling. The RT patch introduces local locks—an API replacing these calls for per-CPU data access. Local locks use mutexes with priority inheritance under RT and prevent two tasks on the same CPU from entering critical sections simultaneously, preserving driver synchronization semantics without disabling interrupts.

**MMIO Access Challenges:** A memorable case: plugging in an Ethernet cable caused 174-microsecond latency spikes (10× worse than 17-microsecond baseline). Root cause: PCI writes are posted (buffered in interconnect); a read command flushes buffers but blocks CPU interrupt dispatch until the read completes (architectural constraint). The workaround—following writes with read-backs—prevents buffer accumulation but sacrifices throughput. Additional hardware quirks include devices entering low-power states during MMIO, introducing exit latency.

**Scheduling Latency Issues:** `preempt_disable/enable()` calls (~60 driver instances) delay high-priority task scheduling. Mainline spinlocks disable preemption; preempt-RT converts them to priority-inheritance mutexes, allowing preemption by higher-priority threads. Raw spinlocks (traditional spinning locks for scheduling-critical code) must be used sparingly—only for code invoked in hard-irq context or interrupt dispatch—with minimal, bounded critical sections.

**Key Recommendations:** 1. Avoid `local_irq_disable()` unless involved in interrupt dispatch; use local locks instead 2. Consider MMIO buffering and interconnect behavior; evaluate read-backs 3. Avoid `preempt_disable()` unless synchronizing with the scheduler 4. Use raw spinlocks only when necessary; keep critical sections minimal 5. Recognize that very tight latency bounds (microseconds) may require architectural changes like separate processors or accelerators

The talk emphasizes that thoughtful driver design—especially regarding interrupt handling and synchronization—directly impacts real-time guarantees for end users.