The Life of a Kernel Bugfix

## Overview

This talk explains how to properly develop and distribute kernel bugfixes from upstream mainline through stable releases to end-user distributions. The speaker targets hardware vendors, distribution maintainers, and contractors maintaining kernel code, emphasizing an upstream-first philosophy to maximize scalability and reduce maintenance burden.

## Key Topics

**Linux Kernel Release Process**: The kernel follows a structured cycle starting with a 2-week merge window after each release, followed by 7–8 release candidates (RC) before the next stable version. Specific versions (6.6, 6.1, 6.12) are designated as long-term stable (LTS) releases, which continue receiving backported fixes from newer mainline versions.

**Distribution Kernel Categories**: Four types exist: (1) pristine upstream kernels with only binary/config differences, (2) upstream kernels with small patches for security or unapproved fixes, (3) kernels with significant downstream patches requiring merges rather than simple rebasing, and (4) kernels with extensive subsystem backports mixing code from multiple upstream versions.

**Distribution Policies**: Each distribution has different acceptance criteria: bug reports or CVE numbers, test cases/reproducers, multiple internal reviews, and varying maintenance windows. These policies differ significantly between Debian, Ubuntu, Red Hat, and others, making direct patching of all distributions non-scalable.

## Technical Details

**Mainline and Stable Process**: Contributors submit patches to subsystem maintainers for review. Patches submitted during the 2-week merge window are typically held until after RC1. Stable maintainers (Greg, Sasha) automatically cherry-pick bug fixes from mainline if tagged with `CC: stable@vger.kernel.org` or containing a Fixes tag. These are queued, tested by community volunteers, released weekly, and eventually picked up by distributions.

**Best Practices**: - Include a Fixes tag pointing to the commit that introduced the bug - Add `CC: stable@vger.kernel.org` to signal stable maintainers - Include clear bug description, stack traces, and reproducers - Keep patches small and self-contained - Run `scripts/checkpatch.pl` for code compliance - Send to appropriate subsystem maintainers

**Backporting Challenges**: When automatic cherry-picking fails (due to code changes), manual backports require: annotating with upstream commit references, adding sign-offs, and explaining conflicts with a scissors separator (---). This applies to patches that don't meet upstream stable policy but fit distribution policy.

**Stable Kernel Requirements**: Fixes must address bugs (crashes, hangs, data corruption), not features or trivial changes. Since February 2024, all commits in upstream releases receive CVE assignments.

## Takeaways

The optimal path is: fix → mainline → upstream stable → distributions. This scales because upstream stable has fewer requirements than individual distributions. When a fix includes a Fixes tag and stable CC, it flows automatically through the entire pipeline.

For distribution-specific issues, direct submission requires following each distribution's documented process—labor-intensive but sometimes necessary. Features face significantly higher barriers than bug fixes; the upstream-first approach is recommended for practical maintenance.