Challenges with running fstests and blktests

This talk addresses challenges and lessons learned when scaling filesystem (fstests) and block device testing (blktests) automation, primarily through experience with the kdevops project. A central theme is extreme non-determinism: failures often occur rarely (e.g., ext4 failure 1 in 300 times, block test failure 1 in 80 times) yet produce catastrophic errors, making them difficult to reproduce and debug. The speaker cites an example of a block test failure that took eight months to diagnose and was ultimately fixed in kernel 5.12.

The speaker proposes a resource allocation model where testing effort should be divided: 25% test design, 25% tracking results, 25% reporting bugs, and 25% fixing low-hanging fruit. Currently the community excels at test design but lags in the other three areas, creating a bottleneck.

The kdevops project addresses testing variability using kconfig, supports both local virtualization and cloud deployments, and uses ansible for management. Testing architecture employs loopback devices and truncated files to simulate NVMe drives, testing both XFS and Btrfs backends. Running 100 loops of fstests takes approximately 5-6 days; approximately 50GB capacity is needed in worst-case scenarios.

A significant discussion centers on module removal testing conflicts. While validating cleanup paths and reference counting via module removal is important, tests like those for scuzzy debug and nvme modules fail due to module dependencies unrelated to actual functionality. The consensus emerging from discussion is to separate module removal tests into distinct categories so failures in dependency cleanup don't cascade into false test failures. The speaker notes that config-based configuration (rather than module parameters) and the patient module remover help mitigate these issues.

Other technical challenges identified include: discrepancies between .bad result files and junit XML output (failures appearing in one but not the other), inconsistent dmesg logging across test suites, and improper handling of tests that should be skipped versus tests with expected failures. The speaker advocates for capturing dmesg logs in all cases and properly annotating expected failures rather than skipping tests.

Establishing a baseline for a new filesystem takes one to two months. The speaker proposes a 'lazy baseline' approach: once a failure appears in at least two different test configurations, it's expunged for all configurations and deferred for investigation. Proper baselines require at least 100 loops to establish confidence.

The speaker emphasizes that testing strategies vary by organizational goals: QA pursuing theoretical zero-defect approaches may invest heavily in rare bugs, while production-focused teams prioritize bugs affecting actual deployments. Both are valid but require different resource allocation.

Final recommendations include centralizing filesystem testing documentation (similar to Btrfs wiki pages), volunteering for collaboration on baseline establishment using the employer-provided server, and delivering additional kdevops demonstrations at conferences like Linux Plumbers. The talk concludes with an invitation for community collaboration on shared testing infrastructure.