WAFs FTW! A modern devops approach to security testing your WAF

### Overview In this presentation, speakers Christian Papathanasiou, Chaim Sanders, and Zack Allen introduce FTW (Framework for Testing WAFs), an open-source Python tool built on pytest to bring automated, quantitative testing and modern DevOps practices to Web Application Firewalls (WAFs). Evaluating WAF solutions has historically relied on qualitative, subjective assessments, such as Gartner Magic Quadrants, Forrester Waves, or OWASP WAFEC checklists, which fail to provide repeatable, data-driven guarantees about attack coverage, regression detection, or zero-day readiness. FTW solves this by formalizing attack payloads into structured YAML test definitions, enabling continuous integration testing across rule releases and edge deployments.

### Key Topics - **Qualitative vs. Quantitative WAF Evaluation**: Traditional analyst reports and compliance checklists (e.g., PCI DSS checkboxes) do not prove whether a WAF reliably protects against nuanced attack variants. Quantifiable testing provides baseline metrics for vendor bake-offs and engineering validations. - **FTW Architecture and Design**: Modeled after the Scapy network packet library, FTW automatically constructs baseline minimal HTTP/1.1 requests, allowing users to override specific attributes in human-readable YAML. Tests can intentionally violate RFCs (such as placing a message body on GET or HEAD requests) without being blocked by standard client libraries like curl or Python Requests. - **DevOps Integration for OWASP CRS**: The OWASP Core Rule Set (CRS) project uses FTW inside Travis CI using Docker containers. As of CRS v3, all rule commits and pull requests require corresponding FTW test cases to prevent regressions. - **Edge CDN Implementation at Fastly**: Fastly translates ModSecurity SecRules into Varnish Configuration Language (VCL) to run WAF logic directly at the edge. Fastly wrapped FTW in a custom test harness to verify behavior parity, validate regex ReDoS safety, and correlate injected test headers with WAF execution journals.

### Technical Details FTW integrates with pytest to execute attack fixtures against target web applications or firewalls and verify expected HTTP status codes or log output. Key technical aspects include: - **Non-RFC Compliant Payloads**: By crafting raw request streams, FTW can test rules like CRS 920170 (blocking GET/HEAD requests containing a `Content-Length` or request body), which standard HTTP client libraries prevent sending. - **Fastly Translation & Validation Toolchain**: Fastly converts SecRules into VCL while preventing evasion bugs caused by discrepancies in request parsing or transform functions (e.g., lowercase conversions). Fastly's harness injects custom headers tracking rule timestamps and test IDs, reconciling journal logs with WAF logs to ensure payloads trigger the intended rule rather than failing on an unrelated check. - **Vulnerabilities Identified and Fixed Upstream in CRS**: - *Restricted Extension Chaining Bug*: A protocol enforcement rule had a lowercase transformation applied in the wrong chain, allowing bypasses using uppercase file extensions like `.EXE`. - *Remote File Inclusion (RFI) Evasion*: The rule used a case-sensitive regex and missed the `file://` scheme, allowing bypasses via uppercase schemes like `HTTP://`. - *Session Fixation Flaw*: A rule used a `beginswith` operator comparing Referer and Host headers, which attackers could bypass using subdomains. The rule was corrected to use `endswith`.

### Takeaways FTW transforms WAF testing from ad-hoc manual verification into repeatable CI/CD regression suites. Moving forward, the project roadmap includes developing YAML linters for rule authors, expanding negative testing by replaying traffic from top domains (e.g., Alexa top 1M) to reduce false positives, benchmarking cross-WAF performance, and maintaining community-accessible exploit tests for emerging vulnerabilities. FTW is open-source and installable via PyPI.