APPSEC CA 2017 Devdatta Akhawe and Emily Stark

### Overview In this AppSec California 2017 presentation, Emily Stark and Devdatta Akhawe examine the systemic challenges surrounding TLS certificate revocation on the web. When server private keys are compromised or certificates are misissued, web clients require a mechanism to determine whether a certificate has been revoked. The speakers explain why conventional revocation mechanisms fail, how OCSP Stapling and Must-Staple provide a viable security architecture, and how Dropbox deployed OCSP stapling and used Chrome's experimental "Expect-Staple" mechanism to monitor failure modes in production.

### Key Topics - **Failures of Traditional Revocation:** Early mechanisms such as Certificate Revocation Lists (CRLs) proved impractical because list sizes scaled poorly, resulting in unacceptable client performance costs and wasted bandwidth. While browser-curated mechanisms (such as CRLSets) helped mitigate scale for high-value intermediates, they did not support scalable revocation for arbitrary websites or non-browser clients. Online Certificate Status Protocol (OCSP) solved list size issues by querying individual certificate validity but introduced latency on every handshake, leaked user browsing habits to Certificate Authorities (CAs), and forced browsers to "fail open" when CAs were unreachable or unresponsive. - **OCSP Stapling and Must-Staple:** OCSP Stapling addresses privacy and critical-path latency by having the server fetch and cache a signed OCSP response from the CA, then staple it to the TLS handshake. However, standard stapling can be bypassed by an attacker simply omitting the staple. To enforce validation, the Must-Staple TLS feature extension mandates that browsers reject connections lacking a fresh, valid stapled OCSP response. Yet browsers hesitate to enforce Must-Staple universally due to fear of showing disruptive certificate error pages when servers misconfigure stapling. - **Expect-Staple Reporting:** To solve visibility and rollout risks, Chrome introduced "Expect-Staple," an experimental report-only header/directive. Similar in concept to report-only CSP or HPKP, Expect-Staple instructs Chrome to submit telemetry reports back to the site owner whenever a stapled OCSP response is missing or invalid, allowing operators to diagnose failures without breaking live traffic.

### Technical Details & Dropbox Implementation - **In-House OCSP Fetching Daemon:** Default OCSP stapling in standard web servers (like NGINX) suffers from limitations, such as synchronously querying CAs on worker processes or failing to gracefully manage timeouts. Dropbox engineered a dedicated background service that queries the CA (such as DigiCert) every eight minutes for fresh OCSP responses, verifies the response, and writes valid responses to disk for NGINX worker processes to reload seamlessly. - **OpenSSL CLI Verification Quirks:** During development, Dropbox discovered significant quirks in OpenSSL CLI utilities. The `openssl ocsp` verification command would return a zero exit status regardless of certain verification failures, requiring Dropbox's automation to parse stdout text (such as checking for `Response verify OK`) rather than relying purely on exit codes. - **Production Error Telemetry Analysis:** Telemetry collected via Chrome's Expect-Staple revealed two primary classes of real-world failures: 1. *TLS Interception Proxies:* Corporate Middleboxes and inspection appliances often generate local root certificates and strip stapled OCSP responses, triggering Expect-Staple reports. 2. *Client Clock Skew:* A massive percentage of verified Expect-Staple errors stemmed from user device clock skew. Around 95% of skew-related errors involved client clocks running behind the real time (often days or weeks behind due to CMOS battery depletion or manual adjustments), causing valid OCSP responses to appear as issued "in the future."

### Takeaways - OCSP stapling yields direct performance and privacy benefits by eliminating client-to-CA network round-trips. - Deploying robust OCSP stapling requires reliable out-of-band caching and verification rather than relying on naive web server defaults. - Expect-Staple provides critical operational observability, proving that most apparent validation failures stem from middlebox proxy behavior and client clock skew rather than server-side outages.