Enabling Message Level Protection at Uber Debosmit Debo Ray & Jovon Itwaru

### Overview At Uber, handling massive amounts of sensitive data—including rider and driver personal information, health records, credit card numbers, and government identifiers—requires robust safeguards as information moves across a complex microservice ecosystem. In this presentation, Uber security engineers Jovon Itwaru and Debosmit "Debo" Ray outline the motivations, architectural trade-offs, design, and implementation of **CryptoFX**, an internal client-side field-level encryption module developed to secure sensitive data at rest and in transit.

### Key Topics - **Goals & Requirements**: Protecting sensitive data against threats such as rogue engineers or database breaches; enforcing strong cryptography defaults (e.g., authenticated encryption) while preventing developer mistakes like nonce reuse; providing an intuitive, low-friction API; ensuring cryptographic agility across algorithms and Key Management Systems (KMS); and satisfying regulatory mandates like PCI-DSS, PSD2, and GDPR. - **Evaluation of Encryption Approaches**: Comparing volume/disk encryption, database transparent data encryption (TDE), column-level encryption, server-side encryption (e.g., S3 SSE), and client-side encryption. Uber determined that client-side field-level encryption was the only option providing adequate protection against network snoopers and compromised databases while remaining flexible enough for their infrastructure. - **Architecture of CryptoFX**: A Go-based library leveraging dependency injection, envelope encryption, and configurable abstractions over ciphers and key providers. - **Operational & Developer Experience Challenges**: Driving adoption via "white glove" onboarding, accommodating searchability requirements, managing legacy schema migrations, and handling API extensibility in a monorepo.

### Technical Details - **Field-Level Envelope Encryption**: Rather than encrypting entire objects or volumes, Uber encrypts specific sensitive fields (e.g., address, SSN) to minimize performance overhead. For database storage, symmetric encryption using **AES-GCM (AES-GCM-256)** is employed via envelope encryption: a Key Encrypting Key (KEK) encrypts an ephemeral Data Encryption Key (DEK), which in turn encrypts the plaintext. Both the ciphertext and the encrypted DEK are stored together. - **Flexible Deployment Models**: CryptoFX supports three modes depending on performance and security trade-offs: 1. *Local Encryption*: DEKs and crypto operations remain local within the application process. 2. *Remote Encryption*: Data is sent to a central remote service (e.g., HashiCorp Vault Transit engine), and the returned ciphertext only needs the key namespace and Base64 ciphertext. 3. *Hybrid Model*: Services obtain DEKs from a remote KMS and cache/refresh them locally at configured intervals for high-throughput batch operations. - **Configuration & Namespaces**: CryptoFX uses a concept of namespaces (similar to keyrings) specifying the KMS, cipher, key identifier, and key version. The `KeyProvider` abstraction enables seamless interoperability between local secret delivery (mounted files backed by `tmpfs`) and remote KMS solutions like HashiCorp Vault or AWS KMS without breaking callers. - **Struct-Level Encryption & Search**: Using Go struct tags, developers can annotate struct fields (supporting strings, byte slices, and embedded structs) so that multiple fields can be encrypted in bulk under a single generated DEK. To support database search queries over encrypted fields without adopting leaky schemes, CryptoFX provides a `Hasher` interface generating deterministic **HMAC-SHA256** digests that are stored alongside ciphertexts.

### Takeaways - **Avoid Building on Ad-Hoc Solutions**: Off-the-shelf tools and raw standard libraries often fail to fit complex, polyglot microservice environments or lead developers into misconfigurations like nonce reuse or weak ciphers. Sane defaults and centralized abstractions are critical. - **Early Engagement & White-Glove Support**: Security teams driving crypto adoption must engage product teams early during design reviews and SDLC phases, providing direct onboarding assistance and establishing recognizable "happy path" code patterns. - **Future Roadmap**: Ongoing work includes migrating legacy keys to HashiCorp Vault, adding high-level signer and verifier interfaces (navigating constraints like cloud CDN RSA verification limitations), and continuing to monitor searchable encryption techniques (noting that fully homomorphic, property-preserving, and functional encryption remain either too slow or vulnerable to inference attacks on low-entropy data).