VI

## Overview

Larry Lids, CESO at a large corporation, presents on "verifiable effectiveness" - the importance of testing security controls to verify they actually work before a breach occurs. The talk is framed around two cautionary stories: a stolen laptop in London where encryption status couldn't be verified within the GDPR 3-day notification window, and a red team exercise that successfully created an unauthorized account despite having an alerting control in place. These stories illustrate that well-designed controls can fail if not properly tested and verified.

## Key Topics

**Technology Controls:** Starting from his experience in higher education, Lids describes how machines compromised in ~10 minutes without proper security configuration. In corporate environments, he implemented a disciplined approach using CIS (Center for Internet Security) benchmarks customized for their environment. Configuration baselines grew from 2 platforms to approximately 50 platforms over time, tested weekly (automated) or monthly/quarterly (manual). The approach becomes increasingly critical for cloud environments, where misconfigurations like exposed S3 buckets pose immediate public risk. Cloud requires continuous verification rather than periodic checks.

**Process Controls:** Implemented a Do-Review-Approver (DRA) cycle for approximately 900 controls (security and operational). The doer executes the control and documents evidence, the reviewer validates the evidence, and the approver confirms completion and provides accountability to management. Evidence collected in IPE (Information Provided by Entity) format allows the same data to satisfy both internal verification and external auditors. The process took 2-3 years to mature with another 2-3 years needed for full effectiveness. Key lesson: starting with too many controls at once was a mistake; should begin with the most critical ones.

**People Controls:** Weekly simulated mini red team exercises, starting at 5-6 tests but scaling to 40-50 weekly for adequate statistical evidence. Evaluation criteria: (1) Did the SOC detect the attack? (2) Did they respond appropriately per process (lock/delete unauthorized accounts, investigate compromise, etc.)? Eventually made this a contractual measure with service providers, including financial penalties for detection or response failures. Results were instantaneous, revealing deficiencies in the SOC that weren't apparent without regular testing.

## Technical Details

The talk emphasizes parallels to backup testing: organizations test backups regularly to verify restoration works, yet rarely apply this discipline to security controls. Configuration verification evolved from scripts to automated tools. The process side required significant coordination, with many control executors being managed service providers. The evidence collection in IPE format was challenging and required training, as not all technologists naturally know how to structure this data.

Cloud security fundamentally changes the calculus: on-premises environments have multiple defensive layers (firewalls, antivirus, monitoring), so SMB permission misconfigurations may be caught elsewhere. In cloud, a single misconfiguration can immediately become public news, making configuration verification critical.

## Takeaways

**Confidence Without Perfection:** The approach doesn't claim bulletproof controls. Every technology scan found things needing fixing; process controls failed regularly; and sophisticated bad actors will eventually exploit novel attack vectors. However, by knowing controls are actively working, if a breach occurs, it won't be due to "basics" (like unencrypted laptops). The reputational message shifts from negligence to sophisticated attack.

**Red Team Value and Resistance:** Red teams are powerful but underused due to fear of results. Critical success factors: (1) notify relevant stakeholders before red team starts to prevent accidental escalation; (2) maintain visibility to prevent production outages; (3) frame findings as learning opportunities, not failures. Red teams effectively drive executive attention and funding for fixes that other assessments cannot.

**Continuous Improvement Culture:** Regular testing creates a feedback loop. Weekly attack simulations against the MITRE ATT&CK framework revealed deficiencies the SOC didn't know existed, driving improvements in both detection and response. This is preferable to waiting for actual breach notification.

**Maturation Timeline:** Technology: ~10 years to mature. Process: 2-3+ years and ongoing. People/detection-response: instantaneous improvements from regular testing.