"Building Cloud Security Automation at Scale"

## Overview Rich presents a comprehensive journey through cloud security automation architectures at scale, progressing from basic single-container approaches to fully event-driven serverless systems. Drawing on personal experience including Black Hat talks from seven years prior, the speaker illustrates how automation challenges evolve as organizations grow from dozens to thousands of cloud accounts across multiple regions.

## Key Topics

**Architectural Evolution** The talk progresses through five distinct architectural patterns: (1) basic single instance/container running automation against multiple accounts, (2) auto-scaling groups with pull-based worker patterns using SQS queues, (3) distributed instances/containers running within each target account for local access, (4) centralized serverless with one Lambda per region per account for parallelization, and (5) fully event-driven architecture using Kinesis pipelines.

**IAM and Permission Scaling** A critical challenge is managing permissions at scale without creating security vulnerabilities. The speaker describes implementing a two-role system: a provisioner role (with escalated privileges to update worker permissions across all accounts) and a worker role (with limited permissions to perform actual automation). Permission boundaries lock these roles to prevent either from escalating its own privileges or undermining the other.

**Handling Service Limits** Repeated API calls hit AWS service limits during assessments of large environments. Rather than calling live APIs each time, the solution caches assessment results in ElasticSearch, enabling GraphQL queries against historical data. Scheduled reassessments (approximately hourly) and event-based triggers (on infrastructure changes) keep data current without overwhelming service limits.

**Event-Driven Architecture** The "sweet spot" for large-scale automation uses Kinesis pipelines (or Kafka equivalents) to ingest events from CloudWatch, schedules, and API calls. These trigger either assessments or remediation actions, with results stored in DynamoDB for state tracking. This enables rapid detection and response—for instance, immediately shutting down a cryptocurrency miner upon a Guard Duty alert.

## Technical Details

**Job Orchestration and State Management** Central to distributed serverless automation is tracking job IDs across parallel Lambda executions. Results are tagged with job IDs, allowing correlation and aggregation. The system maintains state centrally while executing work distributed across accounts and regions.

**Observability and Debugging** Visibility in serverless systems remains challenging. The speaker emphasizes continuous instrumentation layers, state tracking, and reproducible debugging. Tools like X-Ray and commercial observability platforms help, but careful logging and job ID tracking are essential. Early systems ran into unexpected issues (orphaned network interfaces, VPC Classic remnants in legacy accounts) that broke assumptions at scale.

**Testing and Rollout Strategy** Initial deployments run in read-only mode, generating events sent to Slack for human approval before executing writes. This hybrid approach allows teams to validate rule logic and tune environments before enabling full read-write automation.

## Organizational and Operational Considerations

A significant blocker is organizational—security teams often lack the necessary cross-account access to implement automation. Technical solutions (permission boundaries, distributed execution, external IDs) can mitigate risk, but addressing this requires both technical design and organizational alignment.

Non-repudiation and detecting account owner tampering require strict central account controls, multiple independent verification tracks, and potentially dual-account setups balancing each other.

## Takeaways

No single architecture suits all organizations. Teams should evaluate their constraints: IAM delegation policies, organizational structure, scale requirements (dozens vs. thousands of accounts), and whether they can use native cloud provider tools or require open-source solutions. Tools like CloudMapper, Prowler, and StreamAlert provide reference implementations. The progression from simple to event-driven reflects real-world scaling challenges, and each step introduces new operational complexity that must be managed through careful state tracking, observability, and testing strategies.