Bartłomiej Pisulak: Embracing Chaos: Breaking Your System to Make It Indestructible | DOD Warsaw

### Overview In this talk presented at a DevOps conference in Warsaw, Bartłomiej Pisulak (BK), a Senior Manager at Pegasystems, introduces the fundamentals of chaos engineering and argues for its adoption in modern distributed architectures. Drawing a direct connection between mathematical Chaos Theory and modern IT architectures, Pisulak explains that complex, distributed systems are inherently chaotic—where minute fluctuations can trigger massive downstream failures. He outlines the history, core principles, business justification, and practical execution of chaos experiments using cloud tooling.

### Key Topics - **Chaos Theory Foundations:** The talk reviews Edward Lorenz's work (and early insights by Henri Poincaré), explaining the butterfly effect and how small changes in dynamic initial conditions yield drastically different outcomes, contrasting chaotic systems (which are deterministic if all variables are known) with purely random systems. - **Origins of Chaos Engineering:** Pisulak traces chaos engineering back to Netflix's 2008 AWS migration after a three-day DVD outage. This spurred the creation of Chaos Monkey, the Simian Army (including Chaos Gorilla and Chaos Kong), and eventually specialized industry tools such as Gremlin and AWS Fault Injection Simulator (FIS). - **Core Principles:** Developed at Netflix, the five foundational principles are: building hypotheses around steady-state behavior, simulating real-world events, running experiments in production, automating experiments continuously in delivery pipelines, and minimizing blast radius. - **Prerequisites for Chaos Testing:** Before introducing chaos, organizations must first fix all known bugs and establish clear observability to define and monitor their system's normal steady-state baseline. - **The Business Case (Resiliency, Reputation, Money):** Pisulak emphasizes that downtime causes severe human strain (such as 2 a.m. emergency pages) and brand damage, citing user impatience when websites take over 10 seconds to load. Financially, Gartner and other research estimate average downtime costs between $5,600 and $9,000 per minute (averaging around $7,000 per minute), with major historical incidents like Delta Airlines' $150 million loss during a five-hour outage in 2016 and significant cross-service blast radiuses during AWS outages.

### Technical Details Pisulak demonstrates a basic chaos engineering experiment on AWS using Terraform and AWS Fault Injection Simulator: - **Infrastructure Provisioning:** Terraform deploys an Application Load Balancer (ALB), target group, security group, and an Auto Scaling Group managing EC2 instances (configured with desired capacity 2, max size 4) labeled with the tag `to_be_destroyed: true`. - **Observability:** A lightweight Python script (`observe.py`) continuously polls the ALB DNS endpoint, flagging non-200 HTTP responses and recording total downtime seconds. - **Fault Injection:** In AWS FIS, an experiment template targets running EC2 instances with the designated tag and triggers an instance termination action. - **Results & Findings:** The termination caused 87 seconds of total website downtime before the Auto Scaling Group restored healthy instances. While the hypothesis was confirmed (the architecture survived and self-healed without manual intervention), the experiment revealed architectural latency and measurable downtime that cost money, prompting engineers to evaluate whether bare EC2 instances are the right architecture.

### Takeaways - Chaos engineering does not create flaws; it uncovers pre-existing vulnerabilities before they manifest as unplanned customer outages. - Fixing known defects and implementing robust observability are strict prerequisites before introducing simulated chaos. - Tools like AWS Fault Injection Simulator make it straightforward to execute controlled chaos experiments against infrastructure. - The ultimate objective is moving experiments toward production pipelines, reducing human operational stress, protecting revenue, and continuously proving system self-healing capabilities.