Grzegorz Sztandera

### Overview In this talk, Grzegorz Sztandera (introduced phonetically as Jagger Stander) discusses the realities of managing mission-critical production systems and operating a 24/7 DevOps competence center in Poland. Using personal travel photographs from Chernobyl as a visual backdrop, he examines 14 real-world production incident cases. The presentation highlights the high stakes of production operations—such as life-saving aviation ground communications and emergency eCall automotive response systems handling millions of incidents on AWS—where downtime can have life-or-death consequences. Sztandera frames the talk around a post-mortem game comparing how many incidents stemmed from his own DevOps team versus external factors, developers, or other teams.

### Key Topics - **Incident Preparedness and Documentation:** Sztandera emphasizes the necessity of maintaining comprehensive operational documentation, step-by-step troubleshooting manuals, Confluence runbooks, disaster recovery tests, and post-incident root cause analyses (RCA). - **High-Stakes Production Operations:** Managing 24/7 mission-critical architectures like emergency eCall automotive telemetry and aircraft-to-ground communication, where missing 100 calls an hour can cost lives. - **SLA Management and Escalations:** Dealing with rigid operational SLAs, ticket automation breaches, and organizational challenges when dependencies across external teams impact response metrics. - **Cross-Team Failures vs. DevOps Ownership:** Walking through 14 production case studies, culminating in a 2 to 12 score showing that the vast majority of severe production outages originate outside the direct control of the DevOps team.

### Technical Details Sztandera walks through several specific outages and operational failures: - **eCall Interface Glitches:** After a deployment, emergency call center agents were unable to hang up because a frontend deployment omitted the end-call button. A fallback procedure (automatically retrying three calls) prevented complete failure, but highlighted testing gaps. - **Authentication & High Availability Outages:** An outage occurred when an upstream team’s two-factor authentication (2FA) service failed. The DevOps team used a configuration toggle to temporarily disable 2FA for emergency agents. In another case, primary and secondary firewalls failed sequentially at 3:00 AM, and an on-premise datacenter experienced a half-day blackout when maintenance crews cut the active power line while servicing the redundant one. - **Cloud Automation and CI/CD Collisions:** A central cloud team introduced automated Terraform scripts that repeatedly wiped custom AWS configurations. Another ongoing challenge involved unreliable shared and dedicated Jenkins instances, which caused dozens of deployment incidents and demonstrated the need to practice manual deployments. - **Performance and Capacity Bottlenecks:** Examples included missing database pagination causing list-of-refunds pages to take 2.5 minutes to load, a missing database index slowing down end-to-end (E2E) testing pipelines, and hitting 100% thread/queue limits during message processing. - **Security Incidents (Log4j):** When the zero-day Log4j exploit surfaced, the team took six hours to patch and deploy. Potential reconnaissance on an internet-facing E2E testing stage occurred 16 minutes after notification, prompting a complete snapshot, database dump, and clean redeployment. - **External and Human Factors:** One prolonged Docker container restart loop on production was traced to unoptimized queries and rogue internal testing servers hammering the live environment with invalid GET requests resulting in constant 404 responses. Another global three-hour 'critical outage' investigation occurred simply because a remote call center user experienced local office network issues and went to lunch without answering notifications.

### Takeaways Sztandera concludes that production failures are inevitable; the key is proactive preparation through disaster recovery drills, automated monitoring, and clear operational context. Organizations must maintain tested manual fallback procedures and root-cause documentation to avoid rediscoveries during off-hours emergencies. Finally, software engineers and product teams must write code with operational awareness, recognizing that negligence or poor architecture directly impacts the on-call engineers answering alerts at 3:00 AM.