AppSec EU 2017 Creating An AppSec Pipeline With Containers In A Week by Jeroen Willemsen

### Overview At AppSec EU 2017, security architect Jeroen Willemsen shared how he and a colleague built an automated application security (AppSec) pipeline using containerized tooling within a single week (spending approximately 34 hours). Undertaken in late 2016, the project aimed to demonstrate that integrating baseline security controls into an existing CI/CD workflow does not require extensive time or complex custom architectures, even when dealing with varied technology stacks, legacy systems, and rapid delivery cycles.

### Key Topics - **Integrating Security into Existing CI/CD**: Layering automated security checks onto an established workflow powered by Jira, Git/GitLab, and Jenkins Job DSL without disrupting ongoing development. - **Containerized Security Tools**: Running scanners inside Docker containers to simplify local testing, streamline cloud deployments (AWS), and allow modular replacement. - **Vulnerability Management & Triage**: Consolidating scanner output into vulnerability management platforms (ThreadFix / DefectDojo) and feeding actionable items back to developers via Jira. - **Overcoming Technical Hurdles**: Dealing with legacy non-containerized APIs, filtering noisy false positives, handling custom language stacks (Scala, Swift), and maintaining developer trust.

### Technical Details The target environment comprised AWS infrastructure hosting Docker containers, modern APIs, an Angular front end, and native Android/iOS mobile apps, alongside non-containerized legacy warehouse systems. The pre-existing CI/CD pipeline used Jenkins (configured via Job DSL) to pull code, run unit and end-to-end integration tests, and deploy container artifacts.

To construct the AppSec pipeline, the team introduced several automated stages: - **Dependency & License Checking**: Added OWASP Dependency-Check alongside language-specific plugins (such as Maven and Go plugins) to detect known vulnerable libraries and enforce organizational open-source licensing policies early. - **Container Layer Scanning (Clair)**: Implemented CoreOS Clair to scan Docker container layers against vulnerability databases. Willemsen noted Clair's limitation: while it flags issues in base OS packages (e.g., finding `wget` in an intermediate layer), custom and thin application layers often pass unnoticed. - **Dynamic Application Security Testing (OWASP ZAP)**: Evaluated OWASP ZAP and Burp Suite. Burp Suite was set aside due to headless containerization and REST API integration challenges at the time. OWASP ZAP was compiled and run in Docker, triggered via its REST API following end-to-end integration tests for quick scans and scheduled for deeper full scans. Results were stored in AWS S3 and sent to ThreadFix. - **SAST Considerations**: Traditional static analysis tools (e.g., Fortify, Checkmarx) struggled with the project's highly customized Scala backends and modern Swift codebases, leading the team to prioritize other automated checks instead. - **Handling Legacy Services**: Automated security scanning initially crashed a fragile legacy warehouse backend. To fix this, the team stubbed the legacy APIs with a mock service, allowing aggressive DAST against Scala APIs while leaving legacy systems to scheduled manual assessment. - **Triage & Traceability**: The pipeline mapped findings back to Git commit messages and Jira issue keys. ThreadFix was selected over DefectDojo (at its late-2016 state) to centralize findings before creating Jira backlog items.

### Takeaways - **Automate and Containerize Early**: Containerizing security tooling makes scanners portable between developer machines and cloud pipelines, enabling modular upgrades as tools mature. - **Provide Rapid, Filtered Feedback**: Fast feedback prevents developer frustration. Unfiltered scanner noise damages trust, making upfront false-positive tuning critical. - **Collaborate with Development Teams**: Rather than dumping security tickets onto team backlogs, security engineers should sit alongside developers, coordinate sprint work, and jointly refine issues. - **Manual Testing Remains Necessary**: Automated tools miss context-heavy vulnerabilities. During the Q&A, Willemsen highlighted that scanners like ZAP could submit a persistent cross-site scripting (XSS) payload and later retrieve the rendered page without correlating that the injection succeeded, underscoring the ongoing need for targeted manual penetration testing.