Meet Idem, An Infrastructure As Data Open Source Project

### Overview In this interview, host Bhartiya speaks with Thomas Hatch (founder of SaltStack and Senior Staff Engineer / Director of Engineering at VMware) about Project Idem, an open-source initiative designed to provide "Infrastructure as Data" and simplify cloud management and automation. Originating as a research endeavor inside SaltStack prior to its acquisition by VMware, Idem was further developed within VMware to shift how teams manage infrastructure by moving beyond pure code-heavy configuration enforcement into an integrated, full-circle data pipeline.

### Key Topics - **Origins of Project Idem**: Started as an internal SaltStack research concept right before VMware acquired the company, motivated by lessons from building SaltStack SecOps tools where scanning products frequently failed to deliver automated, large-scale fixes. - **Infrastructure as Data vs. Infrastructure as Code**: While users can still interface with Idem using Infrastructure as Code (IaC) principles, the engine compiles and reduces execution logic down to structured, malleable raw data under the hood. This eliminates the burden of authoring and maintaining thousands of lines of complex code. - **Plugin-Oriented Programming (POP) and Architecture**: Hatch re-architected concepts from SaltStack's configuration management backplane using Plugin-Oriented Programming (POP). This introduced asynchronous programming capabilities, faster performance, and more flexible handling of orchestration and interdependencies. - **The SRE Crisis**: Modern cloud infrastructure scale has drastically outpaced the industry's ability to hire and train Site Reliability Engineers (SREs). Hatch highlights the need for automation tooling that reduces the cognitive and maintenance load on SREs rather than requiring continuous manual scripting. - **DevOps Evolution and Silo Recreation**: Hatch notes that over the last 15 years, DevOps transitioned from breaking down barriers between developers and operations into creating heavy, slow, and bloated CI/CD pipelines, inadvertently isolating SREs into new organizational silos.

### Technical Details Idem relies on a full-circle data feedback loop: it gathers state and configuration data directly from cloud environments, converts that insight into structured raw data, and directly turns those insights into execution and enforcement. Built on Plugin-Oriented Programming (POP), the system incorporates Salt's modular and pluggable design principles while rewriting the configuration management engine to leverage modern asynchronous patterns, speeding up execution and simplifying complex dependency orchestration.

### Takeaways Project Idem targets the growing complexity of modern cloud operations and the bottleneck facing SRE teams. By reducing infrastructure management to structured data rather than bloated procedural codebases, Idem aims to provide faster, lighter automation that scans, understands, and remedies infrastructure state in a unified cycle.