Sacrificing the IaC Sacred Cow for my Birthday

It’s an auspicious day today, both because it’s my 55th birthday and because it’s the birthday of Swamp Club! A big congrats to the crew on their successful fundraise¹ and an amazing product. Adam Jacob is a friend and a long-time partner in crime (for the record, I have no stake in Swamp). We rode through the early Puppet days in late 2006 and early 2007, before the world had coined the terms DevOps or infrastructure-as-code. The main problem Puppet had early on was non-deterministic runs: each run could differ depending on how its dependency graph resolved, so different runs failed in different ways. That led to Adam’s creation of Chef, which solved many of those early Puppet issues. Together, Chef and Puppet drove a renaissance in automation for IT operations. Automation has always been near and dear to my heart and a part of every mission I’ve been on over the years. I’ve been wondering for a while:

What comes after IaC, DevOps, and the like? In particular, how do AI agents change the landscape, as they surely will?

I don’t know if the answer to “what’s after IaC?” is Swamp or something like it, but I think Adam and his co-founders (shout out to the rest of the “SWAMP”²) have all zeroed in on my own conclusion: if AI agents are best at writing code, then why don’t they simply write the automation used to run complex systems? Why use Claude or Codex directly to manage systems, burning tokens religiously at the GPU altar, rather than using them to build deterministic workflows? Do it once with Claude? OK. Do it twice with Claude? Why isn’t it a workflow? Do it thrice with Claude? Shame on you.

The Swamp crew has a name for this: behavior centric AI design, as opposed to the agent centric design most of the industry is building today. In an agent centric system the LLM is the heart of the program. You hand a frontier model context and tools, and it decides the control flow at every step. That’s tolerable for the person who built it, who knows why their trip planner occasionally forgets to book the hotel. It falls apart as you add users and responsibilities, because the fix for a fragile agent is often more agents checking the first agent’s work, and the token bill balloons right along with them. They call it the ROI death spiral, and I suspect it’s part of why Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027.³

Behavior centric design flips that around. You use frontier intelligence at build time, through a coding harness like Claude Code or Codex, to encode your intent into reusable “behaviors”: typed inputs and outputs, structured and versioned data, their own isolated credentials, composed into deterministic workflows. A model only gets called at runtime when a step genuinely needs judgment, and even then it’s scoped to that one job; it never drives the control flow. In their own toy example, the agent centric version of a trip planner used 14x more tokens and cost 6x more per plan than the behavior centric one.⁴ And every behavior you build is one more piece the next workflow can reuse, so the library compounds instead of the token bill.

I think this also dovetails with where software is going in general: given the cost of writing software is driving towards zero⁵ and that Jevons paradox has never been disproven, we can see a coming era of “form-fit software.” I’ll be writing more on this soon, but put simply: if you only use 10% of a given piece of software, why don’t you just use only that 10%? Do you really need the rest of it, or is it baggage?

Recently, I had the pleasure of testing these ideas by building a semi-autonomous GPU cluster: bootstrapping the initial cluster to the point where it could use open weights models, then building a library of Swamp workflows⁶ that finished building the cluster and went on to perform many of the cluster’s ongoing day-2 operations: deploying passthrough GPU VMs, managing GPU switch fabrics, imaging bare metal, managing Ceph cluster operations, provisioning end users, and so much more.

On a personal note, as I ran into problems using Swamp, I’ve been filing issues with the team, and they are usually fixed in less than 24 hours!

One quite interesting approach I took in building my autonomous GPU cluster was to create a full digital twin where I could test new Swamp workflows or upgrades. The digital twin was an isolated, faithful representation of the production system (including IP addresses, MACs, Redfish interfaces, etc.) using nested virtualization and oversubscription. This allowed building staging environments on demand, testing upgrades or changes to system state, and gave AI agents a playground to freestyle in. This kind of environment is traditionally challenging to create and expensive to upkeep, but again, we need to rethink operations in this brave new world, and AI agents are going to change the way we do IT ops, sooner or later. I’ll write up more of what I learned building it next week, in Lessons Learned in Building Autonomic GPU Clusters.

While IaC solutions such as Ansible, Terraform, Puppet, and Chef have their place, they can’t do everything. Running systems autonomously at scale requires autonomic functions. You’re building a holistic system, not individual boxes. Only the builders of a system understand its function and desired operating state. IaC attempts to describe a system and make it adhere to a specific state, but it mostly ends there. What about all of the other day-2 operations? Troubleshooting, diagnostic dumps, provisioning of new systems and users, reporting, looking for other signs of drift, watching for new CVEs, automatically assessing them, and crafting a response?

AI agents, empowered by Swamp or Swamp-like solutions, can learn and iterate on a system over time, orchestrated and overseen by human operators. Working together, we have an opportunity to break into this next era of Agentic Ops.

Happy birthday, Swamp!