Herding Pets

Over the past year I’ve had the opportunity to build or operate a number of different GPU clusters with a wide variety of NVIDIA configurations: Hoppers, Blackwells, InfiniBand, RoCE, with and without BlueField DPUs. Through that experience, I had a chance to begin forming an opinion around my old pets vs. cattle meme and its applicability to today’s AI factories. Infrastructure is definitely about to change, particularly for AI factories. It’s clear that while much of the intent behind the notion of pets vs. cattle holds true, it is no longer an either/or. Rather, we find ourselves moving into a unique future where we are herding pets, rather than cattle.

Let me explain.

Shooting Cattle?#

When I started pushing the pets vs. cattle meme around 2011 or 2012, the pitch was simple. In the old world, servers were pets: Bob the mail server, lovingly maintained and never allowed to die. In the new world, servers were numbered cattle. When one got sick, you took it out back, shot it, and replaced it.¹

What I cared about was disposability: could you destroy and replace a server without bringing down the service?²

Even then, I knew the meme would eventually lose its relevance. I thought containers and serverless would make the distinction less interesting. I wasn’t thinking about, and hadn’t foreseen, a $400,000 GPU server.

Here’s what got lost: the meme never said cattle had to be cheap. Commodity x86 servers made disposability economical, and applications absorbed failures so individual boxes didn’t matter. Hardware cost and architectural disposability go hand in hand in hyperscaler (cloud) land.

In an AI factory, the picture is much more complex than a traditional disaggregated white-box-based hyperscaler architecture. The servers are fantastically expensive, they get sick in surprising ways, and one sick animal can impact the entire herd.

Return of the Supercomputer#

An AI factory isn’t merely like a supercomputer. For large-scale training, it is one: a tightly coupled HPC system in which thousands of accelerators advance through synchronized work. As with traditional MPI workloads, the interconnect must be fast and tightly coupled enough for the cluster to behave as a single machine. Collectives such as all-reduce, together with model parallelism, make progress depend on communication and, often, on the slowest participant.

We’re building these systems at cloud scale and trying to operate them with cloud-era assumptions.

Inside a GPU node, NVLink and NVSwitch tie accelerators together. Between nodes, InfiniBand or RoCE fabrics move enormous amounts of data, often across multiple independent rails. Add PCIe, NICs, DPUs, host networks, and storage, and you have layers of interdependence that would make a traditional web farm operator twitch.

The cloud era deliberately pushed in the opposite direction. Shared-nothing systems shrank failure domains. Lose one server, lose roughly one server’s worth of capacity. AI factories bring back tight coupling because the workloads demand it. You can’t fix synchronous training with a load balancer and a few retries.

Big failure domains are back. This time, the blast radius is measured in GPU-hours and dollars.

Gold-plated Cattle, the new Pets#

In The Rancher’s Dilemma, I argued that cloud-native resilience let us stop gold-plating infrastructure. Put availability in the application and run on commodity hardware. The unstated assumption was that the hardware was cheap enough to be disposable.

Now consider the price tags:

• A BlueField-3 DPU can run roughly $2,600 to more than $11,000, depending on model and supplier. That’s an entire cloud-era white-box server just for a SuperNIC, and some GPU nodes have more than one.
• An eight-H200 GPU node runs $300–400K. That’s a rack’s worth of yesterday’s cattle, or more.
• A single accelerator can cost tens of thousands of dollars. And a failed cable, optic (~$1,500), or DPU can strand the productivity of far more expensive equipment.

Now hardware is precious. Yet still the software must treat it as logically disposable. In the original cattle era, those ideas mostly aligned. Now they’ve split apart.

The arithmetic of a sick herd#

Imagine a 1,000-node factory at $350,000 per node: $350 million in GPU servers alone, before networking, power, and buildings. How much of that capital is actually doing useful work at any given moment?

Outright downtime is the small number. NCSA measured GPU-node availability on the University of Illinois Delta system at roughly 99.3% to 99.4%, so on average our factory has six or seven nodes out of service at any given time. Annoying, but not the problem.

The problem is what failures do to the jobs. When Meta trained Llama 3 on 16,384 H100s, they logged 419 unexpected interruptions in 54 days, roughly one every three hours, about 78% of them hardware-related. Meta still reported “higher than 90%” effective training time, which is a very good result, and it still means up to a tenth of the run went to something other than training: failures, restarts, recovery, and planned maintenance.³

If you lose even 3% to 10% of productive capacity, which looks like a realistic band for shops with world-class reliability engineering,⁴ that’s the equivalent of $10.5 million to $35 million of hardware doing no useful work. Because the remaining capacity must carry the capital cost of all of it, capital cost per productive unit rises by as much as 11.1%. The curve is convex: at 20% lost, it rises 25%. At 30%, nearly 43%.

And that ignores lost output, energy, cooling, and the healthy GPUs sitting idle because a sick one is holding up the job.

AI factory failures hit different#

The cattle model worked best when a machine was either alive or dead. Dead? Replace it. Alive? Give it work.

In AI factories, some of the most expensive machines are the ones that keep running badly. Call it degraded-but-not-dead infrastructure.

These aren’t exotic edge cases. NVIDIA documents Xid 79, and DCGM exposes PCIe widths and generations, replay counters, NVLink state, and diagnostics. The signals exist. Making sense of them is the hard part.

Take PCIe downtraining. Dropping from x16 to x8 roughly halves the link’s nominal bandwidth. It does not automatically halve GPU throughput. A compute-bound workload might barely notice; an inference workload moving substantial data across that bus takes a serious hit.⁵

Here’s what that looks like in real life. On one brand-new eight-GPU node, the PCIe switch uplink in front of one of the GPUs trained at x8 instead of x16. The server’s BMC logged it on every single boot. Yet the obvious health check, nvidia-smi, reported a perfectly healthy x16 link, because it reads the GPU’s own link, not the switch hop upstream of it. The problem only showed up when we measured GPU-to-GPU bandwidth: traffic between GPUs on either side of that hop ran at roughly half the speed of an identical node (about 19 GB/s versus 35 GB/s), which is about what an x8 hop would predict. An earlier bandwidth investigation had actually ruled out link training, because it checked the wrong link. Re-imaging the node didn’t fix it, and it was still there two months after we first spotted it.⁶

When the malaise hits the herd#

In hyperscaler cloud environments, a certain amount of sharding, blast radius containment, and a design-for-failure mentality insulate you from large-scale failures. If you are servicing millions of users and a pod/cluster/shard fails, the overall system fails gracefully, where only a small subset of users see the outage or service impact. Running large-scale training workloads changes that. A service degradation can have a large ripple effect.

In synchronized AI training, a step often can’t finish until the slowest participant does. One throttled GPU, flaky interconnect, or degraded fabric path can force hundreds or thousands of healthy GPUs to wait. HPC operators call these stragglers or fail-slows. They’ve understood the problem for decades; cloud operators are now getting reacquainted with it.

The classic mechanism of route around failures in hyperscaler environments winds up being a much harder problem to solve when it’s less about an individual server and more about a fully rail-optimized data path, touching NUMA islands, PCIe switches, high-speed NICs, and RoCE/InfiniBand fabrics. Routing around a failure may mean routing around an entire gold-plated server and all of its aggregate throughput potential.

The evidence isn’t merely anecdotal. GREYHOUND, a 2025 USENIX ATC paper studying a production cluster with more than 10,000 GPUs, found fail-slow events in 16 of 27 large training jobs examined, with an average completion-time multiplier of 1.34x.⁷ A roughly one-third increase in completion time for ostensibly running jobs should get anyone’s attention.

The sick animals also tend to be the same animals. On one of Meta’s research clusters, a small set of what they call “lemon” nodes, about 1.2% of the cluster, touched 13% of daily jobs. Detecting and pulling those nodes cut the failure rate for large (512+ GPU) jobs from 14% to 4%.

Some of the sickness is there from day one. Microsoft’s SuperBench flagged 10.36% of nodes as defective, a mix of outright failures and performance regressions, during build-out validation of more than 24,000 A100s, before any of them saw production.⁸

Some hardware keeps running and quietly gets the math wrong. ByteDance reports that standard synthetic tests miss over 60% of the GPUs that cause silent data corruption (SDC) in production LLM training.⁹ A GPU producing bad results may never throw an Xid. It can quietly corrupt a training run.

I’ve watched one sick animal take down its neighbors. On another node, a single GPU fell off the PCIe bus three times in ten days, and each time the other GPUs sharing its NVLink domain started throwing fatal NVLink errors right behind it. One of those incidents, under live serving load, crashed the whole host before the kernel’s error messages ever reached its logs; the server’s BMC caught what the host didn’t.¹⁰

Network failures can compound the problem. Losing a rail may interrupt a job, or the remaining paths may carry the traffic at reduced performance. Multi-rail designs improve resilience; they don’t make degradation disappear.¹¹

The most expensive part of a sick GPU may not be the GPU itself. It’s all the healthy hardware waiting for it, depreciating, consuming power, and delivering less useful work.

This is also why I’d resist framing the business goal as revenue per token. Pricing and workload mix largely determine that number. Operational reliability improves revenue per installed GPU-hour, cost per delivered token, and ultimately contribution margin per token.¹² The point is to extract more productive output from capital you’ve already bought.

Pets vs Cattle?#

There are really three kinds of disposability:

• Economic: can I afford to replace or sideline this machine?
• Architectural: can the workload survive losing it?
• Operational: can my systems isolate and substitute it quickly?

Cheap cloud servers made those three questions look like one. AI factories force us to reevaluate them individually. A GPU node may be economically precious, architecturally replaceable for some workloads, and operationally painful to replace. A software worker may be disposable while the physical accelerator underneath it needs careful diagnosis and repair.

But here’s the counterintuitive part: expensive hardware is not a reason to nurse a degraded node along in production. If it is slowing a thousand healthy accelerators, the opportunity cost of keeping it active can dwarf the cost of quarantining it. Take it out of the job, keep the rest productive, and fix it off the critical path wherever possible.

To be clear, the software should still treat these machines as cattle: logically disposable, quarantined quickly, and repaired off the critical path. What breaks is the assumption that one decision covers all three kinds of disposability. Pulling a node is cheap for the scheduler, expensive for the CFO, and slow for the operator, and someone (or something) has to weigh all three at once.

The meme is breaking under pressure and demands a rethink.

A new Rancher’s Dilemma?#

The original rancher’s dilemma was about pets and cattle sharing infrastructure, and the danger of letting competing layers fight over availability decisions.

Now we’ve got GPU firmware, PCIe, network fabrics, cluster schedulers, training frameworks, model replicas, and inference routers, each with its own definition of healthy. DCGM might see a downtrained link. The scheduler sees a ready node. The training job sees a straggler. The business sees millions in underproductive capital.

No single layer sees the whole picture.¹³ Worse, several layers may try to repair the same problem independently.

That is the real operational challenge of the AI factory: building an entire new software stack and operational model that can understand failure in terms of its effect on the workload and the economics, then act fast enough to matter. Sometimes that means repair. Sometimes isolation, substitution, or routing around a fault. Sometimes it means leaving a harmless degradation alone. The right answer depends on the workload, the topology, and the cost of doing nothing.

We cannot afford to throw away these machines like cheap cattle. We also cannot afford to hand-tend every sick GPU, DPU, and network rail while the factory bleeds productive capacity.

So we need a new kind of rancher: automation that can connect symptoms to impact and coordinate decisions from bare metal all the way up to the token factory. Agentic operations may finally make that practical.

We spent a decade learning how to shoot the cattle. Now we have to keep a herd of expensive, temperamental pets productive without letting one sick animal hold up the rest.

Welcome to the era of pet herding.

Footnotes

1. Framing it this way has caused no end of furors, but it was meant in jest. I guess now the cattle are also having their revenge? :) ↩

2. I borrowed the analogy from Bill Baker at Microsoft, who used it to describe scale-up versus scale-out SQL Server, and bent it toward disposability. I wrote up the history in 2016. For the record, Tim Bell has been gracious about the lineage: CERN’s presentation of pets and cattle was derived from mine, and Bill Baker’s before that. ↩

3. ByteDance, with a heavily automated detection and recovery stack, reports 97% effective training time on a three-month, 9,600-GPU job. Meta’s research clusters show the skew: hardware failures touched only 0.2% of jobs but 18.7% of runtime, because the big, long-running jobs are the ones that get hit. ↩

4. To be precise, these sources measure different things in different environments: node availability, effective training time, and share of runtime affected. None of them is an industry-wide failure rate. NCSA did run a projection, though: holding 99.9% availability at the job level would take roughly 5% overprovisioning, which in their example would cost over $1 million per month. ↩

5. It gets worse. A GPU may legitimately report a lower PCIe link generation when it isn’t busy (NVIDIA’s nvidia-smi notes say these values “may be reduced when the GPU is not in use”). So a naive health check can mistake normal behavior for a fault, or a persistent fault for a healthy, schedulable machine. You have to know what the workload is doing, not just what the hardware reports. ↩

6. And this wasn’t for lack of trying. The hardware vendor made five separate trips to try to fix PCIe downtraining issues with one or more GPUs on these brand-new H200 systems. ↩

7. That’s a result for the studied jobs, not a fleet-wide GPU failure rate. The cluster is Alibaba’s, and the paper is a joint HKUST and Alibaba effort. Worth noting: the paper attributes most of those slowdowns (13 of the 16 jobs) to network congestion rather than sick hardware, which is a systems problem in its own right. ↩

8. Memory is the usual suspect. The same paper attributes 64.4% of a month of Azure GPU-infrastructure incident tickets to GPU memory. Newer parts aren’t necessarily better here either: NCSA measured a 3.2x lower per-GPU mean time between memory errors for H100s than for A100s, which the authors attribute at least partly to there being more memory per GPU to go wrong. ↩

9. A separate Tsinghua and ByteDance team found 18 SDC incidents traced to 13 faulty GPUs across 35 million GPU-hours. ↩

10. No GPU alert fired for any of it. Every one of those faults reached a human because a tenant reported it or someone went digging through logs. The signals existed (see DCGM above); nothing was connecting them to a person, let alone to a decision. ↩

11. Microsoft’s SuperBench team found that once more than half of the redundant InfiniBand uplinks on some top-of-rack switches were down, all-reduce bandwidth degraded significantly, even though the fabric was technically still up. ↩

12. For training, the comparable measures are useful training progress per GPU-hour and time-to-model. ↩

13. A small example from the same clusters: with the GPUs passed through to tenant VMs, the host couldn’t run nvidia-smi at all, and the infrastructure team wasn’t allowed to look inside the tenant’s VMs. The tenant saw the Xids; from the host, the operator saw nothing at all. ↩