Sacred CowsFor Connoisseurs of AIconoclasm
Appearance
Sacred Cows
· Randy Bias

Are Neoclouds Really "Hyperscale"?

Hyperscale is a whole other level, and most neoclouds aren't close.

As I’ve been engaged with various parts of the ecosystem, I’ve frequently heard folks referring to neoclouds as “hyperscalers.” I think this is a case of mistaken identity.

I’m not using hyperscale as a synonym for a large GPU count. I mean an operating model in which architecture, software and automation let capacity grow without operational cost, failure exposure and human intervention growing at the same rate.

What gets confused is the difference between the ability to deploy massive amounts of CapEx at scale versus being able to deploy massive amounts of technological prowess (R&D+operations) at scale. These are not the same. Just about anyone can “buy into” the neocloud game. And, in fact, NVIDIA has in many ways made it easier than ever.

By providing reference architectures, NVIDIA makes it much easier for companies with access to large amounts of capital to follow pre-validated patterns. Some of the reference architectures are quite detailed. Because of NDAs I can’t post it, but as an example, NVIDIA provides what is called the Spectrum-X Deployment Guide, which is a very technical specification for how to build out a RoCE fabric using NVIDIA Spectrum-X switches for GPU fabrics, down to topology, IP addressing standards, switch configurations, and more.

But even with all of that, it’s just not enough for a neocloud to be called a “hyperscaler”. The problem isn’t failing to become a hyperscaler; it’s mistaking the hardware purchase for having acquired the capabilities.

This is a well trod road, actually. I talked about it over 10 years ago.

Understanding the Hyperscaler Model#

In 2014, I wrote in Public Cloud Economies of (Web-)Scale Aren’t About Buying Power:

In that light, how can anyone else even pretend to the throne without similar levels of investment? Buying a hosting company is not going to get you there. This isn’t a game of buying power or outsourcing. It’s an innovation game and that’s it. The number of players who can pull this off are vanishingly small.

I could just as easily rewrite this today, replacing “hosting company” with “neocloud.” Same movie, twelve years later, with a much bigger budget.

Earlier, I wrote about how hyperscalers innovate and manage vast numbers of servers per administrator.

Of course, now with AI and agentic coding, perhaps the playing field will be somewhat leveled, but I’m skeptical. The key here is the mindset. Hyperscalers architect their systems differently, build their software differently, run operations differently, and basically think differently.

Your average neocloud doesn’t do this, although there are some bright lights such as CoreWeave and Nebius. In fact, neoclouds are highly challenged operationally and architecturally and it’s not just the speed of build out.

Buying 100,000 GPUs does not make you hyperscale. You become hyperscale through two key mechanisms:

  1. You get big enough that you have to solve infrastructure, software, operations, and security problems in whole new ways.
  2. You invest heavily in R&D (not so easy), rather than just infrastructure (easy).

How to Hyperscale (So to Speak)#

It’s almost certainly outside of the scope of this article to draw up a detailed recipe for success to even get close to hyperscaling, but there are some lessons learned from history that are very clear. Tenets, if you will.

Tenet 1: Software Matters More Than Hardware#

It may be that hyperscalers spend more on hardware, but the software is what glues it together and allows for truly operating “at scale.” The term cloud-native arose, in part, because it was the adoption of scale-out software patterns used by the hyperscalers. “Native to cloud” or “born in the cloud” has some pretty specific connotations. I talked about some of these patterns in Pets vs Cattle: The Elastic Cloud Story. Much of what I wrote is still true today, although in new GPU deployments white boxes are now replaced by rack-scale NVIDIA systems for the most part, so while much is still true, much has changed.

But the importance of software hasn’t.

Neoclouds can’t aspire to becoming hyperscalers by wiring together a bunch of open source software and praying for success.

Tenet 2: Architectural Patterns Matter More Than Software#

This might be splitting hairs, because architectural patterns drive software selection, but the reality is that, to a degree, you could use almost any software, if you deployed it with the correct architectural patterns. Software gives you the capability; architecture decides whether it scales. I’m starting to see anti-patterns arise, like the belief that you can run a hyperscale datacenter as one big failure domain behind a single control plane. Hyperscale GPU datacenters will have many bounded failure domains at different layers, with control planes kept separate from data planes so that running workloads survive a control plane failure, minimizing blast radius, allowing for incremental software version rollouts one domain at a time, and so on.

Hyperscale is first and foremost a set of architectural patterns and best practices, more than it is software or hardware.

Tenet 3: Operations Matters, Especially With GPUs#

GPU-based systems present a unique challenge in that they blend the notion of “Pets” and “Cattle.” Their architectural model is largely scale-out, yet each individual server can cost well over an order of magnitude more than a single white-box server. At $300–400K a pop for an eight-H200 node, having them down or idle is a significant revenue hit.

Importantly, large GPU training systems bring failure scenarios that go beyond those of conventional CPU fleets. In the Llama 3 paper, Meta writes that the complexity and potential failure scenarios of its 16K-GPU training exceeded those of much larger CPU clusters it had operated. A single GPU failure could require restarting the entire job; even a still-functioning but slow GPU could hold back thousands of others. Exacerbating all of this is that larger workloads like model training use many GPU systems at a time and problems in production can directly impact training runs, so blast radiuses are inherently larger even though they use a scale-out model.

During a 54-day snapshot, Meta recorded 419 unexpected job interruptions. Yet it achieved more than 90% effective training time, with significant manual intervention required only three times; the rest were handled by automation. That’s the operational capability I’m talking about.

This means that operations matters. It matters a lot. It’s got to be highly automated, it’s got to adjust in real-time to changing conditions. Token factory and inference stacks have to be highly instrumented. The tight binding of AI workloads to hardware means that in real-time you may need to detect a partially failing GPU or GPU server and route around it, rejiggering the software deployment in real time. For inference, that takes a whole new model for load balancing, as the Linkerd folks found out: which GPU already holds the right cache can matter more than which one is least busy. For training, you can’t route around a bad GPU at all; you have to checkpoint, evict it, and restart the job, ideally with no human involved.

The pieces exist, but there’s nothing baked out there that does this end to end, and humans won’t do it by hand, although they need to be “in the loop” to help.

And it’s not just reliability. Consider the security testing behind SemiAnalysis’s ClusterMAX 3.0 findings. In Most Neoclouds Suck At Security, the team describes a neocloud where tenant separation was enforced only in the Grafana display. The underlying Prometheus API credentials could retrieve other tenants’ logs and metrics, including those of an unnamed country’s national intelligence agency. The researchers say they disclosed the problem and verified a patch. This was cross-tenant monitoring-data exposure; the source does not establish packet sniffing or access to classified documents.

The team discusses the Grafana failure in this interview, at 3:43. This is ultimately a failure in both operational and architectural capability and discipline. It represents the kind of basic tenant-isolation failure that true hyperscale prowess gets right most of the time and by default.

I haven’t even touched on tuning your AI workloads, which is a whole other huge ball of wax.

Tenet 4: Top End Talent Matters#

The first three tenets don’t implement themselves. Somebody has to decide where the boundaries go, make the tradeoffs, and get the software, the architecture, and the operations working as one system.

AI agents are amazing. I use them every day to do coding, operations, and more. They can do all kinds of crazy things. You know what I haven’t seen them do? Systems thinking.1 Holistic systems thinking can’t even really be taught if you ask me. It’s experiential. I’m sure there are courses out there, but until you are in the trenches, you won’t get the scars and tread marks to truly grok systems thinking. It requires that most rare of capabilities: cross-disciplinary mastery (networking, servers, storage, application architecture, and software) combined with an innate awareness of how systems interact across boundaries.

You can find these folks, but they are largely already employed at hyperscalers, extremely well paid, and comfortable. Not only that, but the pool has seemingly been shrinking over time IMHO. One of the keys to hiring these kinds of folks is that they want to work with others of their kind. You won’t build a team of them quickly or easily.

There are probably some other areas I’m forgetting, but that’s the high level blueprint and most neoclouds aren’t following it today.

Where Does That Leave Neoclouds?#

Most neoclouds aren’t destined to become true hyperscalers. Their fate will be more similar to the DigitalOceans, OVHclouds, Hetzners, and the like. Still, with NVIDIA’s focus on delivering their NCP program and reference architectures, the need for true regional sovereign clouds as counterpoints to the hyperscalers, the maturity of cloud-native open source software, and the rise of AI agents for coding, the outcomes will likely be different from the cloud days.

At a guess, this time we will see a robust second tier of neoclouds that deliver a solid portfolio of AI services. They are much more likely to look like the hyperscale infrastructure combined with the hyperscale software and architecture. More like a Together AI than a simple bare metal service. More likely to use technologies that virtualize and extract maximum revenue per GPU.

Most importantly, they will have built out at least some significant R&D capacity and invested in software innovation, both to create differentiation and to pace the hyperscalers as best they can. This is probably part of why IREN acquired Mirantis, for example. IREN’s completion announcement explicitly describes adding the software layer to turn infrastructure into a platform. We can already see investment in this layer with the neocloud leaders, including CoreWeave’s SUNK integration of Slurm and Kubernetes and Nebius’s in-house Soperator software.

So ultimately my message is this:

You don’t need to be a hyperscaler. You can use the branding if it makes you happy, but the market knows it’s not true (expect eye rolls). What you need is to walk the hyperscaler path as far and fast as you can. You need to acknowledge that buying 100,000 GPUs and deploying them is the beginning of your journey, not the end. And if you want to succeed, you need to adopt the hyperscaler tenets I outlined, find outstanding technology partners in addition to NVIDIA, and have a clear strategy to differentiate in the marketplace beyond cost.

Footnotes

  1. More on systems thinking: Michael Goodman’s Systems Thinking: What, Why, When, Where, and How?, Donella Meadows’s Leverage Points: Places to Intervene in a System, and an interview with MIT’s Ed Crawley. ↩

Ask about this blog:
Share on:
Figure
Open original
Ask about this blog post

Nothing is sent anywhere. Copy this prompt, which includes the full post, or download it, then paste or attach it in your assistant.