Filling Empty Cathedrals

 

Why the $600 billion AI buildout only pays off when the machines go to work, the rapid rise of inference!

 

Speed Read

• The crossover already happened. Enterprises now spend more running models than building them. Nobody announced it. It simply became true.
• Agents are the demand. One agentic workflow burns roughly 30x the inference of a chatbot exchange. Automation is a furnace with no off switch.
• Modernization fills the racks: Legacy migration, research, trading, and cyber defense, alongside the quiet in-housing of open-weight models as tokens fall toward the price of electricity.
• The winners will not be the firms that bought the most AI. They will be the ones that built a machine-readable map of their own business, chose their model bridge deliberately, and organized their agents like an institution instead of a junk drawer.

Part I: The cathedral problem

Drive across northern Virginia, central Ohio, the outskirts of Phoenix, or the flatlands past Abilene and you will see them rising: windowless halls the size of aircraft carriers, ringed by substations, fed by their own gas turbines. The four largest US hyperscalers will commit north of $600 billion in capital expenditure this year, most of it aimed at AI. Not the Vanderbilt railroads, not the Carnegie steel mills, not the Rockefeller oil fields, not the fiber boom of the 1990s: nothing in the history of private industry has moved this much capital this fast.

These are cathedrals, and they were built on faith. Like the medieval originals, they were designed for a primary purpose: training. Long, monolithic, compute-saturated runs producing a frontier model the way a foundry produces a bell.

Here is the uncomfortable arithmetic. Training is episodic. A handful of labs do it, and every run, however enormous, ends. If these buildings only ever trained models, most would sit dark most of the time, stranded assets waiting for the next run. The bell gets cast once. The cathedral needs a congregation every day, or the collection baskets go empty.

That congregation is inference: a trained model actually doing something. Answering the query. Reading the claim. Refactoring the function. Screening the molecule. Blocking the intrusion. Inference is not a capital expense that ends. It is an operating expense that never turns off.

The industry has quietly conceded the point. Inference is now roughly two-thirds of all AI compute, up from about one-third in 2023. The buildout was the bet. Inference is how the bet pays.

The rest of this piece is about what generates the inference, and what to do about it before your competitors finish reading.

Part II: The physics of the switchover

Training and inference are not the same workload in different clothes. Training is a throughput problem: saturate every GPU with as much arithmetic as possible for as long as possible. Inference is a latency and memory problem: serve a request in milliseconds, keep the model’s working state resident, and do it for millions of concurrent users at a price the customer will tolerate.

The silicon has noticed. Custom accelerators are the fastest-growing line in data center compute spend, and every major chip player has spent eighteen months buying, licensing, or poaching its way into a serving story. The cathedrals are being retooled for the congregation.

But hardware is a second-order effect. The first-order question is what work we are asking these models to do, and there a single number matters more than any chip roadmap.

A simple linear workflow, the 2023-vintage kind where a human asks and a model answers, costs about four cents. A modern agentic workflow, where the model reasons, calls tools, checks its own work, and iterates until a goal is met, costs about $1.20.

A chatbot is a polite intern. An agent is a department that never goes home, never takes PTO, and never stops billing. A single long-horizon coding session now runs eight hours or more of continuous tool-calling, generating more tokens than any human would sit through in a month of conversation. When you automate a process instead of assisting a person, you replace one inference call with hundreds.

Which is why “inference demand” is the wrong headline. The headline is automation. Inference is just the meter on the wall.

Part III: Tokens approach the cost of electricity

Every industrial revolution has a moment when an input becomes too cheap to meter. Steam did it to muscle. Electricity did it to steam. The AI version is underway now, and it has a specific mechanism: open-weight models running on hardware the enterprise controls.

The spread is not subtle. Frontier APIs are priced in dollars per million tokens. Self-hosted open-weight inference, run at genuine utilization, is priced in dimes. At enterprise volume that is not a discount. It is a different unit of account.

And the open models have stopped being a compromise. The current open frontier clears 80 on SWE-bench Verified for under a dollar per million tokens, good enough for production code, document extraction, and structured reasoning across the overwhelming majority of enterprise work. Cost at a given capability level is falling roughly 10x a year. Whatever you are paying today is the most you will ever pay.

Two caveats keep this honest, and both make the case stronger.

Self-hosting only works at volume. An idle H100 is the most expensive token machine ever built, and the operational overhead is real: model updates, evals, on-call, and the engineer who actually understands the serving stack. Break-even sits somewhere between two and five million tokens a day. Below that, you are running a hobby data center with a capex line.

The labs will keep cutting to defend share. Frontier pricing shed most of its value in the last year alone, and it will keep falling.

Put those together and a flywheel appears. Cheap tokens make automation economic. Automation creates volume. Volume makes in-house inference economic. In-house inference makes tokens cheaper still. The firms that automate first arrive at the bottom of the cost curve first, and everyone else pays retail for the privilege of catching up.

When an input’s price approaches its marginal cost, the input stops being the constraint. The constraint moves to whoever can use it. Which brings us to the largest pool of unused leverage in the global economy.

Part IV: Software eats its own legacy

Roughly 220 billion lines of COBOL are still running in production, inside banks, insurers, government agencies, and hospital systems. About a tenth of the people who understand it retire every year. For three decades, “mainframe modernization” meant a five-year program, a nine-figure budget, and a better-than-even chance of quiet cancellation after the second missed milestone. It was the enterprise equivalent of a New Year’s resolution.

Then, on February 23 of this year, a frontier lab announced its model could autonomously explore and analyze legacy COBOL estates. IBM’s stock fell more than 13% in a single session.

The market was not reacting to a feature. It was repricing an entire economic arrangement in which the cost of understanding old code was so high that nobody ever paid it. Could AI finally provide the catalyst to one day retire all of the COBOL Identification, Environment, Data, and Procedure Divisions?

Coding tools were the first mass consumers of inference, and the reason is instructive: software is the one domain where a machine can grade its own homework. Code compiles or it doesn’t. Tests pass or they fail. That closed loop is what lets an agent run for hours with no human in the chair, and it is why legacy migration goes fully agentic first.

The interesting question is not how the old code gets migrated. It is what it gets migrated into. And here the industry is lining up to repeat a mistake it has made before. The naive version takes a 1985 batch process and reproduces it faithfully in Java on Kubernetes. Same logic, same tables, same buried assumptions, now with a REST API and a service mesh.

 

The alternative is a two-layer rebuild. The first layer is the system of record: the authoritative truth about customers, contracts, claims, positions, patients, assets. The second is the system of work: the processes that act on those records, which for fifty years have lived in human heads, email chains, and procedure manuals nobody has opened since 2011. The old world automated the first layer and left the second to

people. The new world automates both, and the second layer is where the agents live.

That only works if the system of record treats machines as first-class users: the semantics of the business made explicit rather than buried in code comments and tribal memory.

Part V: What fills the racks

If modernization is the wave, these are the places where the surf is already breaking. What unites them is simple: each is a loop, not a lookup.

Banking and insurance. Core migration is the obvious one; the durable load comes after. Underwriting agents that read every document in the file instead of the three someone had time for. Claims adjudication that reconciles policy language against evidence. KYC and AML agents that investigate rather than flag and forward. Each case is dozens of model calls, and a large insurer runs millions of cases a year.

Investment banking and asset management. Portfolio construction is going from periodic to continuous. Research agents synthesize filings, transcripts, and alternative data across thousands of names; scenario engines rerun stress tests as conditions move; execution agents hunt liquidity. A desk that once ran its models overnight now runs them every minute, and every run is inference.

Healthcare research. Literature agents that read the corpus instead of a sample. Trial-design agents that draft protocols and simulate enrollment. Screening pipelines that evaluate millions of candidates before a single assay is ordered. The bottleneck in drug discovery was never ideas. It was the cost of evaluating them, and that cost is now a token bill.

Cybersecurity. The arms race with no off switch. Attackers have already automated the full lifecycle, from personalized phishing to autonomous exploitation to adaptive ransom negotiation. Leading agentic platforms already absorb the high-80s percent of Tier 1 triage. Volume here scales with the threat, not with your budget: a sentence every CISO should read twice.

Manufacturing, supply chain, and government. Digital twins that reason instead of merely displaying. Provenance agents that trace a component from fab to field. Case

processing, benefits adjudication, bio-surveillance. The public sector holds more legacy code per capita than any industry and the least tolerance for failure, which is precisely why it will move last and matter most when it does.

Notice the pattern. None of these is a chatbot. Each is a process that used to require a person sitting in a loop, and each becomes an economic proposition only when the loop costs cents instead of hours.

Part VI: Where the value actually accrues

Cheap inputs and expensive incumbents are the conditions under which new companies form. Two archetypes are already visible.

The modernization factory. An AI-native services firm that migrates systems of record with agents doing discovery and translation and humans doing verification and governance. The old IT services model was labor arbitrage. The new one is inference arbitrage, and the economics are unforgiving to anyone still billing by the hour, which is, at the moment, nearly everyone.

Agentic-native software. Companies that never build a human interface, because the operator is an agent. They sell outcomes: per resolved case, per migrated module, never per seat. Their cost of goods sold is a token bill and their moat is how well they understand the domain they automate. Call the category SaaS.AI.

Here is the counterintuitive part, and it is the most important claim in this piece. As models commoditize toward the cost of compute, the model is not where value accrues. Value accrues in the map: the explicit, machine-readable understanding of how a particular enterprise actually works: its entities, relationships, rules, and exceptions. Models are interchangeable and cheaper every month. The map of your business is neither.

Part VII: The progressive Fortune 500 company, 2031

Picture a large financial services firm, or a health system, or a global manufacturer, five years out. Not the average one. The one that moved early.

The mainframe is almost gone, or serving read-only history behind a reference feed. The system of record is a governed semantic layer, an ontology of the business, where agents hold service accounts alongside employees and every record carries provenance. Nobody argues about which spreadsheet is right, because the question stopped making sense.

Inference is a line on the P&L: reported monthly, forecast quarterly, managed by a function that looks like FinOps for tokens. Between 60% and 70% of those tokens are served in-house on open-weight models running on capacity the firm owns or reserves. Frontier APIs are used the way a hospital uses a specialist: for the novel and the hard.

The org chart changed shape more than size. Headcount is roughly flat; throughput is five to ten times what it was. The Tier 1 analyst role, in security, in finance, in customer service, is nearly phased out as a human job. In its place are commanders: people whose work is judgment, escalation, exception handling, and improving the agents they supervise. The audit trail of what the agents decided and why is the most-read document in the company.

And the cathedral outside Abilene is full. Not because someone trained a bigger model, but because ten thousand ordinary processes that used to run on human attention now run on inference, all day, every day.

Part VIII: Three things to do before your competitors do

Most enterprise AI programs are a collection of pilots in search of a thesis. Here is the thesis, in three moves. They are ordered deliberately. Attempting the third without the first is how you end up with an expensive mess that has a very nice dashboard.

1. Build the ontology map before you buy the machines.

Ontology and context are foundational, and they are the part everyone skips because they do not demo well.

An agent reasoning over a well-modeled enterprise is reliable. An agent reasoning over a data swamp inherits the swamp, then automates it, at machine speed, in confident prose. Retrieval is not a substitute for meaning. If four systems each have a “customer” and nobody can say which records are the same human being, no amount of context window will save you.

In practice: make the semantics of your business explicit and machine-readable. What a claim is. How a position relates to a counterparty. Which customer in system A is the customer in systems B, C, and D. What the exceptions are, and who may grant them. Ontology, knowledge graph, semantic layer: the label matters far less than the discipline. Every dollar spent here makes every subsequent inference call more accurate and less expensive. It is the only AI investment that compounds, and the only one your competitors cannot buy off a shelf.

The test: can a new agent be pointed at your business and get a correct account of what an entity means without a human explaining it? If not, you do not have a map. You have documentation.

2. Choose your bridge between closed and open models, deliberately, and now.

The dominant failure mode is drift. Every team reaches for a frontier API because it is easy, the token bill compounds quietly across a dozen budget lines, and eighteen months later you discover you have outsourced your operating margin to a vendor with pricing power over you. The opposite failure is just as expensive: a purity project to self-host everything, staffed by people who have never run a serving stack under load, burning capital on idle accelerators.

The bridge is a routing decision, not a vendor decision. Rent frontier models for the novel, the ambiguous, and the genuinely hard. Smaller open-weight models you own or reserve for the high-volume, well-specified, repetitive work, which is most of it, and which is where the token bill actually lives. One abstraction layer so the choice is made per request and stays reversible, and no application code knows or cares which model answered.

Two numbers govern the decision. Break-even for self-hosting sits between two and five million tokens a day per workload. And cost at a given capability level falls roughly 10x a year, so what cannot be justified in-house today may pencil out in two quarters. Revisit quarterly, not annually.

The test: can you move a production workload from a frontier API to a self-hosted model in a week, without touching application code? If not, you do not have a bridge. You have a dependency.

3. Organize your agents like an institution, not a junk drawer.

This is where most firms will lose the next three years, and it will not look like failure. It will look like success: forty agents, then four hundred, each solving something real, each built by a different team, none of them aware the others exist.

That is not an AI strategy. That is shadow IT with a larger electricity bill: agents duplicating each other’s work, contradicting each other’s conclusions, competing for the same records, and collectively driving toward no vision at all. The hodgepodge never announces itself. It accumulates.

Structure prevents it, and structure means four unglamorous things.

• A registry. Every agent has a declared purpose, a named owner, defined inputs and outputs, and permissions scoped like an employee’s. If you cannot list your agents, you do not have agents. You have incidents in waiting.
• A hierarchy. Agents that do work, agents that coordinate work, humans who own the outcome. Flat swarms are a research demo, not an operating model. Someone (a person, by name) owns every consequential decision path.
• A shared substrate. Every agent reads from and writes to the same governed semantic layer from day one. This is why the map comes first. Agents that carry private copies of the truth will generate private versions of the truth.
• An audit trail as a product. What each agent decided, on what evidence, under whose authority. In 2031 this is the most-read document in the company. Build it now, while the trail is still short.

And measure the layer, not the agent. The right question is never “Is this agent working?” It is “Did this function get faster, cheaper, or better, and can we see why?” Agents that do not move a business metric should be retired without ceremony. Most organizations will find this the hardest instruction in this document, because somebody was promoted for building them.

The question

Not whether to buy more AI. Everyone will buy more AI, and it will distinguish no one.

The question is simpler, and harder.

 

In five years, the firms that answered it will be running on inference. The ones that did not will be writing very polite RFPs to the ones that did.