Reflections from OCP 2026 - The Enterprise AI Gap in Infrastructure Standards
Published: November 13, 2026
Bottom line: OCP Global Summit 2026 focused on hyperscale AI infrastructure and largely left out enterprises, which will run a growing share of everyday AI inference. Enterprises need right-sized reference designs, inference-aware networking, simple operations, built-in security and sovereignty, and transparent economics from the OCP community and their suppliers.
What did OCP Global Summit 2026 show about AI infrastructure?
The Open Compute Project (OCP) Global Summit 2026 in San Jose in mid-October showed an industry building AI infrastructure at gigawatt scale. It also showed a conversation aimed almost entirely at hyperscalers, with enterprises, the organizations that will run a growing share of everyday AI inference, mostly missing from the stage and the talks. OCP is now extending its standards well beyond servers and racks into silicon, cooling, power, facilities, and the utility grid, multi-gigawatt accelerator deployments, 100,000-GPU clusters, 250 kW racks, and 800 VDC power distribution.
Enterprise voices were rare. Some vendors touched briefly on enterprise needs, and OCP said it plans to extend hyperscale-derived standards to enterprises. Those were useful signals, but they were footnotes next to the main program.
This blog covers what I saw in AI hardware and software, how training and inference differ at the infrastructure level, what the economics of tokens look like on premises versus at a hyperscaler, and why sovereignty and security are pushing enterprises to ask for more.
What did the AI hardware announcements mean for enterprises?
Hardware announcements centered on four themes: accelerators built around inference, faster open Ethernet fabrics, higher-voltage power, and more serious cooling. Open Ethernet is winning the argument. The ESUN (Ethernet for Scale-Up Networking) effort has grown to more than 175 organizations.This is where enterprises need guidance and standards. None of these designs maps cleanly to an enterprise data center. Many enterprise data centers were designed for air-cooled racks at a fraction of 250 kW, and few enterprises have their own grid-interface engineering or 800 VDC plant. Enterprises need the same open-standards benefits, but in smaller, air-cooled or hybrid-cooled, power-constrained building blocks they can deploy at existing sites.
The result: Enterprises need open-standards benefits in smaller, air-cooled or hybrid-cooled building blocks that fit existing sites.
What is missing from AI software for enterprise infrastructure teams?
Software was the quieter half of the summit, but it carried the same message: open, automated, and built for very large operators. Hyperscaler announcements included things like open-source RDMA transport for AI Ethernet fabrics. Some vendors also highlighted validated SONiC for AI networks and a planned various new approaches to shorten the exposure window between a vulnerability disclosure and a patch.
Two software trends stood out. First, agentic AI is changing the infrastructure mix. AMD said inference is about 60% of AI compute and agentic traffic grew roughly 14-fold in 2026, which is why CPUs, orchestration, and isolated execution environments for agents featured so heavily. Second, operations are heading toward autonomy. Google described using AI for hardware design, data center operations, and capacity planning, and speakers described infrastructure that detects and predicts failures with less human intervention.
Most of this tooling assumes a large, specialized platform team, so there is an enterprise gap here. A typical enterprise network and infrastructure group must run AI alongside everything else, across multiple vendors, often with a small staff. What they need is simple cloud-managed or on-premises operations, open interfaces, and built-in security. That kind of operational simplicity was largely left unaddressed at the summit.
The result: Enterprise teams need simple, open, secure operations that a small staff can run, and the summit largely left that need unaddressed.
How do AI training and inference differ at the infrastructure level?
Training and inference stress infrastructure in nearly opposite ways, and the summit's design center was clearly training-class and hyperscale inference. Some of the hyperscalers put the training case plainly: “AI training moves forward at the speed of the slowest conversation that you can have between two GPUs." Inference is a different job, an always-on service that has to answer quickly, cheaply, and reliably for many users at once.
| Dimension | Training | Inference |
|---|---|---|
| Primary goal | Shortest time to a finished model | Low latency, high availability, lowest cost per token |
| Typical scale | Thousands to more than 100,000 accelerators in one tightly coupled cluster | One accelerator to a few racks per model replica; scales out by adding replicas |
| Network | Non-blocking, lossless back-end fabric; scale-up, scale-out, and scale-across; collective operations dominate | Mostly client-to-server traffic; moderate east-west traffic for sharded models and caches |
| Failure behavior | A single failure can stall the whole job; frequent checkpointing | Failures isolated to a replica; load balancers reroute |
| Load pattern | Long, scheduled runs at very high utilization | Continuous, diurnal, and spiky; autoscaling matters |
| Power and cooling | Densest racks; liquid cooling increasingly required | Wide range; smaller models can often run on air-cooled or hybrid-cooled systems |
| Data | Large datasets and checkpoints streamed at high bandwidth | Model weights, vector databases, and caches placed near applications and data |
| Location | A few large, centralized sites | Close to users, data, and applications: cloud, colocation, edge, and enterprise sites |
The distinction matters because most enterprises do not pretrain frontier models. They fine-tune, retrieve, and serve models against their own data, which makes inference the workload that lands in their data centers. Some of the vendors described inference as the dominant AI workload in 2026, which means the enterprise requirement is not a scaled-down training cluster. It is a right-sized, secure, and operationally simple inference platform, with network and power designs that match.
The result: Enterprises need an inference platform sized for their sites and teams, not a smaller copy of a training cluster.
Is on-premises AI inference cheaper than a hyperscaler's cloud?
Owning inference capacity can be materially cheaper per token than renting it, but only when utilization is high and steady. While the unit price of AI tokens is falling, overall enterprise spending is rising. An AI factory can become the most cost-effective option once token production passes a threshold. In one simulation, an AI factory could deliver more than 50% savings over three years compared with API-based and neocloud options, and about half of an AI factory's cost comes from items other than GPUs, such as networking, power, cooling, facilities, and software. The breakeven math is where the story gets practical.
In practice, it comes down to utilization. Steady, predictable inference on a stable set of models tends to favor on-premises or co-located capacity. Bursty experimentation, short training runs, and unpredictable demand tend to favor the cloud. Most enterprises will land on a hybrid model, and network capacity, power, and operations tooling are a large share of the bill either way.
The result: Utilization decides the answer: steady inference tends to favor on-premises or collocated capacity, bursty demand tends to favor cloud, and most enterprises will land on a hybrid model.
Why do sovereignty and security push enterprises to own the AI stack?
For many enterprises, the case for running AI on premises is less about cost per token and more about control. In one survey it was found that 91% of respondents would choose on-premises, private cloud, or hybrid over public cloud for AI involving sensitive data, and 58% said data residency concerns had delayed or scaled back an AI initiative. Coverage of the same research describes sovereignty as moving from a compliance concern to a board-level priority, with financial services and legal firms exposed through rules such as GDPR and PCI-DSS when sensitive records are sent to external AI services.
Sovereignty has several layers that enterprises need to separate:- Data residency - where prompts, retrieved documents, embeddings, and logs are stored and processed
- Operational control - who can administer the infrastructure, update firmware, and access telemetry
- Legal jurisdiction - which governments can compel access to the provider or its data
- Technology choice - whether open standards and multivendor options keep an exit path available
Security adds a second layer of pressure, as in the same survey, 74% of respondents called shadow AI, meaning employees using unapproved cloud AI tools, a critical or significant concern, and nearly one in four reported documented incidents of confidential data uploaded to external AI services. Only about half had put controls in place to prevent it.
For enterprises, the network is where sovereignty and security meet. That means segmentation to keep AI workloads and data separated, identity-based access for users, devices, and agents, and consistent policy across on-premises, colocation, and cloud environments.
The result: The network is where sovereignty and security meet, so segmentation, identity-based access, and consistent policy matter across on-premises, colocation, and cloud.
What do enterprises need next from the OCP community and their suppliers?
OCP's open, collaborative model has done a great deal for hyperscalers. The next step is to carry that model to the organizations that will run AI inside hospitals, banks, factories, campuses, and government agencies. Based on what we saw, enterprises need five things from the OCP community and from their suppliers:
- Right-sized reference designs - inference-first racks and pods that fit existing power and cooling envelopes, with a clear path to liquid cooling when it is needed
- Inference-aware networking - open Ethernet fabrics sized for enterprise clusters, not only for 100,000-GPU training jobs
- Operational simplicity - AI-assisted, cloud-managed or on-premises operations that a small team can run
- Security and sovereignty by design - identity-based access, microsegmentation, and policy that stays consistent across on-premises, colocation, and cloud
- Transparent economics - shared tokenomics methods that include networking, power, cooling, and operations, not only accelerators
The focus for the enterprise AI solution providers should be on helping enterprises build simple, secure networks with a choice of cloud or on-premises management, so organizations can place each AI workload where it makes the most sense for cost, performance, and control. As open AI infrastructure standards mature, we would like to see enterprise voices help shape them.
The result: Enterprises need right-sized designs, inference-aware networking, simple operations, built-in security and sovereignty, and shared cost methods.
If 2026 was the year of the AI factory, the year ahead can be the year those lessons reach the rest of the market including enterprises.
Key takeaways
- OCP Global Summit 2026 centered on hyperscale AI infrastructure, with few enterprise voices on stage.
- Enterprise inference needs right-sized, air-cooled or hybrid-cooled, power-constrained building blocks, not scaled-down training clusters.
- On-premises inference tends to cost less per token when utilization is high and steady, and most enterprises will land on a hybrid model.
- Sovereignty and security requirements make segmentation, identity-based access, and consistent policy across environments central to enterprise AI networks.
Frequently Asked Questions (FAQs)
Q: What is the enterprise AI gap?
A: The enterprise AI gap is the distance between AI infrastructure standards built for hyperscalers and what enterprises can deploy at existing sites with small teams. At OCP Global Summit 2026, most announcements targeted hyperscale training and inference, and enterprise needs received only brief attention.
Q: How is AI inference different from AI training?
A: Training aims for the shortest time to a finished model and runs on large, tightly coupled clusters where a single failure can stall the job. Inference is an always-on service built for low latency, high availability, and low cost per token, and it scales out by adding replicas close to users, data, and applications.
Q: Why don't hyperscale AI designs fit most enterprise data centers?
A: Hyperscale AI designs assume thousands or even tens of thousands of GPUs, along with 250 kW racks, liquid cooling, and 800 VDC power distribution. Most enterprises will have only tens or hundreds of GPUs and operate air-cooled data centers with far lower power densities, making hyperscale designs impractical and unnecessarily complex for their environments
Q: Is on-premises AI inference cheaper than using the cloud?
A: It can be when utilization is high and steady. In one simulation, an AI factory could deliver more than 50% savings over three years compared with API-based and neocloud options. Bursty or unpredictable demand tends to favor the cloud, and most enterprises will land on a hybrid model.
Q: What is AI sovereignty and what does it include?
A: AI sovereignty is control over where AI data is processed, who operates the infrastructure, which legal jurisdictions apply, and whether open standards keep an exit path available. Its four layers are data residency, operational control, legal jurisdiction, and technology choice.
Q: What is shadow AI and why does it matter?
A: Shadow AI is employees using unapproved cloud AI tools. In one survey, 74% of respondents called it a critical or significant concern, and nearly one in four reported documented incidents of confidential data uploaded to external AI services. Only about half had controls in place to prevent it.
Q: What is ESUN?
A: ESUN stands for Ethernet for Scale-Up Networking, an open Ethernet effort for AI infrastructure. It has grown to more than 175 organizations.
Q: What do enterprises need from the OCP community and their suppliers?
A: Enterprises need five things: right-sized reference designs, inference-aware networking, operational simplicity, security and sovereignty by design, and transparent economics that include networking, power, cooling, and operations.