TL;DR

AI in production is won or lost in the gaps between a promising model run and a production AI system that can reliably validate, route, orchestrate, and decompose work under real-world constraints.

The four gaps between model demos and production AI systems

When I sat down with Barak Lenz, our CTO here at AI21 for a recent episode of YAAP, I tried to do what every host does. I asked him for the TL;DR. I wanted the one liner that would hook the audience. His response was very typical of him.

“I don’t really want to start with a one liner,” he told me. “I don’t like one-liners. They tend to throw away the specific details that actually explain what’s going on.”

It wasn’t surprising if you know him, but a refreshing moment in the AI landscape. In a world of hype, Barak isn’t interested in research papers that look good on Twitter. He is interested in systems that actually run on time, within budget, and without failing silently after thirty minutes of compute.

Our conversation centered on a single, provocative thesis. The industry is looking at average scores on leaderboards, the community is obsessed with yet another OpenClaw version. but production is won or lost in the Gaps. To build what Barak calls an AI Operating System, we have to stop looking at LLMs or Agents as magic boxes and start looking at the distance between current performance and theoretical potential. Here is my breakdown of the four gaps Barak identified, and why they change everything for those of us building in this space.

1. The Validation Gap (The “Oracle” problem)

Barak began by pointing out a gap that is well-known in the literature but rarely solved in production: the Oracle Gap. This is the mathematical difference between Success@1 (getting it right on the first try) and Success@N (getting it right at least once across several attempts).

If you run a model once, you are simply pulling a result from a distribution. But if you run it N times and pick the best one, success rates on reasoning benchmarks often spike from the 70s into the 90s. The gap exists because the system doesn’t always know which of its own answers is correct.

The principle is almost a law: more spend leads to more quality. But it’s intuitive, not trivial. Simply spending more with stronger models or multiple runs doesn’t necessarily work, because the system still has to identify the best answer. You can see it clearly in the chart below: the Oracle line climbs steadily as ensemble size grows, while the Average flatlines, and the Adversary collapses.

Oracle vs. Adversary performance
Figure 1: Oracle vs. Adversary performance (from this paper) across ensemble sizes (N) for Gemini 2.5 Pro, Claude 3.7 Sonnet, GPT-4.1, and a Mixture approach. The Oracle line – representing the best possible pick from N runs – climbs steadily toward 75–80%, while the Adversary (worst pick) collapses toward 25%. The Average flatlines. The gap between Oracle and Average is exactly what better validation could recover.

This isn’t just about “agents” guessing a reasoning path. I’ve seen this exact behavior in the way we handle data retrieval. In our recent research at AI21 on multi-scale indexing, we applied this “Success@N” logic to RAG. Most systems commit to a single chunk size upfront, but we found that different queries fundamentally require different resolutions.

By indexing the same corpus at multiple sizes and using a principled way to aggregate them, we were essentially trying to close that same “Validation Gap.” When we ran “Oracle” experiments – choosing the optimal chunk size per query – we found 20-40% headroom left on the table by traditional, single-path systems. Barak’s point is that we shouldn’t settle for the performance of a single rollout when a verified, multi-path approach is within reach.

2. The Contextualization Gap 

As Barak and I moved deeper into the architecture, I began to see these gaps not as isolated hurdles, but as a logical progression. The Validation Gap we discussed is essentially the delta between a model and repeated calls to that same model. But what if we could choose entirely different models for different inputs?

This is where the Contextualization Gap comes in. Barak’s core insight here is that leaderboards are fundamentally misleading because they report what is best on average, while real systems operate on specific, wildly varying inputs.

“This gap is the difference between what’s best on average and what’s best for a specific input.”

SWE-bench: Per-instance resolution rates across four models
Figure 2: Each column is a benchmark problem; green means a model solved it. Notice how GPT-5 Mini (56%) and the frontier models (70–76%) share a massive overlapping green block on the left – the “easy” problems every model gets right. The expensive flagship models earn their keep only on the harder right-hand side. Routing the easy problems to the cheaper model wouldn’t cost you much quality – but it would save you a fortune.

When you look at benchmarks input-by-input, a pattern emerges: easier problems are often solved by much cheaper, smaller models with almost perfect overlap with the flagship ones. In practice, this means a massive fraction of your compute budget is likely being wasted on “overkill” models for tasks that never required them.

We can see the data for this in the way LMArena has always maintained a division of different tasks and specific category scoring alongside overall rankings. The “best” model overall isn’t always the best – or the most necessary – for every specific sub-task.

To me, the missing piece in the current conversation is a clear presentation of where this gap comes from and what our other options are. It isn’t just about running multiple models or agents on the same input and choosing between them after the fact. Even if we had a perfect validator, that approach is incredibly wasteful.

The real question we should be asking is: What if we could know which path to choose during the run and not after?

The solution Barak proposes is to move away from a monolithic “agent” and toward a Portfolio of Actions. This requires an “Action Model”: an execution strategy that estimates the value and cost of different paths before taking them. If an input is easy, the system routes it to the cheapest model. If it’s hard, we spend the budget only where it actually moves the needle.

3. The Latency Gap

Once you’ve figured out the quality and routing of a system, you are still left with the twin pressures of latency and cost. Barak draws a distinction often missed: latency and dollar cost are not the same constraint.

If your goal is only to save money, you run things sequentially. But if you want to save time, you must run things in parallel and use early termination, killing every other process the moment a single one passes validation. Barak defines the Latency Gap as the delta between the average trajectory and the shortest successful path. That shortest path is often three times faster than the average.

Most systems today are “depth-first,” taking long, sequential steps token-by-token. To close this gap, an AI system needs to move beyond simple inference and into true orchestration: managing timeouts, retries, and the ability to kill work that no longer makes sense.

The fix is to separate orchestration logic from LLM reasoning. By using structured plans to improve context engineering rather than just raw ReAct loops, we can enforce parallel execution and step-level dynamic scaling. Structured plans front-load understanding; ReAct loops get stuck trying to fix before they comprehend.

Example of predefined plan vs ReAct
A breakdown of the different steps across the major plan parts on an example sample from the benchmark. The ReAct steps were manually allocated by reading its outputs.

4. The Decomposition Gap

The fourth gap is the most significant. It’s also the hardest to close, because it’s not purely an engineering problem. Most systems today decompose a task only at the top level, then execute the rest sequentially, never asking at each intermediate step: should I branch? Should I stop? Should I route this sub-task to a cheaper model?

Barak argues that if you decompose fractally throughout execution, your capabilities explode. You move from a simple “ReAct loop” to something that looks like a real search problem, with explicit width and depth at every node. The analogy he reaches for is apt: an LLM playing chess by guessing the next move versus Stockfish actually searching the board. Today’s agents are guessing, sequentially, without real exploration of the possibility space.

But I think the reason this gap is the hardest to close is that it isn’t just about orchestration architecture. It goes deeper, into how these models are trained. A model that has never been rewarded for pausing mid-task to reassess won’t do it reliably at inference time, no matter how well-designed the surrounding system is. Closing this gap requires both a smarter execution engine and models that are trained to think in search trees, not just linear chains of thought.

What does this look like in practice? In our work on Maestro, we found that when a system is given explicit branching points – moments where it evaluates its current trajectory and decides whether to continue, restart, or delegate – performance on complex, long-horizon tasks improves substantially compared to a standard ReAct loop running to completion. The system stops doubling down on bad paths. That alone accounts for a significant share of the gap.

The four gaps between demo agents and production systems

The shift to the AI OS

This is why Barak and I have grown wary of the word agent. It implies a single, blackbox trajectory. What we’re actually building toward is an AI Operating System.

An OS is defined by its ability to manage resources. It doesn’t just call an API, it tracks what is running, handles dependencies, manages environments, and crucially, decides when to spawn or kill work. It manages the entire execution strategy to optimize the Pareto curve of cost and value.

Value isn’t static. Sometimes it means minimizing cost. Sometimes it means getting the right answer as fast as possible regardless of spend. A true AI OS adjusts its orchestration to match.

I think about the urgency of this shift whenever I see a post like Matt Shumer’s “Something Big Is Happening,” which recently went viral for describing the “shaking ground” the industry felt on February 5th. Matt describes AI systems that don’t just draft work but produce finished, tested apps autonomously. To me, the ultimate proof of Barak’s thesis arrived with that GPT 5.3 Codex release Matt highlighted. OpenAI admitted that the model was instrumental in creating itself, debugging its own training, and managing its own deployment.

When the AI starts building itself, you aren’t just looking at a better model. You are looking at an autonomous system that requires a principled OS to manage its own complex trajectories and reinforcement learning rollouts.

None of this is magic. The individual components – validators, routers, and parallel execution – already exist as heuristics. What has been missing is a principled system that brings them together. The future of this industry isn’t just about finding a single better model. It is about building a system that understands its own capabilities, knows what they cost, and can recognize when a task isn’t worth the effort at all.

That is what useful AI looks like in production. Anything else is just a demo that doesn’t really run.

Check out the full conversation with Barak below.