In brief

We present a new state-of-the-art result on the SWE-rebench benchmark: a 60.9% issue resolve rate for 123 issues between December and March, which surpasses the previous best published result on this same slice.

This achievement came from rethinking the agent’s context extraction phase – a counterintuitive design choice we pursued and refined. Instead of applying an “enrich then solve” approach, where context extraction happens first and solution generation follows, we reverse the order: solution generation at large scale comes first, and then the generated candidates are exploited for better context extraction.  Reaching state-of-the-art took more than the reversal itself – it required carefully optimizing how the extractor exploits those candidates to surface the most relevant code.

Our findings suggest that simple, initial parallel N attempts at solving a task can significantly aid codebase exploration, producing a highly targeted contextual map that can be utilized during the final evaluation step of the agent to generate more accurate output. This execution strategy noticeably improves the agent’s accuracy without driving up cost, and surpasses naive scaling of the generation step in its traditional order. 

This work joins AI21’s broader line of agent-optimization research, alongside earlier case studies (SWE-bench Verified, BrowseComp-Plus, Deep Research Bench), each benefitting from a different strategy of the AI21 Maestro optimization approach. We hope our contributions around coding agents’ pipeline order can serve as a blueprint for architecting increasingly accurate and cost efficient AI software engineers. 


In this post, we walk through each step from a baseline ReAct loop to this result – and show how we pushed accuracy higher while keeping total cost below that of a leading coding agent.

The architecture of agentic coding

​Agentic coding has rapidly become one of the most exciting and heavily researched frontiers in artificial intelligence. Moving beyond simple code-completion autocomplete tools, modern coding agents – like Claude Code, Codex, and Cursor – are designed to act as autonomous software engineers – reasoning through complex problems, navigating large codebases, and executing multi-step tasks to resolve real-world issues.

​The anatomy of a coding benchmark

​To evaluate these agents accurately, the industry has gravitated toward a standardized benchmark architecture. The setup typically looks like this:

  • ​Input: The agent is given a natural language issue description (such as a GitHub issue) describing a bug or a feature request.
  • ​Environment: The agent is dropped into a Docker container that houses the target code repository alongside the necessary runtime environment.
  • ​Execution: The agent interacts with this environment by executing actions – most notably, running bash commands through the terminal to search files, run tests, and edit code.
  • ​Output: The goal is for the agent to produce a solution, typically in the form of a git patch. The benchmark then evaluates this patch against a set of hidden unit tests to verify if the issue was genuinely resolved.

Establishing a baseline: A classic ReAct loop

​When building a coding agent from scratch, the most straightforward starting point is a classic ReAct (Reasoning and Acting) loop.

​To run this basic implementation on the coding benchmark environment, the agent is equipped with just a single, powerful tool: access to the Docker terminal. The agent reads the issue, reasons about what to do next, executes a terminal command (e.g., grep, cat, or pytest), observes the output, and repeats the cycle until it believes it has solved the issue.

To establish our baseline, we implemented this classic ReAct loop and evaluated it on the 12/15/2025–3/1/2026 evaluation window of SWE-rebench using GPT-5.2 as our generation model, set at medium reasoning. Under these conditions, the vanilla agent achieved a 53.8% resolve rate. We use SWE-rebench precisely because it’s built to resist the contamination and overfitting that affect older datasets – and the gains that follow come from general design choices that apply beyond this benchmark. While this is a decent baseline score, we’d need more sophisticated orchestration strategies to push our success rate further. 

Reaching a higher resolve rate: Utilizing common optimization strategies

Researchers have proposed different ways to improve agent accuracy, such as context enrichment (gathering relevant code repository information), horizontal scaling (generating and evaluating multiple candidate solutions in parallel) and the combined benefits of an “enrich then scale” approach to agent architecture. We will consider each one separately below. 

Strategy #1: Context enrichment

In a basic ReAct loop, there are no explicit steps; the agent operates with the single, broad goal of solving the issue. While this setup naturally leads the agent to perform some context-gathering actions on its own, starting completely from scratch makes navigating large codebases a massive hurdle.

Our experiments show that introducing a dedicated step for context enrichment at the very beginning improves results. It doesn’t mean the agent won’t need to gather more information on the fly, yet it begins from a more informed starting point than the basic ReAct loop implementation.  When we introduced this dedicated step – architected as a short, focused ReAct loop powered by the same model with the sole goal of collecting relevant source code – we saw our baseline resolve rate rise from 53.8% to 55.6%.

Strategy #2: Horizontal scaling

Since LLM-powered coding agents are inherently non-deterministic, running the exact same agent on the exact same issue description multiple times can yield entirely different trajectories and results. An agent might get stuck in a loop on its first attempt, but hit the right solution on its second or third run simply by chance, a possibility we demonstrated in earlier work on SWE-bench Verified.

Horizontal scaling leverages this behavior by treating compute budget as a lever for accuracy. Instead of relying on a single, linear run, we scale horizontally:

  1. Generate parallel rollouts: We run the agent multiple times in parallel to generate a diverse set of candidate patches.
  2. Select the best: We use a selection mechanism, such as an LLM judge, to evaluate the candidates and choose the best one; alternatively, if no candidate seems right, it synthesizes a better one. 

To validate this strategy in our experiments, we ran our baseline ReAct loop five times in parallel for each issue, using an LLM judge to select the optimal patch from the generated candidates. This simple horizontal scaling strategy successfully boosted our resolve rate from 53.8% to 55.4%, demonstrating the power of trading inference compute for performance. This also reveals a useful ceiling. Imagine a “perfect” judge – an oracle that always picks a correct patch whenever at least one of the five candidates is correct. Under that oracle, our resolve rate would reach 63.8% (a metric known as pass@5). The distance between what our real LLM judge achieves and this oracle is the selection gap: correct solutions we already generated but failed to pick. In other words, scaling raises the ceiling; how much we capture is a factor of the judge’s quality.

By this point, we had two proven strategies in our hands that each individually boost performance. How much higher could we push our resolve rate if we combined them?

Combining optimization strategies: The power of reversing pipeline order

Once you decide to implement both horizontal scaling and context enrichment, the next design choice is in what order to combine them.

The intuitive approach: Enrich, then scale

The standard, linear way to orchestrate these two strategies is well documented, as mentioned above. 

  1. Context enrichment: Run a dedicated step to gather relevant codebase information.
  2. Horizontal scaling: Feed that enriched context into the agent to generate multiple candidate rollouts.
  3. Selection: Choose the best patch from the generated candidates.

This approach seems logical – you give the agents the context upfront so they can generate better initial attempts. And indeed, that hypothesis bore out. By implementing this approach, we managed to bump our resolve rate up to 57.7%.

However, while this approach yields clear gains – especially above our starting point of 53.8% – our current selection mechanism leaves significant performance on the table. Selecting the right candidate often requires seeing the broader context of the relevant codebase rather than just evaluating the isolated final git patch. This is exactly the selection gap we defined earlier – and it’s wide here, with our real-life judge – not an oracle – falling well short of the 63.8% pass@5 ceiling. 

The likely cause: the judge evaluates each isolated git patch without the broader codebase context needed to tell a correct fix from a plausible-looking one. 

Ultimately, this gap highlights a powerful opportunity: to bridge the distance to our pass@5 potential, we need to ensure the judge has the full perspective required to make the optimal choice. How could we suffuse our agent with additional relevant data? 

Our approach: Scale, then enrich

The selection gap pointed to a root cause: Our judge was choosing between patches without enough context about the code the patches touched. Enriching first didn’t solve this – context built from the issue description alone is only partial, because the extractor has no signal about where the fix actually belongs. 

But that signal already existed in our pipeline. Each of the five rollouts we generate is effectively an hypothesis about where the fix lives: which files and functions to explore. When working according to “enrich, then scale,” we used those rollouts only for the final vote and discarded their localization signal. Reversing the order would make use of that signal: Let the extractor see the candidate rollouts first, and use them to guide a far more targeted search before the final selection.

Now our pipeline was as follows: Initial rollouts → Context enrichment → Output selection

That’s where we really started seeing the needle move: By inverting our approach, we outpaced the intuitive setup, bringing our score up to a 59.7% resolve rate.

It works – but why? Context extraction is notoriously difficult when a tool has to search a massive codebase blindly based on nothing but a raw issue description. By shifting the pipeline, we feed our initial candidate rollouts directly into our source code extractor (powered by GPT-5.2 with medium reasoning) – effectively injecting some context into the context enrichment phase. Here’s the system prompt we used:

Round-1 system prompt (_LLM_GUIDED_SYSTEM):
You are helping select relevant source code for a code-review task.
You will be given a problem statement and one or more candidate patches (unified diffs).
Your job is to output up to 10 bash commands that we will run in the repository root.
1. Show the full definition of functions/classes that the patch modifies (grep -n or sed to extract a range).
2. Show callers of those functions (grep -rn 'func_name' src/).
3. Show test files that cover the changed code (grep -rn 'func_name' tests/ or find tests/ -name '*.py').
4. Show imports and dependencies of the changed files (head -n 30 file.py to see imports).
5. Show any related config, schema, or fixture files referenced in the patch.
Rules:
- Do NOT produce commands that output more than ~150 lines (avoid plain `cat` on large files).
- Prefer: `grep -n -A 10 'pattern' file`, `sed -n 'N,Mp' file`, `head -n 80 file`.
- Output format: one command per line, no numbering, no explanations.
- You may wrap the block in ```bash ... ```.

Then it runs multiple rounds, with the follow-up prompt (_LLM_GUIDED_FOLLOWUP) on each subsequent round:
Based on the outputs above, suggest up to 10 more bash commands to dig deeper.
Focus on gaps revealed by round 1 — for example:
- If a function was referenced but its definition wasn't shown, fetch it now.
- If a test file was found, show the relevant test cases.
- If an import pointed to another module, show the relevant parts of that module.
- If output was truncated, use `sed -n` to fetch the missing range.
Same format: one command per line, no numbering, no explanations. Only commands that add NEW information not already shown above.

Instead of searching in the dark, the extractor can now analyze where the initial rollouts attempted to make fixes and focus its search on those high-probability areas of the repository. As a practical bonus, since we’ve already committed to horizontal scaling, we are leveraging candidate rollouts we already “paid for” in our compute budget, using them to guide the extractor. This targeted enrichment allows the final selection step – powered by GPT-5.2 now with high reasoning – to work with far more precise data, resulting in a more effective pipeline overall.

Measuring the “context coverage”

To demonstrate that the injected rollout context was actually relevant, we measured its effectiveness against the benchmark’s “gold patches.” While a definitive concept of “gold context” doesn’t exist, the benchmark does provide the ground-truth patch required to solve each issue. We used this information to track how successfully our extractor surfaces the files that actually need fixing, comparing the two extractors:

  • Issue-only extractor: Fed the problem description only, as seen in the enrich → scale approach
  • Issues + candidate extractor: Fed the problem description + five candidate patches, as seen in our scale → enrich approach. 

The results from our experiments show a clear performance difference between the two extractors across every critical file-coverage metric:

Looking at the data, the most striking advantage of a patch-aware pipeline isn’t just a marginal bump in general coverage – it’s a massive guardrail on possibly catastrophic failures.

The standout finding here is the decrease in instances where the extractor failed to visit any gold source files, dropping from 9.3% down to just 2.7%. This represents an impressive 71% reduction in total blind spots, demonstrating that giving context to the extractor effectively prevents it from going completely down the wrong rabbit hole.

While the patch-aware extractor does open slightly more files on average (10.4 vs 9.2), that minor increase in exploration is a tiny price to pay. By utilizing candidate rollouts we already generated, we virtually eliminate the scenarios where the agent is left completely blind, ensuring the final selection phase is always grounded in the right repository context.

We now had a solid method in our hands for improving agent accuracy, as evidenced by SWE-rebench performance.

The cost-quality tradeoff: Navigating the Pareto frontier

​Up to this point, our focus has been entirely on maximizing agent quality. In production engineering, however, accuracy cannot be evaluated in a vacuum – we have to pay attention to compute cost, as well. By treating our inference budget as a controllable variable, horizontal scaling and context enrichment allow us to navigate a classic Pareto curve where we can explicitly trade predictable spend for gains in quality.

​To put this into perspective, we held our pipeline variants against a real-world financial ceiling. We took the reported cost of Claude Code from the SWE-rebench leaderboard as our budget limit, aiming to see exactly how much quality we could achieve from the architecture without exceeding that cost.

​Our experiments revealed a highly efficient progression along the cost-quality curve:

  • ​Breaking the efficiency barrier: Simply generating five initial rollouts and applying our patch-aware, source-code-enriched context extractor significantly improved agent quality over the baseline – all while keeping the total execution cost comfortably below our reference ceiling. By utilizing candidates we already paid for to guide the context phase, we achieved a meaningful quality boost without a compounding explosion in token spend.
  • ​Maximizing the remaining budget: After implementing our approach, we saw we still were around $0.30 below the cost of Claude Code for comparable accuracy. How else could we catalyze that remaining budget into further accuracy gains? So we introduced a lightweight agent layer dedicated to automated verification. This specialized Test Agent dynamically creates repository tests and filters out any candidate patches that fail them, purifying the candidate list before it reaches the final selection phase.

​This multi-stage architecture – scaling horizontally, enriching context via initial patches, and filtering via lightweight test generation – pushed our final success rate to 60.9%. This not only stays within a practical budget but also edges past the reported leaderboard SOTA.

Key takeaways

  • Execution strategy is a hyperparameter. It’s not just which strategies you apply but the order you compose them in – optimizing that execution strategy is a lever in its own right. Horizontal scaling and context enrichment are each independently proven; applied together in the right order — scale, then enrich – they push the agent well past our baseline ReAct loop at no extra compute cost.
  • Reuse what you’ve already paid for. If you’re already paying the cost for horizontal scaling, maximize the asset. Those rollouts aren’t just potential answers; they are incredibly rich data points.
  • Inject context into the context step. Passing these initial rollouts downstream – whether to a source code extractor or any other step in your agent architecture – allows your steps to operate with focus rather than working blindly.

What’s next

We see our work experimenting with horizontal scaling and execution strategy as one piece of a broader push to optimize agent performance. We’re currently exploring how to extend these ideas further, such as applying the lessons about ‘rollouts-as-signal’ to other benchmarks and optimizing model routing across the pipeline’s stages to push the cost-quality frontier even further. Stay tuned for what’s next. 

Acknowledgements

With thanks to Assaf Gerner and Guy Rephaeli for their contributions.