All that glitters: When “gold-like” answers mask functional failures on coding agent benchmarks
TL;DR
While benchmarking our coding agent, we noticed our LLM judge — the component in Maestro that selects the best output from parallel agent runs — was performing suspiciously well. Contamination was the obvious suspect — but when we reproduced the same bias on a clean dataset the model couldn’t have seen, we realized something else was going on: the judge had learned to favor solutions that look like gold answers — minimal, clean, focused — over outputs that actually work. In this blog we isolate and quantify the effect of this ‘gold-like’ preference on the benchmark data and present a strategy for mitigating its impact in order to maintain the integrity of coding benchmark results.
A new data distortion culprit
When LLMs perform suspiciously well on public benchmarks, contamination is the usual culprit. So when we noticed that our reducer – an LLM Judge which selects the best patch from parallel agent runs – was demonstrating a clear preference for gold answers on SWE-bench Verified, we naturally suspected the model was recalling the dataset. Our doubts were further confirmed when OpenAI announced, two months later, that they had reached the same conclusion: Too many recent frontier models were familiar with the SWE-bench Verified dataset. SWE-bench was proclaimed contaminated.
Back in our lab, we replicated the experiment on SWE-rebench, a newer, contamination-resistant dataset; yet the preference for gold answers remained. That was a big alarm bell.
In this case, the LLM Judge wasn’t memorizing specific solutions – but it was learning the traits of “gold-like” patches, such as minimality and clarity. While these flavoring are typically considered positive, here, they introduced a misalignment with our success criteria: The model prioritized surface-level traits over functional correctness, often favoring a “clean” wrong solution over a “messy” correct one. It turns out that LLM Judges were delivering on form over function.
As we investigated, we found these seemingly ‘positive’ stylistic preferences can be just as distorting as explicit contamination. To tackle this, we redesigned our reducer prompts with concrete prioritization rules. Over a few iterations, we managed to create detailed enough prompt guidelines to align the model’s judgment with our actual success criteria. We share that journey below with other builders interested in investigating sources of contamination and distortion in their benchmark results and building more reliable measures of coding agent performance.
What are reducers? A quick intro
Since LLM-based agents are non-deterministic, running the same query multiple times produces different outputs; — with enough runs, at least one is likely correct. The challenge is identifying which one. This is the gap between pass@k (how often at least one of k runs gets it right) and the score you actually realize — which depends entirely on your ability to pick the correct output.
In Maestro, AI21’s orchestration meta model framework for agentic tasks, the system is spinning up multiple agents to work on the problem in parallel, and selecting the best output. That last step is handled by a component we call a reducer — essentially a selector that retrieves all output candidates and hopefully, if we configure it properly, picks the best one.
Reducers can make this selection in a few different ways. For example, some generic guidelines could be:
When creating a reducer for a specific use case, it may be beneficial to get specific; for example, we could instruct the reducer to pay more attention to the available context, or different parts of the output, and rank the candidates according to different criteria.
Same goes for the SWE-bench reducer. We needed to get specific in our guidelines.
Building a reducer for SWE-bench agents
The SWE-bench challenge
SWE-bench is a popular benchmark for evaluating coding agents. The input is a Github issue and a Docker image of a repository, and the expected output is the Git patch that contains the changes needed to resolve the issue. The agent’s job is to explore the repository, understand the issue and generate a patch. As long as the tests pass, the patch is considered successful (regardless of its code quality).
A common approach for SWE-bench agents is to use a ReAct loop with a single tool: a docker terminal that allows the agent to interact with the repository through bash commands. We started by implementing this agent within Maestro to collect a dataset of output patches. We generated the patches with two different models (GPT-5 mini and MiniMax-M2.1), and ran each example multiple times to get pass@k scores:
These results show the potential of a good SWE-bench reducer: the scores increase as we increase k, indicating that running multiple times on each instance returns a mixture of both ‘good’ and ‘bad’ patches. If we had a good reducer, we could achieve a significant improvement by just running the same agent multiple times and relying on the reducer to select the best candidate.
Adapting reducer variants to SWE-bench
Given what we understand about the SWE-bench benchmark, how do we implement a reducer that best supports our agent’s ability to output correct patches? Here’s how we adapted the three types of reducers outlined above to the SWE-bench setting:
- Random reducer: Since the agent often finishes without making any changes, it’s not uncommon to get an empty patch as an output. In SWE-bench, we don’t get extra credit for “abstain”, so any reducer should start by filtering out candidates with an empty patch. This step automatically improves the score of the “random” reducer over that of pass@1.
- Majority-vote reducer: Since the output here is a Git patch, it’s not trivial to define “identical” outputs. “Exact match” would be too strict, so we defined our matching logic to ignore formatting, comments and additional tests code. On average, the resulting score is usually higher than the random reducer, yet may dip below the random reducer on the more difficult examples.
- LLM Judge: This kind of reducer can take on different levels of complexity and cost, depending on what information we include in the context (e.g. only the patch, the full agent trajectory, the repository code, etc.). Since the patch alone is often not enough to make a decision, the patch reducer often exposes some of the LLM’s inherent biases, which makes it a particularly interesting reducer to study.
For that reason, going forward, we’ll focus just on the LLM-based patch reducer.
Reducers in action: Testing a SWE-bench patch reducer
We started by implementing a basic patch reducer – a single LLM call to Claude Opus 4.5 with the following prompt:
You are given an issue from a GitHub repository and a set of proposed solutions (in the form of git patch candidates).
Please choose the patch with the highest chance of resolving the issue.
Your focus should be solely on the correctness of the patches, not on their code quality or style.We ran this reducer on the dataset of reducer-relevant examples (examples where not all candidates are either good or bad, which left us with 400 examples: 234 from GPT-5 mini and 166 from MiniMax-M2.1).
From these numbers, it looks like this simple, three-line prompt is already giving us a lift over our earlier baselines. Which is great in theory… but also raises a concern.
Contamination revealed
We suspected that at least a part of this improvement might be attributed to a known SWE-bench contamination issue. The subset of 400 examples we’re evaluating on are derived from SWEBench-Verified, which was released in August 2024. The model we’re using for our reducer, Claude Opus 4.5, was released in November 2025 with a knowledge cutoff of August 2025. This timeline means our experiments might be contaminated: the model has knowledge about solutions that should be hidden. This was the point made in OpenAI’s blog post, mentioned earlier, which showed some evidence that recent LLMs were trained on the benchmark hidden tests and gold patches.
Given our concerns about contamination, we sought a way to validate the integrity of our results.
Quantifying contamination: An experiment
Initial hypothesis:
A significant part of the improvement of the patch reducer can be attributed to its knowledge of the gold patch due to SWE-bench contamination.
Experiment:
- Take the 400 reducer-relevant examples from the previous section
- Turn each example into two examples of positive candidate + negative candidate pairs:
- For the first example: Choose the positive and the negative candidates randomly.
- For the second example: Use the same negative candidate but replace the positive candidate with the gold patch.
- Collect the examples into two new separate datasets: “positive vs. negative” and “gold vs. negative”.
- Run the patch reducer on both datasets and check the difference in accuracy.
Results:
The reducer accuracy on “gold vs. negative” was much higher than its accuracy on “positive vs. negative”.
So this proves that the model remembers the gold patch, right?
Well, not necessarily. To truly validate this hypothesis, we need to see that this phenomenon does not happen on a more recent dataset, which doesn’t have the risk of contamination.
Validating on SWE-rebench
SWE-rebench is a relatively new dataset that is similar to SWE-bench but includes more recent Github issues, some of which – importantly – were created after the cutoff date of Claude Opus 4.5.
We took 258 issues from August 2025 to February 2026, and ran our agent on them (again using GPT-5 mini and MiniMax-M21) to create examples for the reducer.
This resulted in 169 reducer-relevant examples, which were converted into the “positive vs. negative” and “gold vs. negative” datasets as described before.
Now we ran the same experiment, and surprisingly… got the same gap.
It glitters, but it’s not gold: Rethinking contamination bias
Do the SWE-rebench experiment results mean the model remembers the gold patch?
Probably not. But the model does probably have a preference for “gold-like” solutions, ones that glitter but are not actually gold. In other words, it may not know the contents of the gold patch, but it knows the general characteristics of a gold patch.
For example, one of the main characteristics is minimality. Gold patches are usually ones that do not make any irrelevant changes in the code. However, as we discussed earlier, SWE-bench (and SWE-rebench) success criteria is all about passing the hidden tests; code minimality is not required. A coding agent might leave a complete mess in the code and still get the points for resolving the issue.
And indeed, when comparing the number of lines changed by the gold patches and by the positive patch, we see that the positive patch changes about 50% more lines. Below is an example of two valid patches for instance id psf__requests-1724 in SWE-bench: one is the gold patch, while the other is a patch generated by the agent.
Gold patch:
diff --git a/requests/sessions.py b/requests/sessions.py
--- a/requests/sessions.py
+++ b/requests/sessions.py
@@ -12,7 +12,7 @@
from collections import Mapping
from datetime import datetime
-from .compat import cookielib, OrderedDict, urljoin, urlparse, urlunparse
+from .compat import cookielib, OrderedDict, urljoin, urlparse, urlunparse, builtin_str
from .cookies import cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar
from .models import Request, PreparedRequest
from .hooks import default_hooks, dispatch_hook
@@ -309,6 +309,9 @@ def request(self, method, url,
:param cert: (optional) if String, path to ssl client cert file (.pem).
If Tuple, ('cert', 'key') pair.
"""
+
+ method = builtin_str(method)
+
# Create the Request.
req = Request(
method = method.upper(),
Agent-generated positive patch (by MiniMax-M2.1):
diff --git a/requests/models.py b/requests/models.py
index f82f56a..58b0273 100644
--- a/requests/models.py
+++ b/requests/models.py
@@ -307,7 +307,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
"""Prepares the given HTTP method."""
self.method = method
if self.method is not None:
- self.method = self.method.upper()
+ self.method = builtin_str(self.method.upper())
def prepare_url(self, url, params):
"""Prepares the given HTTP URL."""
diff --git a/requests/sessions.py b/requests/sessions.py
index cc72f65..473d30e 100644
--- a/requests/sessions.py
+++ b/requests/sessions.py
@@ -12,7 +12,7 @@ import os
from collections import Mapping
from datetime import datetime
-from .compat import cookielib, OrderedDict, urljoin, urlparse, urlunparse
+from .compat import cookielib, OrderedDict, urljoin, urlparse, urlunparse, builtin_str
from .cookies import cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar
from .models import Request, PreparedRequest
from .hooks import default_hooks, dispatch_hook
@@ -118,7 +118,7 @@ class SessionRedirectMixin(object):
method not in ('GET', 'HEAD')):
method = 'GET'
- prepared_request.method = method
+ prepared_request.method = builtin_str(method)
# https://github.com/kennethreitz/requests/issues/1084
if resp.status_code not in (codes.temporary, codes.resume):
@@ -254,7 +254,7 @@ class Session(SessionRedirectMixin):
p = PreparedRequest()
p.prepare(
- method=request.method.upper(),
+ method=builtin_str(request.method.upper()),
url=request.url,
files=request.files,
data=request.data,
@@ -311,7 +311,7 @@ class Session(SessionRedirectMixin):
"""
# Create the Request.
req = Request(
- method = method.upper(),
+ method = builtin_str(method.upper()),
url = url,
headers = headers,
files = files,
As we can see, the agent-generated patch made more changes than the gold patch (5 lines vs. 2). In our experiment, this positive candidate was paired with the following negative candidate:
diff --git a/requests/models.py b/requests/models.py
index f82f56a..2d49061 100644
--- a/requests/models.py
+++ b/requests/models.py
@@ -307,7 +307,7 @@ class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
"""Prepares the given HTTP method."""
self.method = method
if self.method is not None:
- self.method = self.method.upper()
+ self.method = to_native_string(self.method.upper())
def prepare_url(self, url, params):
"""Prepares the given HTTP URL."""This candidate offered a much cleaner and focused change, which is also more similar to the gold patch.
When we inspected our reducer’s reasoning for this example, it’s quite clear that its judgement was not focused purely on correctness, which is why it ended up selecting the wrong candidate. Let’s take a look:
Patch 1 is the minimal, correct fix that addresses the root cause at exactly the right location.
Patch 0 contains unnecessary redundant conversions (multiple places converting to `builtin_str` when the `prepare_method()` will do the conversion anyway).But does gold always glitter?
Running on SWE-rebench, however, the difference in number of changes between the generated positives and the gold patches was not that significant for all subsets; for the patches generated by MiniMax-M21, it was 7%. Diving deeper into our analysis, we found that, while in most cases the gold patch is shorter, there were also instances in which the gold patch is actually longer. But isn’t the gold patch supposed to be the most minimalistic solution to the issue? How do we explain this discrepancy?
Well, one explanation is related to documentation. When running SWE-rebench, we discovered that more than 20% of gold patches contained documentation files updates (.md, .rst, changelog, examples, notebooks), which don’t exist at all in our generated patches (and very rarely in the gold patches of SWE-bench).
Yet even after filtering out the cases of documentation file changes, we still observed cases where the positive patch offered a more minimal change, while the gold patch seemed to address additional requirements that were not specified in the original issue and not evaluated by the tests. It might technically be gold, but it has lost its shine.
An example of this behavior is instance joke2k__faker-2309 from SWE-rebench. The gold patch adds 101 lines (or 73 excluding comments) to fix a CLI argument parsing bug, while the agent-generated positive patch fixes the same bug in 5 lines and passes all the same tests. Let’s take a look:
Here’s the gold patch (truncated):
diff --git a/faker/cli.py b/faker/cli.py
index 52badf17..793748b0 100644
--- a/faker/cli.py
+++ b/faker/cli.py
@@ -3,6 +3,7 @@ import itertools
import logging
import os
import random
+import re
import sys
import textwrap
@@ -20,6 +21,75 @@ __author__ = "joke2k"
T = TypeVar("T")
+def _parse_faker_args(args, available_locales):
+ """
+ Parse faker CLI arguments, handling the -i case specially.
+ ...
+ """
+ parser = argparse.ArgumentParser(add_help=False)
+ parser.add_argument("-i", "--include", nargs="*", default=[])
+ parser.add_argument("-l", "--lang", choices=available_locales, default=None)
+ parser.add_argument("-r", "--repeat", type=int, default=1)
+ parser.add_argument("-s", "--sep", default="\n")
+ parser.add_argument("--seed", type=int, default=None)
+ parser.add_argument("-v", "--verbose", action="store_true")
+ parser.add_argument("fake", nargs="?", default=None)
+ parser.add_argument("fake_args", nargs="*", default=[])
+
+ parsed, _ = parser.parse_known_args(args)
+
+ if parsed.include and parsed.fake is None:
+ include_values = list(parsed.include)
+ last_non_option_idx = len(include_values) - 1
+ while last_non_option_idx >= 0 and include_values[last_non_option_idx].startswith("-"):
+ last_non_option_idx -= 1
+ if last_non_option_idx >= 0:
+ last_include = include_values[last_non_option_idx]
+ if "." not in last_include and re.match(r"^[a-zA-Z_][a-zA-Z0-9_]*$", last_include):
+ new_include = include_values[:last_non_option_idx]
+ new_fake = last_include
+ try:
+ fake_idx = args.index(last_include)
+ new_fake_args = args[fake_idx + 1:]
+ except ValueError:
+ new_fake_args = []
+ return (list(new_include), new_fake, list(new_fake_args),
+ parsed.lang, parsed.repeat, parsed.sep, parsed.seed, parsed.verbose)
+ return (list(parsed.include), parsed.fake, list(parsed.fake_args),
+ parsed.lang, parsed.repeat, parsed.sep, parsed.seed, parsed.verbose)
+
+
@@ -253,6 +323,37 @@ examples:
arguments = parser.parse_args(self.argv[1:])
+ (providers, fake, fake_args, lang, repeat, sep, seed, verbose,
+ ) = _parse_faker_args(self.argv[1:], AVAILABLE_LOCALES)
+ if providers:
+ arguments.include = providers
+ if fake:
+ arguments.fake = fake
+ if fake_args:
+ arguments.fake_args = fake_args
+ ...And here’s the agent-generated positive patch (by MiniMax-M21):
diff --git a/faker/cli.py b/faker/cli.py
index 52badf17..d31c9a92 100644
--- a/faker/cli.py
+++ b/faker/cli.py
@@ -252,6 +252,19 @@ examples:
arguments = parser.parse_args(self.argv[1:])
+ if arguments.fake is None and arguments.include:
+ last_arg = arguments.include[-1]
+ if "." not in last_arg:
+ arguments.fake = last_arg
+ arguments.include = arguments.include[:-1] Both patches pass all tests. Every example from the issue (faker -i faker_credit_score credit_score, faker -i faker_credit_score ssn) is correctly handled by the 5-line patch. The gold patch’s additional machinery (a second full argument parser, a backward scan loop, a regex identity check, and fake_args extraction) addresses cases not mentioned in the issue and not covered by any test.
To measure the impact of such “patch redundancy” on our reducer, we calculated the rate of reducer errors that can be attributed to this factor. We defined a “redundancy error” as any example where a correct patch was rejected because the reducer’s reasoning labeled it as redundant, unnecessary, unrelated, or irrelevant. The resulting data reveals that these errors are most prevalent in positive-negative examples, but they also constitute a significant portion of failures in gold-negative datasets, including the SWE-bench subset.
So far, in our experiments, we’ve seen evidence that not all that glitters is gold, and sometimes even gold doesn’t glitter. If that’s the case, how can we prevent our reducer from being distracted by shiny aesthetics?
Going back to the (prompt) drawing board
Was our simple three-line prompt enough for our reducer to produce a significant improvement on baseline scores? Yes. Was it for the “right” reason? Probably not.
It appears that if we want to have a reducer that overcomes the model’s inherent biases and prioritizes patch correctness, our 3-line statement (“Your focus should be solely on the correctness..”) is not enough. We must provide a much more detailed set of rules which explicitly address the different qualities of a patch and offer a clear prioritization roadmap for them.
And indeed, our next step was to update the prompt with such a set of rules, and rerun our experiment. The results were unequivocal: the positive-negative vs. gold-negative gap disappeared completely (for SWE-bench) or mostly (for SWE-rebench).
You can take a look at the detailed prompt we used to achieve these results:
You are given an issue from a GitHub repository and a set of proposed solutions (in the form of git patch candidates).
Please choose the patch with the highest chance of resolving the issue.
EVALUATION PRIORITIES (in order):
1. CORRECTNESS & COMPLETENESS (most important)
- The fix must COMPLETELY solve the issue — partial fixes are wrong even if they look clean.
- If the issue implies multiple affected code paths (e.g., serialize AND deserialize, multiple backends), ALL must be fixed.
- A multi-file fix that's complete is better than a single-file fix that's incomplete.
- A candidate may resolve the issue even if it makes some redundant changes. DON'T disqualify a candidate just because it makes redundant changes.
- A messy-looking fix is better than a clean-looking fix that doesn't solve the issue.
- Mentally trace the failing case with the fix applied — does it actually work now?
2. REGRESSION SAFETY
- Ideally every change should directly serve the fix. But don't disqualify a candidate for redundant changes if the fix is still correct.
- Watch for: changed method signatures other code depends on, removed fallbacks, changed return types.
3. MINIMALITY (tiebreaker only)
- Only consider minimality after correctness and completeness are established.
- Adding tests, docs, or helper functions should not affect your selection.
COMPARISON TIPS:
- If multiple candidates make the same core fix — differentiate on errors or edge-case coverage, not on style or readability.
- Ignore differences in test files, comments, formatting, or variable naming.In conclusion: Key experiment takeaways
- Suspiciously high performance on popular benchmarks isn’t always a sign of data leakage; it can be an inductive bias toward the stylistic “fingerprint” of a correct solution.
- When your task is correlated with this bias, a simple prompt may be good enough. But when the right solution doesn’t ‘look’ the way the model expects, you need detailed rules to force a realignment with your success criteria.
Specifically regarding the SWE agent, this experiment emphasizes that measuring functional correctness alone may not be informative enough. Going forward, we may want to add new metrics to measure non-functional qualities like minimality and clarity.
