How we cut GitLab Duo Code Review’s cost by 98%

In Brief

We cut GitLab Duo Code Review’s cost by 98% on Martian’s Code Review Bench without losing review quality, and all of it came from optimizing the harness: which model each stage runs on and how much context each stage reads. Neither change gets there alone. The open-weight model stalled on a handful of oversized reviews, and compacting the context is what made them fit.

An agent’s cost is eventually set by its harness: which stages it runs, what each stage reads, and which model does the work. We took GitLab Duo Code Review, whose harness is source available, measured where its money sits, and, with our own agent helping run the experiments, changed one thing at a time until the bill stopped dropping.

But the real story is not the final changes in the harness; it’s the way we got there. Switching to a smaller model in every stage hurt the quality. Moving to an open-weight model on every stage cut 85% of the cost, but we didn’t stop there. While compaction looks marginal at first glance, the 98% only shows up with both, because the open-weight model’s deployment window decides which requests it can take at all. The model sets the price. The harness decides whether you get to use it.

We ran this on public pull requests, with no GitLab production traffic or customer data. Quality is F1 on one public benchmark, scored by two judges. Cost is token spend at published list prices, including for the model we serve ourselves, so it measures what the harness consumes. What our own infrastructure costs is a separate question. Configurations were chosen and tuned on the same 50 pull requests they’re scored on, so read these as measured results on this benchmark; how they hold on a new codebase is untested. The full methodology is at the end.

A pipeline with four stages

Duo Code Review, the Code Review Flow of GitLab Duo Agent Platform, reviews merge requests and posts comments. At the time of testing, it ran Claude Sonnet 5 by default, and GitLab published the flow, built on LangGraph, in its source-available ai-assist repository. That, plus a public benchmark, is why we chose it.

It runs four model stages, each with its own prompt and at most two tools:

• Exploration lists the repository tree around the changed files.
• File fetching reads related files: tests, callers, custom instructions.
• Analysis turns what was fetched into notes.
• Review reads the diff and most changed files in full, then posts comments.

Every generic optimization we had built targets coding agents like Claude Code, which resend one growing conversation full of tool output and tool definitions. Duo Code Review’s stages are independent: each starts with a fresh prompt, carries at most two tools, and has no history to trim. Our defaults had nothing to act on and changed zero requests. 

It turns out there’s no one-size-fits-all approach to harness optimization; each harness requires its own careful analysis.

Where the money goes

The review is 65% of the bill, and its output alone is a third of the total: free to think before it posts, Sonnet 5 writes about 18,600 tokens per review, nearly all reasoning. Exploration and fetching, which only decide which files to read, are 27%. Analysis is 9%.

The money mainly goes in three places: the context every stage reads, the mechanical stages, and the review’s reasoning. No single change reaches all three but each step below reaches one of them.

Finding 1: less context, more reasoning

Can we cut what the review reads without hurting it? Yes, and it saves less than arithmetic suggests.

The review stage builds its context including changed files in full, plus a diff where every line sits in a tag repeating its type and both line numbers. A unified diff marks the same line with one character.

We compacted both in front of the flow, with rules rather than a model: nothing is summarized. File windowing keeps each changed hunk plus 25 lines either side of both the old and new line ranges, and elides the rest. Diff compaction rewrites the per-line diff as a standard unified diff with the same hunks and line numbers.

The median review prompt fell from 59K to 32K tokens. Spending on review context fell further, by 62%, because the largest prompts shrank hardest.

Yet the review’s cost fell by only a fifth. With compaction on, Sonnet 5 also wrote more: about 23,800 output tokens per review instead of 18,600, nearly all reasoning. We haven’t isolated why, but it caps what compaction can save on its own. On its own, compaction took cost from $0.589 to $0.475, a 19.4% saving, at 47.2% F1 and 1.16x the latency.

Finding 2: the open-weight model held on every stage

How far does changing models get you? That depends on which stage you change.

We started by moving Claude Haiku 4.5 onto one stage at a time. On exploration and fetching, it picked the same files Sonnet 5 did. Early tests on the other two were less kind: quality dipped with Haiku on analysis and dropped sharply with it on the review. Same model, very different results depending on where it sat. Kept on the two mechanical stages, Haiku cut the bill 15.4% at unchanged F1, and 34.6% with compaction.

That was as far as Haiku could go. It couldn’t hold quality on the review, where most of the money is, and at half Sonnet 5’s price it wouldn’t have moved the bill enough if it had. Reaching the review’s reasoning meant leaving the Claude family.

We then turned to an open-weight model: DeepSeek V4 Flash, the first one we tried, which we serve with vLLM on H100s. At Together AI’s list price it costs 14x less than Sonnet 5 per input token and 36x less per output token. With all four stages on it and compaction off, cost fell to $0.086, an 85.3% saving, at 48.4% F1 and 0.37x the latency. On this benchmark it held on every stage, review included.

Two harness settings made that hold. Reasoning effort was set by measurement (see “What didn’t help”). And about one review call in seven on DeepSeek ended without posting; GitLab’s own correction step retried each one, so every review was posted, and the retries are in the costs.

Why the saving stopped at 85%

Our DeepSeek deployment accepts prompts up to 256K tokens; anything longer stays on Sonnet 5 so the review completes. The model supports 1M, but serving that window takes a more complex deployment, and its quality at very long contexts is still debated. Without compaction, some review prompts reach 480K, and 3 of the 50 PRs cross the limit, producing 11 fallback calls.

Six percent of pull requests carried 77% of what was left. And the prompts that overflow are the largest reviews, large for exactly the reasons in Finding 1.

So compaction goes back in, this time to make prompts fit. The largest review prompt drops from 480K to 109K. Every review fits. Nothing falls back. Cost reaches $0.0135 per PR, 97.7% below baseline, at 47.2% F1 and 0.35x the latency, with the second judge within half a point.

At that point compaction’s own token saving is about $0.006 per PR. What it buys is fit. In front of Claude, compaction is a cost optimization. In front of a 256K deployment, it keeps every review on the open-weight model.

A 1M deployment would also have removed the fallback. It would not have shrunk what every review reads, and it would have leaned on the model at the lengths where its quality is least settled.

What didn’t help

Results

Any stage not named runs on Sonnet 5. No setup moved F1 by more than 2 points on either judge. The final setup’s reviews are more precise than the baseline’s (46.8% vs 41.9%) but catch slightly fewer issues (recall 47.6% vs 52.0%), and at about 9,000 tokens against 18,600, run at about a third of the latency.

Summary

• The bill sat in three places, each needing a different change: shared context, mechanical stages, and the review’s reasoning.
• Haiku 4.5 held quality on the two mechanical stages and lost it on analysis and the review.
• With compaction, the median review prompt shrank 46% while reasoning output grew 28%.
• Three of 50 PRs carried 77% of the remaining cost until compaction made every review fit the deployment’s window.

Compaction on its own is worth 19.4%. Added after the model change, the same compaction is worth 84% of what was left, because by then its job was making prompts fit.

That interdependence is what makes harness optimization hard, and why it pays off. The settings in this study kept changing each other’s value. Compaction was worth a fifth of the bill on Sonnet 5 and decided the outcome on a 256K deployment. Haiku was safe on two stages and hurt quality on the other two. Getting to 98% meant searching configurations as a whole, and that search is where the saving came from.

The specific settings belong to this agent, and they’re already dated: GitLab has moved Duo Code Review to Sonnet 5.5 since we ran this. The method travels better. Measure cost per stage before changing anything. Change one variable at a time, and re-test earlier changes after each new one lands. Most teams won’t have a public benchmark to work against; we’d expect the same loop to work on a slice of live traffic, and once a configuration holds on most of it, it becomes a line in the harness.

Appendix: How we measured

Everything in the table below was fixed before the first run. Our own agent helped run the search inside those rules, proposing, executing, and scoring configurations.

To check our scoring, we ran it on GitLab’s own recorded reviews of these PRs, which Martian publishes: 52.3% F1 against Martian’s 54.2%. Those came from a different configuration, so they validate our scoring and tell us nothing about our baseline. These are public PRs, so their content may have been in training data. What we can rule out is the review models reproducing the answers from the question: given only each PR’s title, URL, and commit link, both score about 4% F1. Every file the agent read came from pinned offline copies.

Thanks to GitLab’s team for publishing the Duo Code Review flow and its benchmark results, which made this study possible.