CodeHealth Predicts the Token Cost of Agentic Code Review: A Pre-Registered Study of 465 Agent-Authored Pull Requests

Lo Gullstrand Heander; Markus Borg; Emma Söderberg; Christofer Rydenfält

cs.SEOct 15, 2026

CodeHealth Predicts the Token Cost of Agentic Code Review: A Pre-Registered Study of 465 Agent-Authored Pull Requests

Authors: Lo Gullstrand Heander, Markus Borg, Emma Söderberg, Christofer Rydenfält

Organizations: Lund University, Lund, Sweden · CodeScene AB, Malmö, Sweden

Abstract

Context: Organizations increasingly delegate code review to multi-agent large language model (LLM) pipelines billed per token. Low code quality raises human maintenance effort, but its effect on agent review cost has not been measured. Objective: We test whether baseline CodeHealth predicts the token cost of agentic review and fix beyond pull request (PR) footprint. Method: In a pre-registered observational study, we sampled 465 agent-authored PRs from 246 repositories in AIDev, jointly stratified by CodeHealth and footprint under repository caps. A frozen five-agent review-and-fix pipeline processed every PR with three different LLMs. Per LLM, we fitted a mixed model of log tokens on CodeHealth and log PR footprint, with a repository random intercept. Results: For all three LLMs the CodeHealth coefficient was negative, with 95% CI excluding zero; each CodeHealth point was associated with 3.5% to 6.1% lower token cost. CodeHealth was unrelated to PR footprint, so PR size does not explain the association. Two of three registered smells predicted fix cost; batch fixing left median CodeHealth unchanged. Exploratory analyses located the association in the review stage, where agents iterated more on unhealthy code without writing more comments. Adjusting for the size of the scored files absorbed the association, which we interpret as mediation by file size. Conclusions: A quality metric calibrated against human maintainability also predicts the inference cost of multi-agent review and fix. The effect is modest per point and not carried by PR size, so the refactoring case extends to agent-reviewed code when it changes structure and file size, not only the score. Data Availability: https://doi.org/10.5281/zenodo.22127920

Figures & tables

Panel of Figure 1 — Figure 1: The registered answers to RQ 1 and RQ 2 , fitted separately per LLM. Takeaway: Token cost falls with CH for all three LLMs, and CH does not predict footprint, so larger PR footprints do not account for the extra cost of unhealthy code.

Figure 2: Overview of the study procedure in four phases, with its milestones on a timeline. Takeaway: Hypotheses and sampling were locked after pilot 3, and the pre-registration was finalized after the review agents had started but before the fix agent ran, so no complete row existed at filing (Section 3.12 ).

Panel of Figure 3 — Figure 3: Associational path diagram of the registered model, $\log(\mathit{tokens\_total})\sim\mathit{health}+\log(\mathit{footprint})+\mathit{language}+(1|\mathit{repo})$ . Takeaway: Change footprint is treated as a mediator candidate sitting on the path from CodeHealth to cost, not as a nuisance covariate to be removed, so RQ 1 concerns the direct path $c^{\prime}$ that survives conditioning on footprint and RQ 2 concerns the decomposition of $a$ , $b$ , and $c^{\prime}$ ; the arrows denote regression paths in an observational design and do not imply causality.

Collection	PRs	Date	Sampling and LLMs	Outcome
Pilot 1	21	2026-06-26	CodeHealth tier only, no repository cap; earlier prompts and harness; qwen3-6 without prefix caching	Discarded: CodeHealth tier aliased with PR size
Pilot 2	37	2026-07-07	CodeHealth tier only, no repository cap; qwen3-6 without prefix caching	Discarded: unhealthy tier 92% one repository
Pilot 3	45	2026-07-13 to 2026-08-19	$3\times 3$ grid, 5 per cell, at most 1 PR per repository per cell and 3 overall; all three LLMs	Hypothesis-generating basis for the registration
Main	465	2026-08-19 to 2026-08-24	$3\times 3$ grid, streamed to exhaustion, at most 2 per cell and 9 overall; all three LLMs	Confirmatory

Table 1: The three pilot exercises and the main collection. Takeaway: Pilots 1 and 2 failed for sampling reasons rather than pipeline reasons, and each failure influenced the final design; pilot 3 is the main collection at one tenth of the scale, run with the same three LLMs, prompts, and workflow, with small harness updates in between.

CodeHealth tier	Small PRs	Medium PRs	Large PRs	Tier total
Unhealthy	56	33	40	129
Problematic	56	56	56	168
Healthy	56	56	56	168
Tercile total	168	145	152	465

Table 2: Achieved cell fill on the

3\times 3

CodeHealth by footprint grid, against a target of 56 PRs per cell. Takeaway: Seven of nine cells filled completely. The two short cells are in the unhealthy tier at medium and large footprint, reflecting a real scarcity of low-CH PRs in the AIDev pool.

Figure 4: One PR from each CodeHealth tier traced through the measurement instrument, all three from the medium footprint tercile and all three measured with sonnet-5 . A pass is one invocation of an agent over its whole input; the iteration counts in each cell are the agent turns taken inside that single pass, so a batch fix pass with no retry can still run many iterations. Each is the PR whose total token cost lies closest to its own tier’s median among PRs with 95 to 125 changed lines, so none is an extreme case. Takeaway: At a nearly identical PR footprint, total cost rises as baseline CH falls, and the rise is not explained by the number of review comments, which are unrelated to CH across the whole sample (

\rho=-.03

p=.56

). Agent iterations, in contrast, are correlated with CH (

\rho=-.16

p<.001

). Three PRs illustrate the measurement path and are not evidence for it; the estimates are in Section 4 .

Agent	Review scope	ISO/IEC 25010
SecurityReviewer	Injection, authentication, cryptography, data exposure, path traversal, XSS, and CSRF	Security
PerformanceReviewer	Algorithmic complexity, memory, I/O, N+1 queries, and blocking asynchronous code	Performance Efficiency
DesignReviewer	Coupling and cohesion, layer violations, naming consistency, API design, and error propagation	Maintainability, Compatibility
LogicReviewer	State transitions, calculations, domain invariants, boundary errors, and control flow	Functional Suitability

Table 3: The four review agents and their primary ISO/IEC 25010 quality characteristics ( ISO/IEC 2023 ) . Takeaway: The agent set partitions review concerns along an externally defined taxonomy, which makes the choice of roles auditable; the mapping is approximate only for DesignReviewer, which spans two characteristics.

Identifier	Stack	Version	Cached input
deepseek/deepseek-v4-flash	Hosted API	Provider, v0731	$93.2\%$
nvidia/Qwen3.6-35B-A3B-NVFP4	Local vLLM	Commit 491c2f1 ,	$81.9\%$
		vllm==0.27.1
anthropic/claude-sonnet-5	Hosted API	Provider, v5	$89.7\%$

Table 4: The three LLMs, their serving stacks, versions, and cached input shares. The cached input share is computed as the share of all input tokens served from the prefix cache across the 465-PR collection, pooled over review and fix agents. Takeaway: The three instruments differ simultaneously in provider, quantization, serving stack, and prefix-cache regime, which is why the analysis fits them separately and never compares them inferentially.

Smell	PRs present	Mean adjusted $\|\rho\|$	Rank
Complex Method	35	.339	1
Code Duplication	12	.333	2
Complex Conditional	20	.311	3

Table 5: The three code smells registered as confirmatory hypotheses for RQ 3 , selected from pilot data by the pre-specified rule. Takeaway: The rule retained the three smells present in enough pilot PRs, with a sign agreed by all three LLMs and the largest mean adjusted correlations; each was then registered with a mechanism of its own.

Category	Termination reasons	Treatment
Complete	completed	Kept, primary data
Guard-stopped, review	ceiling , explore_stall , write_drought , spiral , truncated	Kept, flagged for sensitivity check
Guard-stopped, fix	same reasons	Excluded: no completed fix record
Infrastructure	rate_limited , server_error	Retried, never skipped
Error	parse_error , unexpected_error	Retried, never skipped

Table 6: Agent-session termination categories and their treatment. Takeaway: A guard-stopped review session is kept because it is instrument behavior and part of the cost being measured; a guard-stopped fix session leaves no completed fix record or post-fix score, so that PR is excluded for that LLM. Infrastructure failures and unexpected errors are retried until success.

LLM	$\beta_{\mathit{health}}$	SE	95% CI	ICC	MDE a	Required $n$
deepseek-v4-flash	$-0.035$	0.048	$[-0.129,0.059]$	.497	0.048	1,516
qwen3-6	$-0.068$	0.064	$[-0.193,0.058]$	.714	0.067	0 861
sonnet-5	$-0.071$	0.072	$[-0.213,0.070]$	.358	0.070	0 711

Table 7: Pilot 3 per-LLM CodeHealth-to-cost slope and the resulting power characteristics at the achieved main-collection design (

n=465

, 246 repositories). Takeaway: All three pilot slopes are negative and similar in magnitude, but every LLM’s own required sample size exceeds the achieved 465, which is why no single LLM is designated primary and why the confirmatory claim is a direction-consistency criterion across all three.

LLM	$n$	Repos	$\beta_{\mathit{health}}$	95% CI	$p$	ICC
deepseek-v4-flash	453	243	$-0.062$	$[-0.090,-0.035]$	$<.001$	.221
qwen3-6	461	243	$-0.049$	$[-0.085,-0.014]$	$.006$	.032
sonnet-5	453	242	$-0.036$	$[-0.070,-0.001]$	$.041$	.131

Table 8: The registered primary model,

\log(\mathit{tokens\_total})\sim\mathit{health}+\log(\mathit{footprint})+\mathit{language}+(1|\mathit{repo})

, fitted separately per LLM by restricted maximum likelihood on the achieved main collection. Takeaway: All three coefficients are negative and all three 95% confidence intervals exclude zero, so RQ 1 ’s conjunctive criterion is satisfied on both its direction clause and its interval clause.

Figure 5: Point estimates and 95% confidence intervals for

\beta_{\mathit{health}}

, one fit per LLM, from Table 8 . Takeaway: All three intervals sit on the negative side of the reference line at zero, and sonnet-5 ’s interval, whose margin is the narrowest of the three, still clears it, which is why the registered criterion, two of three intervals excluding zero, is met with a margin.

Path	LLM	$\beta$	95% CI	$p$
(a) CodeHealth $\rightarrow\log(\mathit{footprint})$	deepseek-v4-flash	$+0.014$	$[-0.049,+0.078]$	$.657$
	qwen3-6	$+0.014$	$[-0.050,+0.077]$	$.672$
	sonnet-5	$+0.007$	$[-0.056,+0.069]$	$.838$
(b) $\log(\mathit{footprint})\rightarrow$ cost	deepseek-v4-flash	$+0.445$	$[+0.405,+0.484]$	$<.001$
	qwen3-6	$+0.325$	$[+0.272,+0.378]$	$<.001$
	sonnet-5	$+0.595$	$[+0.544,+0.647]$	$<.001$

Table 9: RQ 2 three-path decomposition, every path estimated in the registered mixed-model family with a repository random intercept, fitted separately per LLM. Path (a) is

\log(\mathit{footprint})\sim\mathit{health}+\mathit{language}+(1|\mathit{repo})

; paths (b) and (c) are the

\log(\mathit{footprint})

and

\mathit{health}

coefficients of the primary fit in Table 8 . Takeaway: CodeHealth is not associated with PR footprint for any LLM, footprint is the strongest driver of cost for every LLM, and the CH effect on cost persists with footprint held constant, so the effect is direct with respect to PR footprint; Section 4.6.4 examines file size.

LLM	Smell	$\rho_{\mathrm{adj}}$	$p_{\mathrm{Holm}}$	Confirmed
deepseek-v4-flash	Complex Method	$.153$	$.002$	Yes
	Code Duplication	$.080$	$.089$	No
	Complex Conditional	$.212$	$<.001$	Yes
qwen3-6	Complex Method	$.207$	$<.001$	Yes
	Code Duplication	$.013$	$.774$	No
	Complex Conditional	$.157$	$.001$	Yes

Table 10: RQ 3 confirmatory test on the main collection: partial Spearman correlation with

\log_{1p}

fix-agent tokens, controlling for

\log

changed lines, Holm-corrected within each LLM’s three-hypothesis family. Takeaway: Complex Method and Complex Conditional are confirmed for all three LLMs; Code Duplication is confirmed for none of them, and its sign even reverses for sonnet-5 , so only two of the three pre-registered smell hypotheses are supported.

LLM	$n$	Median	Mean
deepseek-v4-flash	453	$0.000$	$-0.007$
qwen3-6	461	$0.000$	$-0.026$
sonnet-5	453	$0.000$	$-0.008$

Table 11:

\Delta

CodeHealth under a single batch fix pass, per LLM. Takeaway: The batch fix leaves CodeHealth essentially unchanged at the median for every LLM, replicating the candidate finding from pilot 3 on the full main collection.

Panel of Figure 6 — Figure 6: Distribution of $\Delta$ CodeHealth by baseline CodeHealth tier (unhealthy, problematic, healthy, colored red/yellow/green respectively), with the reference line at zero. Takeaway: The mass sits on the zero line in every tier and every LLM, but the spread is two-sided, so the zero median summarizes a distribution in which individual PRs both improve and degrade.

LLM	Outcome	$n$	$\beta_{\mathit{health}}$	95% CI	$p$	$q_{\mathrm{BH}}$	%/point
deepseek-v4-flash	Total tokens ‡	453	$-0.062$	$[-0.090,-0.035]$	$<.001$	—	$-6.1\%$
	Review tokens	453	$-0.069$	$[-0.097,-0.041]$	$<.001$	$<.001$	$-6.7\%$
	Fix tokens	424	$-0.034$	$[-0.067,-0.001]$	$.041$	$.064$	$-3.4\%$
qwen3-6	Total tokens ‡	461	$-0.049$	$[-0.085,-0.014]$	$.006$	—	$-4.8\%$
	Review tokens †	461	$-0.059$	$[-0.097,-0.021]$	$.002$	$.007$	$-5.8\%$
	Fix tokens	379	$-0.030$	$[-0.065,0.004]$	$.087$	$.126$	$-3.0\%$

Table 12: The registered model form refitted on each pipeline stage separately, one fit per LLM per outcome, on the same PRs as Table 8 . Fix-stage fits drop the PRs with zero review comments:

n

reports the rows that remain. Takeaway: The CodeHealth effect survives removing the fix stage from the outcome entirely, and in fact strengthens for every LLM, so the association lies with the review stage and not with the fix stage that follows it.

LLM	Review-stage outcome	$n$	$\beta_{\mathit{health}}$	95% CI	$p$	$q_{\mathrm{BH}}$
deepseek-v4-flash	Iterations §	453	$-0.037$	$[-0.053,-0.021]$	$<.001$	$<.001$
	Tokens per iteration	453	$-0.032$	$[-0.046,-0.018]$	$<.001$	$<.001$
	Input, uncached	453	$-0.045$	$[-0.063,-0.026]$	$<.001$	$<.001$
	Output	453	$-0.027$	$[-0.050,-0.004]$	$.021$	$.038$
	Uncached input + output §	453	$-0.038$	$[-0.056,-0.020]$	$<.001$	$<.001$
	Review comments	424	$+0.010$	$[-0.012,0.032]$	$.373$	$.447$

Table 13: Decomposition of the review-stage effect, same model form and same rows as Table 12 . Because

\log(\mathit{review\ tokens})=\log(\mathit{iterations})+\log(\mathit{tokens\ per\ iteration})

, the first two coefficients in each block sum to that LLM’s review-token coefficient by construction. Comment counts, and the per-comment ratio, are modeled on the PRs with at least one comment. The last row of each block corresponds to the review-token row of Table 12 minus the sixth row of this block, but it is fitted independently on the ratio, on the smaller sample that having at least one comment requires. Takeaway: Lower CodeHealth is associated with more agent turns and more tokens within each turn, and it shows most consistently on the input side, yet it does not correlate with additional review comments, so the additional spend buys no additional output and the cost per review finding rises accordingly.

LLM	Outcome	$n$	$\beta$	95% CI	$p$	$q_{\mathrm{BH}}$
Smell count , $\beta_{\log(1+\mathit{smells})}$
deepseek-v4-flash	Review comments	424	$+0.210$	$[+0.104,+0.315]$	$<.001$	$<.001$
qwen3-6	Review comments	379	$+0.249$	$[+0.120,+0.377]$	$<.001$	$.001$
sonnet-5	Review comments	369	$+0.142$	$[+0.025,+0.258]$	$.017$	$.034$
CodeHealth score , $\beta_{\mathit{health}}$ , same fits
deepseek-v4-flash	Review comments	424	$+0.019$	$[-0.012,+0.049]$	$.224$	$.304$

Table 14: Exploratory refit of the comment-count outcome with both the CodeHealth score and the smell count as predictors on the same rows, controlling for

\log

changed lines,

\log

scored-file lines,

\log

changed files, language, and a repo random intercept. Comment counts are modeled on the PRs with at least one comment, as in Table 13 . Takeaway: The number of findings the agents report tracks how many smells CodeScene counted, while remaining unrelated to the CH score those smells produce, so the null in Table 13 reflects the predictor and is not evidence that the agents attend to unrelated properties of the code.

LLM	Fit	$n$	$\beta_{\mathit{health}}$	95% CI	$p$	$q_{\mathrm{BH}}$	$\beta_{\log\mathit{size}}$ [95% CI]
deepseek-v4-flash	Registered ‡	453	$-0.062$	$[-0.090,-0.035]$	$<.001$	—	—
	+ file size	453	$-0.009$	$[-0.044,0.026]$	$.618$	$.665$	$+0.179$ $[0.104,0.254]$
qwen3-6	Registered ‡	461	$-0.049$	$[-0.085,-0.014]$	$.006$	—	—
	+ file size	461	$-0.023$	$[-0.071,0.025]$	$.352$	$.447$	$+0.082$ $[-0.021,0.185]$
sonnet-5	Registered ‡	453	$-0.036$	$[-0.070,-0.001]$	$.041$	—	—
	+ file size	453	$+0.018$	$[-0.027,0.064]$	$.430$	$.501$	$+0.171$ $[0.074,0.268]$

Table 15: The registered model refitted with

\log

total lines of the scored files added, one fit per LLM on the same rows as Table 8 . The size coefficient is the change in log tokens per unit of log file lines; it is reported as an estimate and is not part of the corrected family. Takeaway: Adjusting for file size absorbs the CodeHealth association for all three LLMs, which in this sample cannot be separated from the size of the files it is computed over.

LLM	PRs complete	Var(between)	Var(within)	$\lambda$	ICC( $k=1$ )
deepseek-v4-flash	25/27	0.939	0.052	0.055	.948
qwen3-6	27/27	0.713	0.313	0.440	.695
sonnet-5	27/27	2.390	0.095	0.040	.962

Table 16: Run-to-run variance decomposition on the 27-PR replication subsample (three runs per PR per LLM).

\lambda

is the run-to-run variance divided by the between-PR variance; ICC(

k=1

) is the implied reliability of a single run. Takeaway: qwen3-6 ’s run-to-run noise is about an order of magnitude larger, relative to its stable PR signal, than either of the other two LLMs, while deepseek-v4-flash and sonnet-5 are both highly reliable at a single run.

Software Engineering

Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.

See all 270 papers

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback
Mar 27, 2026Deepak KumarAutomated Code Review
Independent Researcher
Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding tasks
Aug 26, 2026Jakub SmékalCoding Agents Agentic Code Generation
Stanford University
Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study
May 19, 2026Priyansh Trivedi, Olivier SchmittCoding Agents AI Agent Evaluation
SonarSource

Mar 27, 2026J/K move · Enter open · S save

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

Deepak Kumar

Independent Researcher

Automated Code Review

Open paper AskPDF arXiv

We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at kappa=0.75, 8 frontier models detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations: diff only (config_A), diff with file content (config_B), and full context (config_C), enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113). Dataset, contexts, annotations, and evaluation harness are released publicly.

Abstract

Software Engineering

Explore similar work

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding tasks

Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study