CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents

Kristen Pereira; Neelabh Sinha; Rajat Ghosh; Debojyoti Dutta

cs.SENov 2, 2026

CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents

Authors: Kristen Pereira, Neelabh Sinha, Rajat Ghosh, Debojyoti Dutta

Organizations: Nutanix, Inc.

Abstract

The rapid growth of automated code generation has created a pressing need for automated code review. Yet code review resists full automation: it depends on context beyond the diff, on the author's intent, and on judgments about design and maintainability on which even expert reviewers disagree. Existing code review benchmarks lack standardization and rely largely on noisy human review comments scored with text-similarity metrics, which reward surface overlap and fail to measure precision. We introduce CR-Bench, a defect-centric evaluation framework that recasts SWE-Bench issues as pull-request review tasks with full repository context. After filtering for review-detectable defects and grounding each to a single originating pull request, CR-Bench contains 584 instances, with a 174-instance CR-Bench-Verified subset derived from SWE-Bench Verified. Each instance pairs a pull request with a known defect, its ground-truth fix, and a taxonomy label for bug category, impact, and severity. We propose two complementary metrics: Known-Defect Detection Rate (KDDR), which measures whether an agent identifies the labeled defect, and Usefulness Rate, which measures the fraction of review comments that are actionable. We evaluate eight code review agents over four backbones, plus commercially available Gemini Code Assist, using three independent LLM judges. Even the strongest systems detect only about one-third of known defects (KDDR $\approx$ 33%), and Gemini Code Assist detects 8.1%.

Figures & tables

Figure 1: CR-Bench framework. Offline preparation turns SWE-Bench defects into PR-context instances with taxonomy labels. An LLM judge classifies agent comments, and human annotations validate the classifications. KDDR measures labeled-defect detection. Usefulness Rate measures developer utility.

Table 2

Figure 2: Judge-specific KDDR (left) and Usefulness Rate (right) for the same fixed review comments on all 174 verified instances under the balanced prompt. Single = single-shot, Refl. = Reflexion, and Gemini = Gemini Code Assist. Marker shape identifies the judge. Points are full-sample estimates. Bars are 95% percentile confidence intervals from 1,000 instance-level bootstrap resamples.

Figure 3: Three-judge mean KDDR and Usefulness Rate per system (Table 7 ). Arrows go from single-shot (hollow) to Reflexion (filled). The solid line is the Pareto frontier.

Backbone	Claude	GPT-5.4	DeepSeek
GPT-5.2	+5.7	+3.4	+2.9
GPT-5-mini	+6.9 ∗	+11.5 ∗	+12.6 ∗
Gemma 4 31B	+6.9 ∗	+8.6 ∗	+7.5 ∗
GPT-OSS 20B	+6.3 ∗	+6.3 ∗	+6.3

Table 2: Reflexion minus single-shot KDDR (percentage points) on CR-Bench-Verified , per backbone and judge. ∗

p<0.05

, two-sided paired permutation test over instances. Confidence intervals and the other metrics are in Table 9 , Appendix F .

Figure 4: KDDR for every taxonomy group (Table 11 , Appendix F ), one panel per backbone. Hollow gray markers are single-shot, filled blue markers are Reflexion, and arrows point from single-shot to Reflexion. Each value averages the three judges under the balanced prompt. Faded rows marked † have fewer than 10 instances and are shown for completeness only. RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Fixed	PR
	lines	comments
CR-Bench	10.28	41.03
CR-Bench-Verified	8.69	35.83

Appendix

Table 4: Per-instance means for all 584 CR-Bench instances and the 174-instance verified subset. Fixed lines count lines changed by the known repair. PR comments count comments in the originating pull request.

Figure 5: Tasks per source repository in CR-Bench-Verified (174 instances). Repositories are inherited from SWE-Bench Verified. Django is the largest repository with 63 tasks (36.2%), and Django and SymPy together account for just over half of the benchmark (54.6%). The remaining nine repositories span documentation tooling (Sphinx), scientific computing (Matplotlib, scikit-learn, Astropy, xarray), testing and linting (pytest, pylint), and HTTP and web libraries (Requests, Flask).

Panel of Figure 6 — Figure 6: One CR-Bench instance. The PR context (top) concerns Django’s escaped-URL punctuation handling. Its known defect is that trim_punctuation() slices escaped URLs with indices derived from the unescaped string. The repair $r_{\delta}$ (bottom left) computes the removed punctuation count before slicing. Red and green mark removed and added code. The target comment $r_{c}$ (bottom right) identifies the same defect, so its label is Bug Hit.

Figure 7: Taxonomy Labels for PR defects including Category, Impact, and Severity.

Figure 8: Pairwise counts for the 174 CR-Bench-Verified instances. Panels show category by severity (left), category by impact (center), and severity by impact (right). Only observed taxonomy levels appear. RFF = Requirements, Features and Functionality. IIS = Interface, Integration and System.

Figure 9: Pairwise cross-tabulations for all 584 CR-Bench instances. The panels show category by severity (left), category by impact (center), and severity by impact (right). Each cell counts instances. Only observed taxonomy levels are shown. RFF = Requirements, Features and Functionality. IIS = Interface, Integration and System.

instance_id	django__django-12708
Repository	django/django
Bug Origin PR	8432
PR Link	https://github.com/django/django/pull/8432
Known Bug Description	Deleting an index_together could crash when the same columns are also in unique_together because the schema editor searched for any constraint matching the columns and index=True , which also matches unique constraints. In alter_index_together() it called: self._delete_composed_index(model, fields, {’index’: True}, self.sql_delete_index) This can return both the unique and non-unique index constraints, causing _delete_composed_index() to raise ValueError due to multiple matches. The fix constrains the lookup to non-unique indexes ( {’index’: True, ’unique’: False} ), ensuring only the intended index is deleted.
Known Fix Patch	diff --git a/django/db/backends/base/schema.py b/django/db/backends/base/schema.py --- a/django/db/backends/base/schema.py +++ b/django/db/backends/base/schema.py @@ -393,7 +393,12 @@ def alter_index_together(self, model, old_index_together, new_index_together): news = {tuple(fields) for fields in new_index_together} # Deleted indexes for fields in olds.difference(news): - self._delete_composed_index(model, fields, {’index’: True}, self.sql_delete_index) + self._delete_composed_index( + model, + fields, + {’index’: True, ’unique’: False}, + self.sql_delete_index, + )
Files Changed in Fix	django/db/backends/base/schema.py

Appendix

Table 5: An example of a human review annotation

Panel of Figure 10 — Figure 10: Alternative Annotator Test (Alt-Test) Results. (a) The LLM’s winning rate ( $\omega$ ) across varying cost-benefit penalties ( $\epsilon$ ) ( $N=60$ ). The LLM successfully replaces the human majority ( $\omega>0.5$ ) at $\epsilon\geq 0.25$ . (b) The effect of sample size on winning rate, averaged over 20 random iterations. Test reaches statistical significance within the available data.

Metric	Setting	Value( $\%$ )
IAA score	$k=1$	$71.66\%$
	$k=2$	$71.66\%$
	$k=3$	$28.33\%$
Winning rate	$\epsilon=0.1$	$0\%$
	$\epsilon=0.3$	$67\%$
	$\epsilon=0.5$	$100\%$

Appendix

Table 6: IAA score and Alt-Test winning rate(

\omega

) ( Calderon et al., 2025 ) for classifying reviews using claude-4.5-sonnet .

Figure 11: Pairwise raw agreement over 60 review comments labeled as Bug Hit, Valid Suggestion, or Noise by three human experts and Claude Sonnet 4.5. Each cell is the proportion of identical labels. Diagonal cells show self-agreement.

Figure 12: Human majority vs. AI prediction confusion matrix across three categories (Noise, Valid Suggestion, Bug Hit). Each cell reports the number of instances with a given human majority label (rows) and corresponding AI prediction (columns). The strong diagonal for Bug Hit indicates high alignment on objective cases, while off-diagonal mass, particularly from Bug Hit to Valid Suggestion, highlights systematic ambiguity and the AI’s tendency to produce softer, non-definitive judgments in borderline instances.

Figure 13: Distribution of per-sample inter-annotator entropy across 60 code review comments, computed over four annotators (three human, one AI). An entropy of 0.00 bits indicates unanimous agreement, 0.81 bits reflects a 3-vs-1 split, and 1.50 bits indicates a three-way split across all label categories. The majority of samples (63%) exhibit partial disagreement with a single dissenting annotator.

Figure 14: Shannon entropy of the label distribution for each annotator, computed over 60 code review comments across three categories: Bug Hit, Noise, and Valid Suggestion. The dashed line denotes the maximum entropy (

H_{\max}

= 1.58 bits) for a uniform distribution over three classes. Human-1 exhibits the most concentrated labeling (1.21 bits), while AI produces the most uniform distribution (1.55 bits), suggesting greater diversity in its label assignments.

Figure 15: Judge-specific Known-Defect Detection Rate (KDDR) for the same fixed review comments on all 174 verified instances under the balanced prompt. Points are full-sample estimates. Bars are 95% percentile confidence intervals from 1,000 instance-level bootstrap resamples. Marker shape identifies the judge independently of color.

System	KDDR	Usefulness Rate
GPT-5.2 single-shot	28.9 $\pm$ 2.3	86.2 $\pm$ 11.3
GPT-5.2 Reflexion	33.0 $\pm$ 1.3	89.8 $\pm$ 9.5
GPT-5-mini single-shot	23.4 $\pm$ 2.3	69.0 $\pm$ 29.9
GPT-5-mini Reflexion	33.7 $\pm$ 4.9	72.6 $\pm$ 28.4
Gemma 4 31B single-shot	14.0 $\pm$ 1.2	64.5 $\pm$ 27.8
Gemma 4 31B Reflexion	21.6 $\pm$ 2.1	68.6 $\pm$ 27.3

Appendix

Table 7: Performance on CR-Bench-Verified . Each cell is the mean

\pm

sample standard deviation over the three judges (GPT-5.4, Claude Sonnet 4.5, DeepSeek V4 Flash,

n=3

) under the balanced prompt. The deviation measures spread across evaluators, not sampling error; per-judge estimates with instance-bootstrap confidence intervals are in Table 8 . Percentage metrics omit the % symbol.

Review system	Evaluation judge	KDDR (%) [95% CI]	Usefulness Rate (%) [95% CI]
GPT-5.2 single-shot	Claude Sonnet 4.5	26.4 [20.1,33.9]	81.7 [77.9,85.5]
	GPT-5.4	31.0 [24.7,37.9]	99.1 [98.6,99.5]
	DeepSeek V4 Flash	29.3 [23.0,37.4]	77.8 [75.5,80.4]
GPT-5.2 Reflexion	Claude Sonnet 4.5	32.2 [25.8,39.1]	89.5 [86.3,92.7]
	GPT-5.4	34.5 [27.6,42.0]	99.4 [98.9,99.8]
	DeepSeek V4 Flash	32.2 [25.9,39.1]	80.4 [78.3,82.7]

Appendix

Table 8: Performance by evaluation judge under the balanced prompt. Each judge independently classifies the same fixed review comments. Point estimates use all 174 instances. The 95% percentile confidence intervals use 1,000 instance-level bootstrap resamples (seed 42), with comments grouped by instance. KDDR measures detection of the labeled defect, and Usefulness Rate measures the percentage of beneficial comments. See Section 4.1 for metric definitions and the statistical protocol.

Backbone	Evaluation judge	$\Delta$ KDDR [CI], $p$	$\Delta$ Usefulness Rate [CI], $p$
GPT-5.2	Claude Sonnet 4.5	+5.7 [-0.0,11.5], 0.083	+7.8 [5.2,10.7], 0.001
	GPT-5.4	+3.4 [-2.3,9.2], 0.336	+0.3 [-0.2,0.9], 0.284
	DeepSeek V4 Flash	+2.9 [-3.4,9.2], 0.504	+2.6 [-0.2,5.3], 0.066
GPT-5-mini	Claude Sonnet 4.5	+6.9 [1.1,13.2], 0.036	+3.7 [0.2,7.4], 0.101
	GPT-5.4	+11.5 [4.6,18.4], 0.002	+2.1 [0.4,3.9], 0.012
	DeepSeek V4 Flash	+12.6 [5.7,19.5], 0.004	+5.1 [1.2,9.0], 0.032

Appendix

Table 9: Paired Reflexion-minus-single-shot differences for the same backbone and evaluation judge. Positive differences favor Reflexion. Strategies are matched by instance_id . Confidence intervals use 1,000 paired bootstrap resamples, and

p

-values use two-sided paired permutation tests with 1,000 permutations. Gemini Code Assist is excluded because it has no matched single-shot and Reflexion pair. KDDR and Usefulness Rate differences are in percentage points. See Section 4.1 for metric definitions and the statistical protocol.

		Single-shot		Reflexion
Evaluation judge	Prompt	KDDR	U	KDDR	U
Claude Sonnet 4.5	Balanced	26.4	81.7	32.2	89.5
	Strict	25.9	38.2	32.8	50.7
	Lenient	27.0	44.0	32.2	52.7
GPT-5.4	Balanced	31.0	99.1	34.5	99.4
	Strict	24.7	94.3	32.2	98.8

Appendix

Table 10: Prompt robustness for fixed GPT-5.2 single-shot and Reflexion outputs on all 174 verified instances. Each judge classifies the same comments under balanced, strict, and lenient evaluator prompts. KDDR and Usefulness Rate (U) are percentages. Higher is better.

Dimension	Group	$n$	GPT-5.2	GPT-5-mini	Gemma 4 31B	GPT-OSS 20B
Category	Structural	144	31.5 $\to$ 35.9	25.9 $\to$ 34.3	14.1 $\to$ 22.9	7.4 $\to$ 14.1
	IIS	19	21.1 $\to$ 29.8	17.5 $\to$ 36.8	21.1 $\to$ 24.6	8.8 $\to$ 14.0
	RFF †	7	0.0 $\to$ 0.0	0.0 $\to$ 14.3	0.0 $\to$ 0.0	0.0 $\to$ 4.8
	Data †	3	33.3 $\to$ 0.0	0.0 $\to$ 33.3	0.0 $\to$ 0.0	0.0 $\to$ 0.0
	Memory †	1	0.0 $\to$ 0.0	0.0 $\to$ 33.3	0.0 $\to$ 0.0	0.0 $\to$ 0.0
Impact	Functional Suitability	105	28.9 $\to$ 29.5	20.3 $\to$ 30.5	13.7 $\to$ 19.7	6.7 $\to$ 13.0

Appendix

Table 11: KDDR (%) for every taxonomy group, written as single-shot

\to

Reflexion. Each value averages the three judges under the balanced prompt.

n

is the number of CR-Bench-Verified instances in the group. † Groups with fewer than 10 instances are listed for completeness only. Their confidence intervals are tens of percentage points wide, so we draw no conclusions from them. The dimensions overlap: 12 of the 13 Usability instances are also the Low-severity instances, and Reliability is mostly high severity (26 High, 15 Medium, 0 Low). Differences across impact groups therefore partly reflect severity. Figure 16 plots the severity rows with confidence intervals. Figure 4 plots every row. RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System.

Figure 16: KDDR by defect severity for each backbone. Points average the three judges under the balanced prompt, and bars are 95% bootstrap confidence intervals. The number of instances per severity level is printed on the axis. Table 11 reports the same numbers for every category and impact group.

Automated Code Review

Automated code review uses software analysis and AI to find defects, assess changes, and support pull-request decisions.

See all 54 papers

CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents
Mar 10, 2026Kristen Pereira, Neelabh Sinha, Rajat Ghosh +1Automated Code Review AI Agent Evaluation
Nutanix, Inc.
Code Review Agent Benchmark
Mar 24, 2026Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf +3Automated Code Review Software Engineering Benchmarks
National University of Singapore · Zhejiang University · SonarSource
SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback
Mar 27, 2026Deepak KumarAutomated Code Review
Independent Researcher

Mar 10, 2026J/K move · Enter open · S save

CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents

Kristen Pereira, Neelabh Sinha, Rajat Ghosh, Debojyoti Dutta

Nutanix, Inc.

Automated Code Review AI Agent Evaluation LLM Agent Reliability

Open paper AskPDF arXiv

Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.

Abstract

Automated Code Review

Explore similar work

CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents

Code Review Agent Benchmark

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback