The rapid growth of automated code generation has created a pressing need for automated code review. Yet code review resists full automation: it depends on context beyond the diff, on the author's intent, and on judgments about design and maintainability on which even expert reviewers disagree. Existing code review benchmarks lack standardization and rely largely on noisy human review comments scored with text-similarity metrics, which reward surface overlap and fail to measure precision. We introduce CR-Bench, a defect-centric evaluation framework that recasts SWE-Bench issues as pull-request review tasks with full repository context. After filtering for review-detectable defects and grounding each to a single originating pull request, CR-Bench contains 584 instances, with a 174-instance CR-Bench-Verified subset derived from SWE-Bench Verified. Each instance pairs a pull request with a known defect, its ground-truth fix, and a taxonomy label for bug category, impact, and severity. We propose two complementary metrics: Known-Defect Detection Rate (KDDR), which measures whether an agent identifies the labeled defect, and Usefulness Rate, which measures the fraction of review comments that are actionable. We evaluate eight code review agents over four backbones, plus commercially available Gemini Code Assist, using three independent LLM judges. Even the strongest systems detect only about one-third of known defects (KDDR ≈ 33%), and Gemini Code Assist detects 8.1%.
Figures & tables
Figure 1: CR-Bench framework. Offline preparation turns SWE-Bench defects into PR-context instances with taxonomy labels. An LLM judge classifies agent comments, and human annotations validate the classifications. KDDR measures labeled-defect detection. Usefulness Rate measures developer utility.
Table 2
Figure 2: Judge-specific KDDR (left) and Usefulness Rate (right) for the same fixed review comments on all 174 verified instances under the balanced prompt. Single = single-shot, Refl. = Reflexion, and Gemini = Gemini Code Assist. Marker shape identifies the judge. Points are full-sample estimates. Bars are 95% percentile confidence intervals from 1,000 instance-level bootstrap resamples.
Figure 3: Three-judge mean KDDR and Usefulness Rate per system (Table 7 ). Arrows go from single-shot (hollow) to Reflexion (filled). The solid line is the Pareto frontier.
Backbone
Claude
GPT-5.4
DeepSeek
GPT-5.2
+5.7
+3.4
+2.9
GPT-5-mini
+6.9 ∗
+11.5 ∗
+12.6 ∗
Gemma 4 31B
+6.9 ∗
+8.6 ∗
+7.5 ∗
GPT-OSS 20B
+6.3 ∗
+6.3 ∗
+6.3
Table 2: Reflexion minus single-shot KDDR (percentage points) on CR-Bench-Verified , per backbone and judge. ∗ p<0.05 , two-sided paired permutation test over instances. Confidence intervals and the other metrics are in Table 9 , Appendix F .
Figure 4: KDDR for every taxonomy group (Table 11 , Appendix F ), one panel per backbone. Hollow gray markers are single-shot, filled blue markers are Reflexion, and arrows point from single-shot to Reflexion. Each value averages the three judges under the balanced prompt. Faded rows marked † have fewer than 10 instances and are shown for completeness only. RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Fixed
PR
lines
comments
CR-Bench
10.28
41.03
CR-Bench-Verified
8.69
35.83
Appendix
Table 4: Per-instance means for all 584 CR-Bench instances and the 174-instance verified subset. Fixed lines count lines changed by the known repair. PR comments count comments in the originating pull request.
Figure 5: Tasks per source repository in CR-Bench-Verified (174 instances). Repositories are inherited from SWE-Bench Verified. Django is the largest repository with 63 tasks (36.2%), and Django and SymPy together account for just over half of the benchmark (54.6%). The remaining nine repositories span documentation tooling (Sphinx), scientific computing (Matplotlib, scikit-learn, Astropy, xarray), testing and linting (pytest, pylint), and HTTP and web libraries (Requests, Flask).
Figure 6: One CR-Bench instance. The PR context (top) concerns Django’s escaped-URL punctuation handling. Its known defect is that trim_punctuation() slices escaped URLs with indices derived from the unescaped string. The repair rδ (bottom left) computes the removed punctuation count before slicing. Red and green mark removed and added code. The target comment rc (bottom right) identifies the same defect, so its label is Bug Hit.
Figure 7: Taxonomy Labels for PR defects including Category, Impact, and Severity.
Figure 8: Pairwise counts for the 174 CR-Bench-Verified instances. Panels show category by severity (left), category by impact (center), and severity by impact (right). Only observed taxonomy levels appear. RFF = Requirements, Features and Functionality. IIS = Interface, Integration and System.
Figure 9: Pairwise cross-tabulations for all 584 CR-Bench instances. The panels show category by severity (left), category by impact (center), and severity by impact (right). Each cell counts instances. Only observed taxonomy levels are shown. RFF = Requirements, Features and Functionality. IIS = Interface, Integration and System.
instance_id
django__django-12708
Repository
django/django
Bug Origin PR
8432
PR Link
https://github.com/django/django/pull/8432
Known Bug Description
Deleting an index_together could crash when the same columns are also in unique_together because the schema editor searched for any constraint matching the columns and index=True , which also matches unique constraints. In alter_index_together() it called: self._delete_composed_index(model, fields, {’index’: True}, self.sql_delete_index) This can return both the unique and non-unique index constraints, causing _delete_composed_index() to raise ValueError due to multiple matches. The fix constrains the lookup to non-unique indexes ( {’index’: True, ’unique’: False} ), ensuring only the intended index is deleted.
Figure 10: Alternative Annotator Test (Alt-Test) Results. (a) The LLM’s winning rate ( ω ) across varying cost-benefit penalties ( ϵ ) ( N=60 ). The LLM successfully replaces the human majority ( ω>0.5 ) at ϵ≥0.25 . (b) The effect of sample size on winning rate, averaged over 20 random iterations. Test reaches statistical significance within the available data.
Metric
Setting
Value( % )
IAA score
k=1
71.66%
k=2
71.66%
k=3
28.33%
Winning rate
ϵ=0.1
0%
ϵ=0.3
67%
ϵ=0.5
100%
Appendix
Table 6: IAA score and Alt-Test winning rate( ω ) ( Calderon et al., 2025 ) for classifying reviews using claude-4.5-sonnet .
Figure 11: Pairwise raw agreement over 60 review comments labeled as Bug Hit, Valid Suggestion, or Noise by three human experts and Claude Sonnet 4.5. Each cell is the proportion of identical labels. Diagonal cells show self-agreement.
Figure 12: Human majority vs. AI prediction confusion matrix across three categories (Noise, Valid Suggestion, Bug Hit). Each cell reports the number of instances with a given human majority label (rows) and corresponding AI prediction (columns). The strong diagonal for Bug Hit indicates high alignment on objective cases, while off-diagonal mass, particularly from Bug Hit to Valid Suggestion, highlights systematic ambiguity and the AI’s tendency to produce softer, non-definitive judgments in borderline instances.
Figure 13: Distribution of per-sample inter-annotator entropy across 60 code review comments, computed over four annotators (three human, one AI). An entropy of 0.00 bits indicates unanimous agreement, 0.81 bits reflects a 3-vs-1 split, and 1.50 bits indicates a three-way split across all label categories. The majority of samples (63%) exhibit partial disagreement with a single dissenting annotator.
Figure 14: Shannon entropy of the label distribution for each annotator, computed over 60 code review comments across three categories: Bug Hit, Noise, and Valid Suggestion. The dashed line denotes the maximum entropy ( Hmax = 1.58 bits) for a uniform distribution over three classes. Human-1 exhibits the most concentrated labeling (1.21 bits), while AI produces the most uniform distribution (1.55 bits), suggesting greater diversity in its label assignments.
Figure 15: Judge-specific Known-Defect Detection Rate (KDDR) for the same fixed review comments on all 174 verified instances under the balanced prompt. Points are full-sample estimates. Bars are 95% percentile confidence intervals from 1,000 instance-level bootstrap resamples. Marker shape identifies the judge independently of color.
System
KDDR
Usefulness Rate
GPT-5.2 single-shot
28.9 ± 2.3
86.2 ± 11.3
GPT-5.2 Reflexion
33.0 ± 1.3
89.8 ± 9.5
GPT-5-mini single-shot
23.4 ± 2.3
69.0 ± 29.9
GPT-5-mini Reflexion
33.7 ± 4.9
72.6 ± 28.4
Gemma 4 31B single-shot
14.0 ± 1.2
64.5 ± 27.8
Gemma 4 31B Reflexion
21.6 ± 2.1
68.6 ± 27.3
Appendix
Table 7: Performance on CR-Bench-Verified . Each cell is the mean ± sample standard deviation over the three judges (GPT-5.4, Claude Sonnet 4.5, DeepSeek V4 Flash, n=3 ) under the balanced prompt. The deviation measures spread across evaluators, not sampling error; per-judge estimates with instance-bootstrap confidence intervals are in Table 8 . Percentage metrics omit the % symbol.
Review system
Evaluation judge
KDDR (%) [95% CI]
Usefulness Rate (%) [95% CI]
GPT-5.2 single-shot
Claude Sonnet 4.5
26.4 [20.1,33.9]
81.7 [77.9,85.5]
GPT-5.4
31.0 [24.7,37.9]
99.1 [98.6,99.5]
DeepSeek V4 Flash
29.3 [23.0,37.4]
77.8 [75.5,80.4]
GPT-5.2 Reflexion
Claude Sonnet 4.5
32.2 [25.8,39.1]
89.5 [86.3,92.7]
GPT-5.4
34.5 [27.6,42.0]
99.4 [98.9,99.8]
DeepSeek V4 Flash
32.2 [25.9,39.1]
80.4 [78.3,82.7]
Appendix
Table 8: Performance by evaluation judge under the balanced prompt. Each judge independently classifies the same fixed review comments. Point estimates use all 174 instances. The 95% percentile confidence intervals use 1,000 instance-level bootstrap resamples (seed 42), with comments grouped by instance. KDDR measures detection of the labeled defect, and Usefulness Rate measures the percentage of beneficial comments. See Section 4.1 for metric definitions and the statistical protocol.
Backbone
Evaluation judge
Δ KDDR [CI], p
Δ Usefulness Rate [CI], p
GPT-5.2
Claude Sonnet 4.5
+5.7 [-0.0,11.5], 0.083
+7.8 [5.2,10.7], 0.001
GPT-5.4
+3.4 [-2.3,9.2], 0.336
+0.3 [-0.2,0.9], 0.284
DeepSeek V4 Flash
+2.9 [-3.4,9.2], 0.504
+2.6 [-0.2,5.3], 0.066
GPT-5-mini
Claude Sonnet 4.5
+6.9 [1.1,13.2], 0.036
+3.7 [0.2,7.4], 0.101
GPT-5.4
+11.5 [4.6,18.4], 0.002
+2.1 [0.4,3.9], 0.012
DeepSeek V4 Flash
+12.6 [5.7,19.5], 0.004
+5.1 [1.2,9.0], 0.032
Appendix
Table 9: Paired Reflexion-minus-single-shot differences for the same backbone and evaluation judge. Positive differences favor Reflexion. Strategies are matched by instance_id . Confidence intervals use 1,000 paired bootstrap resamples, and p -values use two-sided paired permutation tests with 1,000 permutations. Gemini Code Assist is excluded because it has no matched single-shot and Reflexion pair. KDDR and Usefulness Rate differences are in percentage points. See Section 4.1 for metric definitions and the statistical protocol.
Single-shot
Reflexion
Evaluation judge
Prompt
KDDR
U
KDDR
U
Claude Sonnet 4.5
Balanced
26.4
81.7
32.2
89.5
Strict
25.9
38.2
32.8
50.7
Lenient
27.0
44.0
32.2
52.7
GPT-5.4
Balanced
31.0
99.1
34.5
99.4
Strict
24.7
94.3
32.2
98.8
Appendix
Table 10: Prompt robustness for fixed GPT-5.2 single-shot and Reflexion outputs on all 174 verified instances. Each judge classifies the same comments under balanced, strict, and lenient evaluator prompts. KDDR and Usefulness Rate (U) are percentages. Higher is better.
Dimension
Group
n
GPT-5.2
GPT-5-mini
Gemma 4 31B
GPT-OSS 20B
Category
Structural
144
31.5 → 35.9
25.9 → 34.3
14.1 → 22.9
7.4 → 14.1
IIS
19
21.1 → 29.8
17.5 → 36.8
21.1 → 24.6
8.8 → 14.0
RFF †
7
0.0 → 0.0
0.0 → 14.3
0.0 → 0.0
0.0 → 4.8
Data †
3
33.3 → 0.0
0.0 → 33.3
0.0 → 0.0
0.0 → 0.0
Memory †
1
0.0 → 0.0
0.0 → 33.3
0.0 → 0.0
0.0 → 0.0
Impact
Functional Suitability
105
28.9 → 29.5
20.3 → 30.5
13.7 → 19.7
6.7 → 13.0
Appendix
Table 11: KDDR (%) for every taxonomy group, written as single-shot → Reflexion. Each value averages the three judges under the balanced prompt. n is the number of CR-Bench-Verified instances in the group. † Groups with fewer than 10 instances are listed for completeness only. Their confidence intervals are tens of percentage points wide, so we draw no conclusions from them. The dimensions overlap: 12 of the 13 Usability instances are also the Low-severity instances, and Reliability is mostly high severity (26 High, 15 Medium, 0 Low). Differences across impact groups therefore partly reflect severity. Figure 16 plots the severity rows with confidence intervals. Figure 4 plots every row. RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System.
Figure 16: KDDR by defect severity for each backbone. Points average the three judges under the balanced prompt, and bars are 95% bootstrap confidence intervals. The number of instances per severity level is printed on the axis. Table 11 reports the same numbers for every category and impact group.
Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.
Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.
Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf +3
National University of Singapore · Zhejiang University · SonarSource
We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at kappa=0.75, 8 frontier models detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations: diff only (config_A), diff with file content (config_B), and full context (config_C), enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113). Dataset, contexts, annotations, and evaluation harness are released publicly.