Organizations: Sun Yat-sen University, Zhuhai Key Laboratory of Trusted Large Language Models, Zhuhai, China · Chongqing University, Chongqing, China · Huawei Cloud Computing Technologies Co., Ltd., Dongguan, China
In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.
Figures & tables
Figure 1. Comparison of traditional hunk-level review, current PR-level code review, and real-world multi-round PR-level code review (this paper).
Figure 2. A task in MCR-Bench sourced from the GitHub Project huggingface/transformers ( Task instance source from huggingface/transformers., [n. d.] ) .
Figure 3. The Construction Pipeline of MCR-Bench
Figure 4. Statistics of MCR-Bench : language distribution, review round distribution, and defect number statistics.
ID
Name
Description
Ratio
E
Evolvability
Improvements that enhance future maintainability without changing externally observable behavior.
–
E.1
Documentation
Changes to in-code information to improve human understanding.
–
E.1.1
Textual
Comments, identifier or name changes, README text, and error-message wording.
19.19%
E.1.2
Language-Supported
Docstrings, type annotations, and language-supported documentation.
5.56%
E.2
Visual Representation
Layout or style changes for better readability.
10.25%
E.3
Structure
Changes to code organization or architecture.
–
Table 1. Defect taxonomy and distribution in MCR-Bench .
Severity
Description
Ratio
Trivial
Cosmetic-only problems, such as misspellings or misaligned text.
22.26%
Minor
Causes a minor loss of functionality, or a defect for which an easy workaround is available.
24.54%
Normal
A regular defect that leads to partial loss of functionality under specific conditions.
29.73%
Major
Causes a major loss of functionality.
20.21%
Blocker
Blocks development and/or testing work; the change cannot proceed or be merged until the defect is fixed.
2.32%
Critical
Causes crashes, data loss, or severe memory leaks.
0.93%
Table 2. Defect severity levels and their distribution in MCR-Bench .
Dataset
Review Granularity
#Review Rounds
#Tasks
#Languages
Context Scope
Min
Avg
Max
Trans-Review ( Tufano et al., 2021 )
Method
1
1.0
1
1,719
1
None
AutoTransform ( Thongtanunam et al., 2022 )
Method
1
1.0
1
14,750
1
None
T5-Review ( Tufano et al., 2022 )
Method
1
1.0
1
17,194
1
None
CodeReviewer ( Li et al., 2022 )
Diff Hunk
1
1.0
1
10,000
9
None
CR-Agent-Dataset ( Tang et al., 2024 )
Commit
1
1.0
1
3,545
9
Related Source Code
Table 3. Comparison between popular code review benchmarks and MCR-Bench .
Metric
QWK
ROUGE-L
0.21
BLEU-4
0.27
LLM Scoring (Claude Opus 4.5)
0.58
LLM Scoring (Gemini 3 Pro)
0.55
LLM Scoring (GPT-5.2 Pro)
0.60
LLM-Hit-Judge (Claude Opus 4.5)
0.70
Table 4. Agreement between automatic metrics and Human Hit Rate measured by QWK on the sampled MCR-Bench evaluation set.
Model
Python
Java
JavaScript
TypeScript
C#
Overall
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
Claude Haiku 4.5
0.556
0.619
0.540
0.561
0.643
0.548
0.640
0.664
0.599
0.552
0.594
0.520
0.587
0.631
0.548
0.579
0.630
0.551
DeepSeek V3.2
0.620
0.442
0.484
0.605
0.481
0.495
0.673
0.476
0.519
0.590
0.430
0.460
0.638
0.447
0.491
0.625
0.455
0.490
Gemini 3 Flash
0.649
0.548
0.554
0.594
0.562
0.529
0.653
0.564
0.553
0.571
0.491
0.486
0.638
0.540
0.538
0.621
0.541
0.532
GLM-4.7
0.597
0.446
0.476
0.604
0.465
0.491
0.626
0.464
0.489
0.553
0.402
0.425
0.602
0.432
0.467
0.596
0.442
0.470
GPT-5.2
0.651
0.563
0.563
0.582
0.574
0.527
0.682
0.566
0.573
0.584
0.545
0.516
0.609
0.557
0.533
0.622
0.561
0.542
Table 5. Performance across programming languages and overall results (Precision / Recall / F1).
Baselines
Model
Python
Java
JavaScript
TypeScript
C#
Overall
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
PR-Agent
Claude Haiku 4.5
0.273
0.236
0.232
0.299
0.306
0.276
0.311
0.278
0.268
0.290
0.260
0.248
0.308
0.266
0.259
0.296
0.269
0.257
DeepSeek V3.2
0.462
0.389
0.391
0.508
0.527
0.466
0.474
0.422
0.407
0.463
0.421
0.403
0.489
0.423
0.414
0.479
0.436
0.416
Gemini 3 Flash
0.271
0.127
0.161
0.315
0.148
0.188
0.351
0.181
0.222
0.292
0.147
0.182
0.328
0.152
0.193
0.311
0.151
0.189
GPT-5.2
0.352
0.283
0.279
0.393
0.420
0.356
0.389
0.353
0.323
0.340
0.325
0.289
0.383
0.318
0.303
0.371
0.340
0.310
Kimi K2
0.381
0.283
0.301
0.488
0.411
0.410
0.456
0.345
0.364
0.413
0.310
0.325
0.436
0.313
0.333
0.435
0.332
0.347
Table 6. Performance of baselines across programming languages and overall results (Precision / Recall / F1).
Model
Py
Java
JS
TS
C#
Overall
Claude Haiku 4.5
80.74%
82.77%
79.76%
78.36%
75.84%
79.69%
DeepSeek V3.2
72.53%
77.24%
73.12%
68.59%
69.34%
72.60%
GLM-4.7
61.39%
73.68%
61.69%
60.11%
61.25%
64.49%
GPT-5.2
70.40%
75.31%
66.60%
70.55%
71.04%
71.23%
Kimi K2
43.57%
48.97%
50.66%
45.00%
41.26%
45.95%
Qwen3 Max
47.26%
45.13%
44.41%
44.93%
40.82%
44.34%
Table 7. Accuracy of different models across programming languages and overall performance.
LLM
E.1.1
E.1.2
E.2
E.3.1
E.3.2
E.4
F.1
F.2
F.3
F.4
F.5
F.6
F.7
Claude Haiku 4.5
0.5899
0.7033
0.5945
0.5685
0.5923
0.5439
0.5805
0.6510
0.6436
0.6526
0.5170
0.5370
0.5714
DeepSeek V3.2
0.4165
0.4923
0.4377
0.4174
0.4097
0.3606
0.3702
0.4362
0.4078
0.4852
0.3693
0.3824
0.3333
Gemini 3 Flash
0.5065
0.5793
0.4957
0.4515
0.5091
0.4798
0.5000
0.5346
0.5498
0.5899
0.4500
0.4300
0.4737
GLM-4.7
0.3828
0.4361
0.4154
0.3619
0.3537
0.3589
0.3655
0.4428
0.3881
0.5123
0.4246
0.3512
0.4167
GPT-5.2
0.5161
0.6000
0.5094
0.4785
0.5008
0.4735
0.5292
0.5820
0.6029
0.6483
0.5028
0.5243
0.4872
Kimi K2
0.2807
0.3746
0.2061
0.1977
0.2748
0.2967
0.3552
0.4160
0.4363
0.4773
0.2924
0.3005
0.3714
Table 8. Hit Rate of different LLMs across taxonomy categories.
LLM
Trivial
Minor
Normal
Major
Blocker
Critical
Claude Haiku 4.5
0.5810
0.5609
0.5947
0.6707
0.6074
0.6607
DeepSeek V3.2
0.4216
0.3970
0.4143
0.4424
0.3858
0.4808
Gemini 3 Flash
0.4888
0.4815
0.4926
0.5761
0.5116
0.5472
GLM-4.7
0.3910
0.3523
0.3892
0.4608
0.3876
0.5000
GPT-5.2
0.4941
0.5069
0.5332
0.5964
0.4453
0.6667
Kimi K2
0.2149
0.2455
0.3522
0.4227
0.3440
0.3800
Table 9. Hit rates of LLMs across defect severity levels.
Figure 5. Distribution of LLMs’ error patterns in multi-round defect lifecycle state tracking.
LLM
R2
R3
R4
R5
R6
R7
R8
R9
R10
Claude Haiku 4.5
0.6495
0.6173
0.5359
0.5310
0.5313
0.4768
0.5227
0.5181
0.2857
DeepSeek V3.2
0.5291
0.5559
0.5359
0.5148
0.4714
0.3964
0.3449
0.3667
0.3333
Gemini 3 Flash
0.5983
0.5985
0.5554
0.5117
0.5371
0.5624
0.4910
0.3600
0.3333
GLM-4.7
0.5025
0.5093
0.5220
0.5334
0.4839
0.4827
0.4967
0.5333
0.3333
GPT-5.2
0.5995
0.5906
0.5659
0.5671
0.5067
0.6238
0.5000
0.6048
0.5000
Kimi K2
0.3946
0.3463
0.3571
0.3450
0.3509
0.3356
0.4410
0.3467
0.3333
Table 10. Average F1 score of different LLMs across increasing review rounds.
Type
Error Category
Description
Prop.
FP
State–Temporal Misalignment
Failure to align defect states with code versions, causing already resolved defects to be repeatedly flagged as new.
32.5%
Over-reviewing
The model continues to produce review comments in the absence of real defects, misclassifying suggestions or speculative concerns as defects.
27.8%
Lack of Domain Knowledge
Insufficient understanding of project- or domain-specific conventions leads to reasonable but non-general implementations being misidentified as defects.
22.4%
Version Compatibility Blindness
Ignoring language or runtime version constraints and backward-compatibility requirements, the model suggests infeasible best-practice changes.
12.1%
Other
–
5.2%
FN
Long-range Dependency Miss
Failure to capture cross-file dependencies or downstream impacts of local changes results in missing real defects.
23.4%
Table 11. Error taxonomy of False Positives (FP) and False Negatives (FN) in multi-round code review.
Model
Pure LLM
PR-Agent
Hybrid-Review
Rel.
Info.
Expr.
Avg.
Rel.
Info.
Expr.
Avg.
Rel.
Info.
Expr.
Avg.
Claude Haiku 4.5
0.7215
0.4329
0.9909
0.7151
0.6887
0.7076
0.8194
0.7386
0.3559
0.3828
0.6445
0.4611
DeepSeek V3.2
0.8891
0.5941
0.9553
0.8128
0.6624
0.6712
0.8684
0.7340
0.4654
0.4597
0.6599
0.5283
Gemini 3 Flash
0.8552
0.6407
0.7977
0.7646
0.3117
0.2175
0.5345
0.3546
0.2649
0.1659
0.4717
0.3008
GPT-5.2
0.8149
0.3598
0.9728
0.7158
0.7536
0.9302
0.9592
0.8810
0.6304
0.7968
0.8804
0.7692
Kimi K2
0.9216
0.5389
0.9467
0.8024
0.7773
0.6526
0.9236
0.7845
0.2940
0.2459
0.5274
0.3557
Table 12. ClearCRC-based quality evaluation results for generated review comments.
Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.
LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes. More specifically, we have built an automated agentic pipeline to autonomously harvest, build, and validate massive real-world bug-introducing commits for popular software systems. To ensure precise evaluation, we also adopt LLM-as-a-judge to credit the findings matching ground-truth vulnerabilities. To date, we have collected 265 tasks from nine critical systems, including Firefox, V8, and OpenSSL with millions of lines of code. Across eight frontier models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT-6 sol ranking at the top. Our study also shows that most misses are not simply oversights: in 68% of them, the agent reports on a file that holds the flaw but argues for a different mechanism. Security-focused harnesses also behave differently: OpenAI's Codex Security lowers recall by 8.3 points, mostly by abstaining, while Anthropic's Defending Code harness raises it by 7.7 points, at a higher cost.
Yihan Yang, Chenyuan Yang, Zijie Zhao +5
University of Illinois Urbana-Champaign · Quantum AI & Cyber Security Institute, FPT Corporation; work done during an internship at UIUC.
Code review is essential for maintaining software repositories by identifying defects, improving code quality, and supporting reliable software evolution. With the growing adoption of LLM-based coding agents, an increasing number of code contributions and repository changes are generated by these agents. Consequently, LLM-based code review agents are becoming increasingly prevalent to evaluate these contributions and maintain software quality. However, LLM reviewers often generate lengthy reviews containing numerous potential issues, many of which are false positives or lack sufficient evidence. These unreliable reviews can mislead developers and downstream coding agents, motivating the need to distinguish actionable findings from unsupported claims. We propose CIVER, a causal intervention-based framework that identifies the core issues underlying code-review findings and distinguishes genuine defects from false positives. CIVER integrates three modules: a review module assesses findings, an intervention module generates repair-oriented perturbations, and a verifier module scores review reliability. Through joint optimization via expert iteration, the three modules identify genuine defects by responding to targeted repairs while remaining stable under unrelated changes. Extensive experiments on SWE-Review-Bench demonstrate that CIVER correctly identifies 77% of genuine defects through targeted repairs. Cross-project and downstream repair evaluations further demonstrate that CIVER generalizes across repositories and improves coding agents' task success rates, highlighting its effectiveness for reliable automated code review.