Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.
Figures & tables
Figure 1: Example of an instance from CR-Bench dataset: In a pull request, a comment should be made to address the specified bug. The fix is removing the lines of code marked in RED and adding the ones marked in GREEN .
Method
# Tasks
Multiple Diffs
Full PR Context
Categorization
Defect-focused
Evaluation
Tufano et al.,2021
1718
✗
✗
✗
✗
Exact/BLEU
Tufano et al.,2022
147,533
✗
✗
✗
✗
Exact/BLEU
Thongtanunam et al.,2022
17K
✗
✗
✗
✗
Exact
Li et al.,2022
54K
✗
✗
✗
✗
CodeBLEU
Zeng et al.,2025b
1000
✓
✓
✓
✗
LLM Judge (P, R, F1)
CR-Bench (Ours)
584
✓
✓
✓
✓
LLM Judge (P, R, F1, U, SNR)
Table 1: Comparison of Benchmarks for Automated Code Review Evaluation. The data instances in CR-Bench are more defect-focused and real bugs found existing in large-scale open-source repositories. Also, alongside measuring the performance traditionally with precision, recall, F1, we also introduce developer trust and factuality parameters with usefulness score (U), and signal-to-noise ratio (SNR).
Figure 2: CR-Bench is a dataset for transforming real-world software defects into an objective code review benchmark with full Pull Request (PR) context and a multi-dimensional taxonomy of Category, Impact, Severity. CR-Evaluator is a method of evaluating code review agents and measuring them in terms of both performance and developer acceptance.
Table 2: The CR-Bench taxonomy elements for classifying code reviews (Note: RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System).
Dataset
# Lines fixed
# PR Comments
PR Description Length
CR-Bench
10.28
41.03
906.63
CR-Bench-verified
8.69
35.83
893.59
Table 3: Average statistics per instance for CR-Bench and its verified subset. All values represent means calculated across the respective dataset instances.
Figure 3: Comparative analysis of bug category, severity, and impact distributions between the verified subset and the full CR-Bench corpus (Note: RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System).
Agent
Model
Recall
Prec.
F1
Usefulness
SNR
Single-shot
GPT-5.2
27.01%
3.56%
6.30%
83.63%
5.11
GPT-5-mini
18.39%
3.51%
5.90%
74.29%
2.89
Reflexion
GPT-5.2
32.76%
5.10%
8.83%
66.10%
1.95
GPT-5-mini
27.59%
3.19%
5.72%
47.72%
0.91
Table 4: Comparative performance of code review agents on CR-Bench-verified on two techniques with two models. Best results are marked in BOLD .
Figure 4: PR Bug Recall analysis across Category, Impact, and Severity for CR-Bench-verified (Note: RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System).
Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.
Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf +3
National University of Singapore · Zhejiang University · SonarSource
Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers. We study technical trustworthiness judgment: determining whether a review comment's core technical claims are correct and applicable to the reviewed code in its repository context. Existing code-review benchmarks primarily evaluate review generation, issue discovery, or general comment quality, but do not directly assess whether an agent can determine the technical trustworthy of an individual review comment. To fill this gap, we introduce CRJudgeBench, a benchmark of 1199 instances constructed from real pull requests and expert-verified perturbations, covering both trustworthy and plausible but untrustworthy comments. We further present Sentinel, a repository-grounded agentic judge that actively gathers code evidence to verify review comments before making judgments. Starting from Qwen3-Coder-30B-A3B-Instruct, Sentinel is trained on the CRJudgeBench training split through iterative action-level learning from a privileged teacher. On the 359-instance CRJudgeBench test set, Sentinel achieves 76.60% accuracy, outperforming GLM-5.3 by 6.13 percentage points and its base model by 19.78 points. These results show that even state-of-the-art general-purpose LLMs struggle to identify untrustworthy comments, while iterative action-level learning substantially improves the accuracy of repository-grounded trustworthiness judgments. Our dataset is available at https://huggingface.co/datasets/dcloud347/CRJudgeBenchmark
Yue Pan, Jiawei Li, Ziyuan Zhang +2
University College London · Amazon · Zhejiang University
Early multi-agent LLM systems often used role-separated teams, yet scaling agent count yields diminishing returns on repository-level coding tasks. Recent alternatives treat agents as passive tools (subagents), yet this removes the benefits of agent interaction entirely. We study whether a subagent paradigm can support a middle ground: minimal agentic cooperation without the overhead of large multi-agent teams. We introduce Adversarial Review (AR), a minimal cooperative code-review protocol in which a main coding agent works with a reviewer and a critic agent. The reviewer evaluates code, while the critic audits the review through structured disagreement before the main agent edits. On LiveCodeBench, AR achieves the highest pass rate among tested methods, outperforming a five-agent baseline while using only three agents. On SWE-PRBench, naive AR exposes a false-consensus failure mode, where agents converge on agreement without sufficient evidence, but a single prompt iteration that adds disagreement explicitly achieves the highest F1 among tested methods. On SWE-bench Verified, AR also shows improvements over the baselines on repository-level coding tasks. Together, AR demonstrates that cooperative code review does not require many agents or complex communication structures: it requires that disagreement be minimal, structured, and evidence-grounded.