Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.
Figures & tables
Dataset
Year
Oracle source
Repo-level Support
Evaluation
Tufano-21 ( Tufano et al., 2021 )
2021
Mined OSS PR reviews
No
N-gram overlap
CodeReviewer ( Li et al., 2022 )
2022
Mined OSS PR reviews
No
N-gram overlap
Tufano-22 ( Tufano et al., 2022 )
2022
Mined OSS PR reviews
No
N-gram overlap
CRScore corpus ( Naik et al., 2025 )
2025
LLM + static analysis
No
Embedding similarity
ContextCRBench ( Hu et al., 2025 )
2025
Mined OSS PR reviews
No
N-gram overlap
SWE-CARE ( Guo et al., 2025 )
2025
Mined OSS PR reviews
Yes
Localization + N-gram overlap + LLM-as-a-judge
Table 1. Comparison of automated code review benchmarks.
Metric
Value
Interpretation
BLEU-4
0.00
The degree of n-gram overlap between human and automated tool reviews.
ROUGE-L
7.02
chrF
20.74
Embedding similarity
54.59
The relative distance between human and automated tool reviews in the embedding space.
Our Work
Test passed?
Yes
Whether the reviews successfully capture concerns raised by humans during review.
Table 2. Comparison of c-CRAB against prior similarity-based metrics.
Figure 4. Idea behind Benchmark Construction.
Pipeline Stage
# PR
# Comments
Initial Dataset
671
1,313
Review Filtering
410
595
Executable Environment Construction
410
595
Converting NL Comments to Tests
339
481
Validation with Coding Agents (Final)
184
234
Table 3. The number of remaining PR instances and Review Comments after each processing step.
Statistic
Value
Pull Requests
Repositories
67
Instances (PRs)
184
Avg. modified lines
418.1
Tests / Review Comments
Total review comments (tests)
234
Table 4. Statistics of the final c-CRAB benchmark.
# Comments
Pass Rate (%)
Reviewer
Total
Avg/PR
Behavioral
Structural
Overall
Automated Tool
Claude-Code
1336
7.3
38.1
30.7
32.1
Codex
324
1.8
38.1
16.1
20.1
Devin
1344
7.3
31.0
23.4
24.8
PR-Agent
524
2.8
38.1
19.8
23.1
Table 5. Performance comparison of automated review tools.
Figure 5. Overlap of passed test for each review tool.
Tool
# Comments
# Useful
Useful %
Claude Code
40
31
78%
Codex
8
7
88%
Devin
26
22
85%
PR-Agent
18
17
94%
Total
92
77
84%
Table 6. Manually labelled usefulness of review comments generated by four AI code review tools across six PRs.
Figure 6. Categories of review comments generated by four tools and comparison with human reviews.
Figure 7. Pass rate of automated review tools grouped by the category of the executable tests.
Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.
Early multi-agent LLM systems often used role-separated teams, yet scaling agent count yields diminishing returns on repository-level coding tasks. Recent alternatives treat agents as passive tools (subagents), yet this removes the benefits of agent interaction entirely. We study whether a subagent paradigm can support a middle ground: minimal agentic cooperation without the overhead of large multi-agent teams. We introduce Adversarial Review (AR), a minimal cooperative code-review protocol in which a main coding agent works with a reviewer and a critic agent. The reviewer evaluates code, while the critic audits the review through structured disagreement before the main agent edits. On LiveCodeBench, AR achieves the highest pass rate among tested methods, outperforming a five-agent baseline while using only three agents. On SWE-PRBench, naive AR exposes a false-consensus failure mode, where agents converge on agreement without sufficient evidence, but a single prompt iteration that adds disagreement explicitly achieves the highest F1 among tested methods. On SWE-bench Verified, AR also shows improvements over the baselines on repository-level coding tasks. Together, AR demonstrates that cooperative code review does not require many agents or complex communication structures: it requires that disagreement be minimal, structured, and evidence-grounded.
AI coding agents are generating code at volumes that exceed the capacity of traditional peer review. At the same time, existing AI code review tools over-index on low-value suggestions such as style and best practices while under-indexing on the concerns human reviewers prioritize most: correctness, security, and performance. We present ARCTIC, an AI-powered Code Critique system that reframes code review around three capabilities: intent prediction, which infers why a change was made from conversation logs and metadata; drift detection, which measures divergence between the developer's intent and the agent's output via backtranslation; and code spotlight, which ranks the regions of a diff most warranting human scrutiny. We ground these capabilities in a six-theme taxonomy derived from 18,000 code reviews. Offline evaluation shows that intent prediction achieves 0.86 F1, drift detection reaches near-perfect ordinal agreement with human annotators (QWK = 0.907), and spotlight outperforms the baseline AI reviewer by 2.4x on quality estimation at 5x fewer tokens. In the experimental rollout, the drift scores reduces code misalignment by an additional 5.76 points (p = 0.026), intent prediction receives 90.2% approval, and zero defects have been attributed to self-reviewed diffs since launch.