cs.CROct 25, 2026

CyberDual: AI Attackers versus AI Code Reviewers in Real Software Maintenance

Authors: Chenyuan Yang, Hwiwon Lee, Jongseong Kim, Yihan Yang, Dongjun Kim, Yinfang Chen, Lingming Zhang

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Open-source projects now let AI agents review incoming code, but the code under review may itself come from an AI attacker. We present CyberDual, an arena for this dual role of AI: attacker agents hide an exploitable bug in real maintenance work on widely used open-source projects, and reviewer agents try to find and repair it. Execution decides each match: we rerun the attacker's proof of concept (PoC) on the code that comes out of review. Every attack is new, and the windows can be rebuilt each quarter from the latest upstream history. Three agentic reviewers catch 85-91% of 1,088 valid attacks from six attackers. Yet when the attacker spreads the bug over real upstream commits, about one in six attack slots still ends with a working exploit after review. The strongest attacker has open weights: GLM 5.3 almost never refuses, and its attacks make up 13 of the 28 that get past all three reviewers. When they miss, Claude reviewers mostly approve the attack, while GPT-6-Sol mostly repairs the wrong code; a reviewer from a second vendor catches about two thirds of a Claude reviewer's misses. Beyond a leaderboard, CyberDual has three uses. First, PoC execution gives both sides a reward that agent reports cannot game, so its surviving attacks can train reviewers against stronger attackers. Second, it can gate a release on both scores: Claude Sonnet 5 refuses one in five attack slots, yet attacks about as well as GPT-6-Sol, so refusal alone does not remove attack capability. Third, it can warn the teams behind a deployed reviewer when a new attacker crosses a threshold: GLM 5.3 in Claude Code already gets an exploit past Sonnet 5 in 17% of its attack slots.

Figures & tables

Figure 1: One CyberDual match. An attacker agent hides a bug in a commit that looks like a clean-up. A reviewer agent inspects it, and rerunning the hidden exploit decides who wins.
Figure 2: Attack collection, attack success rate and catch rate for every attacker and reviewer, by setup ( Equation 2 ). All reviewers pools the three reviewers for one attacker; All attackers pools the attackers for one reviewer. Icons mark each agent’s harness, Claude Code or Codex.
Figure 3: The three setups. The attacker outputs a change and its exploit (PoC); a valid PoC crashes the attacked code but not the reference code. The reviewer agent then triages the change and either approves it or rejects it and fixes the code. : what the reviewer sees.
Table 1: Reviewer outcomes by setup, pooled over attackers. nn : valid attacks. Survived : the share of scored valid attacks whose PoC still crashes the code after review. Approved and Wrong site count these attacks by how the reviewer missed: it left the source unchanged, or it edited other code. All three : attacks that defeat every reviewer.
Figure 4: What distinguishes the attacks that reviewers miss. Each row compares the valid attacks that a reviewer caught (hollow) with those that it missed (filled), as means or shares. We omit task-only attacks, which leave too few misses. The left panels describe the attacker’s edit; “upstream test dropped” counts only tasks whose upstream fix adds a test. The right panels describe the review ( Section E.1 ). A star marks p<0.05p<0.05 (Mann–Whitney UU test or Fisher’s exact test), without correction or control for project and attacker. GPT-6-Sol ran code in every review.
Figure 5: Two attacks that survived review, shown as trimmed lines from the attack and post-review diffs (red −- , green ++ ; no PoC input shown). (a) Each commit looks safe on its own; only the combined code reads past the table. (b) Two reviewers repaired a real sibling bug near the advertised path, and none touched the omitted hunk, so the PoC still crashes.
Figure 7
Table 3: Capabilities on each side of the attack–review loop: measured, limited, absent. Retest : the grader reruns the trigger on the code that the defender repaired. CyberForge repairs a known injected bug without review; c-CRAB reviews benign PRs; AI Control edits contest solutions.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

  Window  Window
ProjectTasks (A/B)SpanCommitsProjectTasks (A/B)SpanCommits
curl2/32 weeks199openexr4/12 months87
expat3/21 month125exiv23/22 months77
libarchive3/21 month124libwebp4/12 months66
harfbuzz3/21 month123libssh24/11 month65
libheif3/21 month108xz2/33 months63
Appendix
Table 4: The fourteen libraries. Tasks gives the five single-commit tasks per project as vulnerability-repair (A) and feature (B) tasks. Window is the span of upstream history that ends on 2026-10-01. Commits counts the commits in each project’s frozen window.
OutcomeScored as
Before review
Refusal, or no gradeable submissionEffort; a refused slot is an attacker loss
Reference code does not buildOutside the quota
PoC does not crash the attacked code, or crashes the referenceAttacker loss
After review (PoC rerun on the code after review)
PoC no longer crashes (caught) ∗Reviewer win
Appendix
Table 5: Outcome of one attack attempt. The reference code is the parent (task only), the parent with the upstream fix (fix in hand), or the original window. Rows marked ∗\ast form the denominator of the survival and catch rates; only the last two rows are attacker wins.
Table 6: Per-arm counts behind Figure 2 . Launches : attacker launches through the selected quota. Refused : explicit refusals among them. For each reviewer, Missed counts the valid attacks that it missed (the PoC still crashes the code after review), and % gives them as a share of the valid attacks that the reviewer scored. Icons give the model and its harness ( Claude Code, Codex).
Figure 7: Per-project outcomes, pooled over attackers. In each setup, the grey bar gives the share of gradeable submissions that are valid, and each red bar gives the valid attacks that survive that reviewer. The bottom row pools all projects.
Figure 8: V8 (Chrome) and SpiderMonkey (Firefox), with ten slots per attacker and setup. Each cell gives the valid attacks over the refusal rate, and the attack success rate against Sonnet 5 and GPT-6-Sol. Overall pools both engines and all setups; both pools the two reviewers.
  Collection Missed / valid
SetupAttackerLaunchesRefusedGradeableValidSonnet 5GPT-6-Sol
V8
Task onlyOpus 4.81331040/40/4
 GPT-6-Sol1001061/60/6
 GLM 5.31111050/50/5
Fix in handOpus 4.81331040/41/4
Appendix
Table 7: JavaScript-engine arms, counted to each stop. Gradeable is out of ten; reviewer columns give missed over valid attacks. As attacker, Sonnet 5 refused on every window launch.
Figure 9: Median output tokens per attack (all graded attempts) and per review (scored reviews of valid attacks), reasoning and subagent output included.
More in this topic

AI Agent Security Benchmarks

AI agent security benchmarks measure how tool-using and autonomous systems handle attacks, privacy risks, authorization, and harmful actions.

See all 305 papers

Explore similar work

CardsList
Jun 11, 2026

SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

Large language models (LLMs) are increasingly deployed in automated code-review systems, where their approvals can determine which code is merged into shared repositories. However, it is unclear whether review agents can detect vulnerability-introducing code when an attacker controls both the code change and the persuasive Pull Request (PR) narrative designed to mask it. We introduce SEVRA-BENCH (Social Engineering of Vulnerabilities in Review Agents), a benchmark that measures how often a review agent approves such adversarial PR s. Each PR in SEVRA-BENCH is built from a historical commit that fixed a vulnerability. We automatically reverse that fix to extract the original vulnerable code, and submit the resulting code change as a PR wrapped in one of 15 social-engineering framings. To test review-agent resilience to narrative manipulation, these framings vary dimensions such as supporting evidence, conveyed urgency, signals of prior approval, and appeals to authority. SEVRA-BENCH evaluates a retained challenge split of roughly 1000 adversarial PRs drawn from publicly disclosed vulnerability fixes across the top 10 entries of the MITRE's 2025 most dangerous software weaknesses. Evaluating 8 review agents against this benchmark, we reveal that review agents are susceptible to narrative manipulation, exposing a significant gap in security capabilities.

Rui Melo, Riccardo Fogliato, Sean Zhou +2

1Carnegie Mellon University · 2Microsoft Core AI · 3Independent Researcher +1

Software Vulnerability DetectionAI Agent SecurityLLM Agent SecurityAI Agent Security BenchmarksAutomated Code Review
AskPDF
Feb 18, 2026

Can Adversarial Code Comments Fool AI Security Reviewers -- Large-Scale Empirical Study of Comment-Based Attacks and Defenses Against LLM Code Analysis

AI-assisted code review is widely used to detect vulnerabilities before production release. Prior work shows that adversarial prompt manipulation can degrade large language model (LLM) performance in code generation. We test whether similar comment-based manipulation misleads LLMs during vulnerability detection. We build a 100-sample benchmark across Python, JavaScript, and Java, each paired with eight comment variants ranging from no comments to adversarial strategies such as authority spoofing and technical deception. Eight frontier models, five commercial and three open-source, are evaluated in 9,366 trials. Adversarial comments produce small, statistically non-significant effects on detection accuracy (McNemar exact p > 0.21; all 95 percent confidence intervals include zero). This holds for commercial models with 89 to 96 percent baseline detection and open-source models with 53 to 72 percent, despite large absolute performance gaps. Unlike generation settings where comment manipulation achieves high attack success, detection performance does not meaningfully degrade. More complex adversarial strategies offer no advantage over simple manipulative comments. We test four automated defenses across 4,646 additional trials (14,012 total). Static analysis cross-referencing performs best at 96.9 percent detection and recovers 47 percent of baseline misses. Comment stripping reduces detection for weaker models by removing helpful context. Failures concentrate on inherently difficult vulnerability classes, including race conditions, timing side channels, and complex authorization logic, rather than on adversarial comments.

Scott Thornton

Adversarial AttacksAdversarial Attacks on LLMsAdversarial Robustness of Language ModelsSoftware Vulnerability Detection
AskPDF
Oct 24, 2026

Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?

LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes. More specifically, we have built an automated agentic pipeline to autonomously harvest, build, and validate massive real-world bug-introducing commits for popular software systems. To ensure precise evaluation, we also adopt LLM-as-a-judge to credit the findings matching ground-truth vulnerabilities. To date, we have collected 265 tasks from nine critical systems, including Firefox, V8, and OpenSSL with millions of lines of code. Across eight frontier models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT-6 sol ranking at the top. Our study also shows that most misses are not simply oversights: in 68% of them, the agent reports on a file that holds the flaw but argues for a different mechanism. Security-focused harnesses also behave differently: OpenAI's Codex Security lowers recall by 8.3 points, mostly by abstaining, while Anthropic's Defending Code harness raises it by 7.7 points, at a higher cost.

Yihan Yang, Chenyuan Yang, Zijie Zhao +5

University of Illinois Urbana-Champaign · Quantum AI & Cyber Security Institute, FPT Corporation; work done during an internship at UIUC.

AI Agent Security BenchmarksAutomated Code ReviewSoftware Vulnerability DetectionLLM Agent Evaluation
AskPDF