VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detection

Renwei Meng; Haoyi Wu; Jingming Wang

cs.SEMar 11, 2026

VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detection

Authors: Renwei Meng, Haoyi Wu, Jingming Wang

Abstract

Software vulnerabilities often depend on cross-file data flow, build options, framework conventions, and runtime guards, so isolated function classifiers produce fragile and poorly calibrated warnings. Repository-level LLM agents can gather richer evidence, but prior variants under-specify reproducibility, verifier behavior, baseline fairness, and statistical uncertainty. We present VulnAgent-R2, a budget-aware agentic auditing framework with three additional reusable modules: counterfactual evidence reweighting, build-aware verification-plan synthesis, and a cost-risk Pareto scheduler. The system combines graph triage, bounded context optimization, role-specialized agents, sceptic counter-evidence, selective dynamic verification, and calibrated fusion. On Devign, Big-Vul, DiverseVul, and PrimeVul, VulnAgent-R2 obtains 0.798/0.895, 0.739/0.871, 0.700/0.842, and 0.385/0.781 F1/AUROC, respectively. On JITVul it reaches 0.606 F1, 0.529 Top-1, and 0.742 Top-3 localization, while reducing online tokens by 38.3% over always-full multi-agent execution. Online time includes retrieval, LLM calls, CER scoring, verifier planning, compilation, and test execution, but excludes one-time shared indexing. Bootstrap tests show the PrimeVul gain over VulnAgent-X is +0.038 F1, 95% CI [0.020, 0.055], Holm-adjusted $p=0.009$ . Treating vulnerability detection as calibrated evidence accumulation improves detection, localization, auditability, and cost control under the evaluated protocol, while remaining a prioritization aid rather than a replacement for manual review.Code is available at https://github.com/renweimeng/Vlun-Agent-X.

Figures & tables

Figure 1: VulnAgent-R2 workflow: graph triage, context optimization, role-specialized agents, counterfactual evidence reweighting, selective verification, memory, and calibrated fusion.

Agent	Input	Action	Output schema	Stop rule
Router	target, static alerts	choose CWE family and tools	CWE prior, required context	one pass
Data-flow	slice, callers/callees	source-sink trace, sanitizer check	support/refute tuple	no new edge
Control/state	CFG, guards, locks	path feasibility, state invariant	path tuple, infeasible reason	2 rounds
Memory-safety	allocation/lifetime API	bounds, UAF, overflow evidence	CWE tuple, span	2 rounds
Sceptic	all positive claims	search counter-evidence and safe guards	negative tuple	confidence shift $<.03$
Verifier	evidence tuples, build files	synthesize and run plan	crash/assert/sanitizer/inconclusive	90s/sample

Table 1: Agent protocol and reproducibility settings.

Dataset	Unit	Repos	Total	Positive	Pos. rate	Split / leakage control
Devign	function	2	21,854	2,731	12.5%	project-time 80/10/10; sample-cluster CI only
Big-Vul	function	348	188,636	10,914	5.8%	CVE-time split; vulnerable/fixed pair grouping
DiverseVul	function	7,514	330,492	18,745	5.7%	project-disjoint test; MinHash dedup
PrimeVul	function	1,968	228,674	6,847	3.0%	official chronological split; no CVE overlap
ReposVul-style	repo target	412	7,840	1,126	14.4%	repo-disjoint; build metadata retained
JITVul	commit	506	20,274	2,413	11.9%	chronological commits; parent-child grouping

Table 2: Dataset statistics and split protocol.

Method group	Static alerts	Repo graph / history	Max ctx	Max rounds	Tool calls	Verifier
Direct LLM	read	summary only	8k total	1	0	no
RAG-only	read	same index retrieve	8k total	1	0	no
Verifier-RAG	read	same index retrieve	8k total	1	1.2 avg	yes
ReAct Agent	read	tool-read same graph	8k/round	4	4.7 avg	allowed
ConfColl	read	tool-read same graph	8k/round	4	5.5 avg	allowed
VulnAgent-X	read	optimize same graph	8k/round	3	5.8 avg	selective

Table 3: Baseline access parity and reproducibility knobs. Token budget counts input plus output.

Method	Devign	Big-Vul	DiverseVul	PrimeVul
CodeQL+Semgrep	.514 [.497,.530] / .732	.467 [.451,.483] / .711	.418 [.401,.434] / .688	.119 [.101,.137] / .566
Devign-GNN	.632 [.615,.649] / .801	.574 [.552,.593] / .758	.525 [.506,.544] / .742	.166 [.145,.187] / .599
LineVul	.664 [.647,.681] / .814	.595 [.574,.615] / .776	.551 [.530,.570] / .762	.194 [.171,.216] / .623
GraphCodeBERT	.681 [.664,.696] / .828	.616 [.594,.636] / .800	.573 [.551,.591] / .776	.206 [.183,.229] / .638
Qwen Direct	.707 [.690,.722] / .845	.648 [.629,.668] / .811	.603 [.583,.622] / .790	.231 [.208,.254] / .663
RAG-only	.725 [.708,.741] / .851	.673 [.653,.692] / .818	.631 [.612,.650] / .804	.271 [.247,.294] / .701

Table 4: Function-level vulnerability detection. F1 with 95% CI / AUROC.

Method	ReposVul F1	JITVul F1	Top-1	Top-3	MRR	Time(s)
CodeQL+Semgrep	.352	.336	.174	.316	.257	.39
GraphCodeBERT-diff	.416	.405	.228	.410	.324	.72
Qwen Direct	.497	.480	.345	.546	.435	2.17
RAG-only	.529	.514	.393	.600	.488	2.86
Verifier-RAG	.542	.525	.407	.618	.501	3.28
ReAct Agent	.551	.537	.419	.642	.516	3.67

Table 5: Repository-level and JIT vulnerability detection.

Method	Big-Vul CP F1	PrimeVul future F1	SecVulEval Top-1	VADER explain F1	ECE $\downarrow$
GraphCodeBERT	.547	.173	.119	.211	.181
Qwen Direct	.585	.205	.153	.286	.164
RAG-only	.609	.242	.186	.311	.149
ConfColl	.644	.294	.234	.346	.124
VulnAgent-X	.668	.317	.253	.371	.090
VulnAgent-R2	.697	.350	.285	.402	.064

Table 6: Robustness, calibration, and contamination diagnostics.

Backbone	Method	PrimeVul F1	PrimeVul ECE $\downarrow$	JITVul F1	Tokens/sample
Code Llama-34B	RAG-only	.236	.171	.466	7,880
Code Llama-34B	VulnAgent-R2	.338	.083	.548	9,960
DeepSeek-Coder-33B	RAG-only	.262	.154	.498	7,690
DeepSeek-Coder-33B	VulnAgent-R2	.371	.069	.585	9,720
Qwen2.5-Coder-32B	RAG-only	.271	.149	.514	7,840
Qwen2.5-Coder-32B	VulnAgent-R2	.385	.061	.606	9,410

Table 7: Cross-backbone check on PrimeVul and JITVul. The same prompts, tools, retrieval limits, and scheduler are used within each backbone.

Comparison	Metric/dataset	Diff.	95% CI	$p_{Holm}$
R2 - VulnAgent-X	Devign F1	+.020	[.003,.036]	.071 ∗
R2 - VulnAgent-X	PrimeVul F1	+.038	[.020,.055]	.009
R2 - VulnAgent-X	JITVul F1	+.025	[.006,.044]	.032
R2 - ConfColl	PrimeVul F1	+.056	[.033,.079]	.004
R2 - Verifier-RAG	PrimeVul ECE	-.049	[-.071,-.030]	.006
R2 - RAG-only	Top-3 loc.	+.142	[.109,.174]	.002

Table 8: Significance tests for VulnAgent-R2 improvements.

CWE group	Trigger	Buildable	Runnable	Positive evid.	Inconclusive
CWE-787 OOB write	.31	.74	.63	.41	.22
CWE-125 OOB read	.24	.69	.58	.34	.27
CWE-20 input validation	.18	.56	.44	.21	.39
CWE-190 integer overflow	.28	.78	.66	.47	.19
CWE-22 path traversal	.16	.61	.52	.25	.31
CWE-416 use-after-free	.33	.71	.59	.38	.26

Table 9: Verifier outcomes by weakness group. Rates are conditional on samples in each group.

Variant	P PV	R PV	F1 PV	ReposVul F1	JITVul F1	ECE $\downarrow$	Tokens
Full VulnAgent-R2	.402	.370	.385	.624	.606	.061	9,410
w/o graph context optimizer	.358	.318	.337	.572	.553	.091	7,810
w/o CER	.368	.351	.359	.597	.579	.098	8,960
w/o Sceptic Agent	.333	.374	.352	.584	.571	.112	8,780
w/o VPS verifier	.363	.330	.346	.591	.562	.121	8,210
w/o calibration	.391	.350	.369	.614	.589	.148	9,160

Table 10: Cross-dataset ablation and cost. PrimeVul columns show precision/recall/F1; other task columns show F1.

Method	Top-10 precision	Alerts/TP $\downarrow$	FP/repo $\downarrow$	Review saved
CodeQL+Semgrep	.220	6.8	9.4	–
RAG-only	.337	4.9	5.7	18.6%
Verifier-RAG	.349	4.6	5.4	21.4%
ConfColl	.388	4.1	4.9	30.2%
VulnAgent-X	.421	3.6	3.9	38.1%
VulnAgent-R2	.468	2.9	3.1	49.3%

Table 11: Audit workload on repository/JIT tasks. Lower alerts per true positive and false positives per repository indicate less manual triage burden.

Figure 3: Performance-cost Pareto frontier. The default scheduler is selected for cost-quality balance, not maximum F1.

Figure 4: End-to-end audit case trace with target, triage reason, retrieved context, agent evidence, verifier result, and final report.

Software Vulnerability Detection

Software vulnerability detection identifies insecure code, pinpoints weaknesses, and helps developers validate or prevent exploitable bugs.

See all 139 papers

AEGIS: From Clues to Verdicts -- Graph-Guided Deep Vulnerability Reasoning via Dialectics and Meta-Auditing
Mar 21, 2026Sen Fang, Weiyuan Ding, Zhezhen Cao +2Software Vulnerability Detection LLMs for Cybersecurity
NC State University · University of Alberta
Code-Augur: Agentic Vulnerability Detection via Specification Inference
Jun 17, 2026Zhengxiong Luo, Mehtab Zafar, Dylan Wolff +1Automated Software Testing Software Vulnerability Detection
National University of Singapore
Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detection
Jul 29, 2026Yikun Li, Ting Zhang, Jiakun Liu +9Agentic RL Software Vulnerability Detection
1Singapore Management University, Singapore · 2Harbin Institute of Technology, China · 3GovTech, Singapore

Mar 21, 2026J/K move · Enter open · S save

AEGIS: From Clues to Verdicts -- Graph-Guided Deep Vulnerability Reasoning via Dialectics and Meta-Auditing

Sen Fang, Weiyuan Ding, Zhezhen Cao, Zhou Yang+1

NC State University · University of Alberta

Software Vulnerability Detection LLMs for Cybersecurity

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly adopted for vulnerability detection, yet their reasoning remains fundamentally unsound. We identify a root cause shared by both major mitigation paradigms (agent-based debate and retrieval augmentation): reasoning in an ungrounded deliberative space that lacks a bounded, hypothesis-specific evidence base. Without such grounding, agents fabricate cross-function dependencies, and retrieval heuristics supply generic knowledge decoupled from the repository's data-flow topology. Consequently, the resulting conclusions are driven by rhetorical persuasiveness rather than verifiable facts. To ground this deliberation, we present AEGIS, a novel multi-agent framework that shifts detection from ungrounded speculation to forensic verification over a closed factual substrate. Guided by a "From Clue to Verdict" philosophy, AEGIS first identifies suspicious code anomalies (clues), then dynamically reconstructs per-variable dependency chains for each clue via on-demand slicing over a repository-level Code Property Graph. Within this closed evidence boundary, a Verifier Agent constructs competing dialectical arguments for and against exploitability, while an independent Audit Agent scrutinizes every claim against the trace, exercising veto power to prevent hallucinated verdicts. Evaluation on the rigorous PrimeVul dataset demonstrates that AEGIS establishes a new state-of-the-art, achieving 122 Pair-wise Correct Predictions. To our knowledge, this is the first approach to surpass 100 on this benchmark. It reduces the false positive rate by up to 54.40% compared to leading baselines, at an average cost of $0.09 per sample without any task-specific training.

Abstract

Software Vulnerability Detection

Explore similar work

AEGIS: From Clues to Verdicts -- Graph-Guided Deep Vulnerability Reasoning via Dialectics and Meta-Auditing

Code-Augur: Agentic Vulnerability Detection via Specification Inference

Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detection