AVIATOR: Towards AI-Agentic Vulnerability Injection Workflow for High-Fidelity, Large-Scale Code Security Dataset

Amine Lbath; Massih-Reza Amini; Aurelien Delaitre; Vadim Okun

cs.CRAug 28, 2025

AVIATOR: Towards AI-Agentic Vulnerability Injection Workflow for High-Fidelity, Large-Scale Code Security Dataset

Authors: Amine Lbath, Massih-Reza Amini, Aurelien Delaitre, Vadim Okun

Organizations: National Institute of Standards and Technology, Software and Systems Division, Gaithersburg, MD, USA · Universit´e Grenoble Alpes, CNRS, LIG, Grenoble, France

Abstract

The increasing complexity of software systems and the sophistication of cyber-attacks have underscored the need for reliable automated software vulnerability detection. Data-driven approaches using deep learning models show promise but critically depend on the availability of large, accurately labeled datasets. Yet existing datasets either suffer from noisy labels, limited vulnerability coverage, or fail to reflect vulnerabilities as they occur in real-world software. This also limits large-scale benchmarking of such solutions. Automated vulnerability injection provides a way to address these limitations, but existing techniques remain limited in coverage, contextual fidelity, or injection success. In this paper, we present AVIATOR, the first AI-agentic vulnerability injection framework. AVIATOR decomposes vulnerability injection into a coordinated workflow of specialized AI agents, tool-based analysis, and iterative self-correction, explicitly mirroring expert reasoning. It integrates RAG and lightweight LoRA-based fine-tuning to produce realistic, category-specific vulnerabilities without relying on handcrafted patterns. Across three benchmarks, AVIATOR achieves high injection fidelity (91-95%) surpassing existing injection techniques in both accuracy and vulnerability coverage. When used for data augmentation to train deep learning-based vulnerability detection (DLVD) models, AVIATOR provides the strongest downstream gains in vulnerability detection. Across models and base datasets, AVIATOR improves average F1 scores by +22% over no augmentation, +25% over VGX, holding the prior best injection success rate, and +3% over VulScribeR, the prior state-of-the-art LLM-based injection model, with +7% higher recall and no precision loss. Its augmented data exhibits the lowest distributional distortion and scales efficiently with <2% syntax rejection at 4.3x lower cost than VulScribeR.

Figures & tables

Fig. 1: Architecture of the AVIATOR framework illustrating the two core modules. The top module represents the Vulnerability Injection component, which utilizes RAG and supervised fine-tuning with LoRA for guiding code transformations. The bottom module depicts the Validation component, combining LLM-based discriminators with static analysis to verify vulnerability instantiation and trigger bounded self-correction. The full framework contains 13 agents including 9 AI agents and 4 tool agents.

Dataset	Vulnerable Samples	Clean Samples	Total	Ratio	CWE Num	Entropy
SARD100 [ 49 ]	102	102	204	1:1	23	3.98
FormAI [ 23 ]	192,712	117,774	310,486	1:0.6	19	3.61
PrimeVul Train [ 11 ]	4,802	169,345	174,147	1:35.3	122	4.76
PrimeVul Validation	593	23,347	23,940	1:39.4	73	5.05
PrimeVul Test	549	24,232	24,781	1:44.1	71	5.09
Devign [ 25 ]	10,768	12,024	22,792	1:1.1	NA	4.84

TABLE I: Statistics of the Studied Datasets

Dataset	Eval. Type	Metric	AVIATOR
SARD100	Automated	$AISR_{5}$	95%
FormAI	Automated	$AISR_{5}$	91%
PrimeVul	Manual	% Analyzable Weaknesses	93%

TABLE II: AVIATOR injection success across datasets: Average Injection Success Rate on 5 runs for SARD100 and FormAI, and on analyzable cases for PrimeVul due to limited context.

Model (*) / Dataset	Accuracy	Source
Graph2Edit * [ 57 ]	13%	[ 18 ]
BigVul [ 35 ]	25%	[ 11 ]
AVIATOR $W_{1}$ *	31%	FormAI validation
CrossVul [ 37 ]	48%	[ 11 ]
Getafix * [ 57 ]	50%	[ 18 ]
CVEFixes [ 36 ]	52%	[ 11 ]

TABLE III: Label accuracy of major vulnerability datasets and injection methods satisfying the BCD properties (Section II-B ). Unmarked rows report audited label accuracy, whereas injection methods (*) report injection success, which by construction is the label accuracy of a dataset built by injection. Results for AVIATOR are obtained via automated evaluation on SARD100 and FormAI, and manual validation on PrimeVul. The reported score for AVIATOR corresponds to the mean injection success rate across all three evaluation datasets.

Dataset	Model Variant	$AISR_{5}$	Std
FormAI	Base model	85%	4
	+ GRPO	84%	1
	+ SFT	91%	1
SARD100	Base model	94%	3
	+ GRPO	94%	0
	+ SFT	95%	2

TABLE IV: AVIATOR

W_{13}

injection success rate (

AISR_{5}

) across five vulnerability injection runs with and without fine-tuning, on FormAI and SARD100.

Dataset	Comparison	$p$ -value
FormAI ( $N=38$ )	SFT vs. no FT	0.049
	SFT vs. GRPO	0.041
	GRPO vs. no FT	0.69
SARD100 ( $N=34$ )	SFT vs. no FT	0.63
	SFT vs. GRPO	1.00
	GRPO vs. no FT	0.46

TABLE V: Paired Wilcoxon signed-rank test on per-sample success rates across 5 injection runs. Significant results (

p<0.05

) are shown in bold.

Dataset	Model	@1	@2	@3	@4	@5	@6	@7	@8	@9	@10
FormAI	$W_{13}$	84.32%	91.05%	93.87%	95.24%	96.03%	96.56%	96.94%	97.18%	97.30%	97.30%
FormAI	$W_{13}$ + SFT	89.91%	93.04%	94.81%	95.62%	96.15%	96.55%	96.87%	97.14%	97.38%	97.58%
SARD100	$W_{13}$	93.58%	95.95%	96.66%	96.91%	97.01%	97.05%	97.06%	97.06%	97.06%	97.06%
SARD100	$W_{13}$ + SFT	95.15%	98.65%	99.51%	99.79%	99.93%	99.99%	100.00%	100.00%	100.00%	100.00%

TABLE VI: AVIATOR

W_{13}

pass@k for 10 vulnerability injection generations with and without fine-tuning, on FormAI and SARD100.

Panel of Figure 8 — Fig. 2: Training dynamics of the vulnerability injection agent. Left: SFT training loss over 5 epochs, with lr= $2\times 10^{-4}$ , batch size = 4, LoRA $r=32$ , and $\alpha=32$ . Right: GRPO CodeBLEU reward over 2 iterations, with lr= $2\times 10^{-4}$ , batch size = 8, 4 generated samples per policy iteration, LoRA $r=32$ , and $\alpha=32$ .

Panel of Figure 9 — Fig. 3: AVIATOR average injection success rate on 5 attempts. Empirical study for LLM agents using Llama-4-Maverick, Qwen2.5-Coder with and without the vulnerability injection agent fine-tuned using SFT and GRPO.

Fig. 4: Injection success rate by weakness group [ 58 ] within the evaluation datasets. Bubble size, and the value annotated inside it, indicate the number of unique code samples in the group. The annotation 5× denotes samples from SARD100 and FormAI evaluated across 5 repeated runs; their reported success rate is the average over the 5 runs. The color gradient reflects the total volume of injection attempts, accounting for all repetitions.

Strategy	Devign Model			Reveal Model			LineVul Model			CodeT5 Model
Strategy	P	R	$F_{1}$	P	R	$F_{1}$	P	R	$F_{1}$	P	R	$F_{1}$
Seed Dataset: Devign ; Test Dataset: BigVul Test
No Aug	6.27	43.24	10.95	7.28	77.27	13.31	7.90	29.19	12.43	6.08	65.71	11.13
ROS	7.54	25.60	11.65	7.96	33.39	12.86	10.91	13.16	11.93	7.52	51.81	13.13
VulGen	6.49	62.32	11.75	6.36	70.59	11.66	8.51	22.15	12.29	7.34	31.05	11.87
VGX	6.06	59.30	10.99	6.89	73.29	12.60	5.35	98.82	10.15	7.20	80.17	13.21

TABLE VII: Downstream vulnerability detection performance (Precision, Recall,

F_{1}

). Each model is trained on the seed dataset augmented with 5k injected vulnerable samples using each strategy. Best

F_{1}

scores per model and setting are highlighted in bold . Results for non-AVIATOR strategies are taken from the VulScribeR paper [ 19 ] ; we report new results for AVIATOR and CodeT5.

Compared Strategy	Precision	Recall	$F_{1}$ -Score
No Aug	+12.2%	+104.9%	+21.8%
ROS	+9.0%	+26.0%	+12.5%
VulGen	+30.3%	+5.7%	+25.2%
VGX	+39.2%	-25.4%	+24.6%
VulScribeR	+1.4%	+7.4%	+2.9%

TABLE VIII: Overall Average of Per-Configuration Relative Improvement of AVIATOR Compared to Other Strategies.

Panel of Figure 13 — Fig. 5: Distributional analysis of augmented datasets. KL divergence measures the deviation from the original, natural distribution of vulnerabilities (lower is better). AVIATOR introduces the least distributional distortion.

Fig. 7: Average injection success rate against the memory required to serve the model, for the two ways of meeting a memory budget. (a) Qwen2.5-Coder family at bfloat16, under the single-agent baseline ( $W_{1}$ , dashed) and the full workflow ( $W_{13}$ , solid). (b) The 32B model at five numerical precisions under $W_{13}$ , obtained with IQ1_S, IQ2_M, AWQ, FP8 and bfloat16 respectively. Model sizes and bit widths are given below the memory values. Bands show the standard deviation over five runs.

Fig. 8: Injection success rate by CWE ID within the evaluation datasets. Bubble size indicates the number of unique code samples available for each CWE. The annotation 5× denotes samples from SARD100 and FormAI evaluated across 5 repeated runs; their reported success rate is the average over the 5 runs. The color gradient reflects the total volume of injection attempts, accounting for all repetitions. CWEs included in the MITRE Top 25 Most Dangerous Software Weaknesses are highlighted with a bold border.

CWE	Name	CWE	Name
16	Configuration	362	Race Condition
19	Data Handling Issues	367	Time-of-Check Time-of-Use (TOCTOU) Race Condition
20	Improper Input Validation	369	Division by Zero
22	Path Traversal	388	Insufficiently Protected Credentials
59	Link Following	391	Unchecked Error Condition
61	UNIX Symbolic Link (Symlink) Following	399	Resource Management Errors

TABLE IX: The 133 CWE types supported by the current AVIATOR instantiation.

Weakness Group	#	Supported CWEs	Weakness Group	#	Supported CWEs
Access control	7	264, 284, 285, 552, 639, 668, 732	Expired memory	2	415, 416
Ante buffer operation	8	119, 125, 129, 466, 786, 787, 805, 823	Initialization	5	457, 665, 824, 908, 909
Authentication	2	346, 613	Input validation	12	20, 74, 77, 78, 88, 89, 94, 99, 116, 134, 611, 707
Calculation	5	131, 193, 369, 468, 682	Invalid pointer	6	466, 476, 690, 763, 823, 824
Cleanup	1	404	Loop and recursion	2	674, 835
Code quality	1	489	Memory leak	1	401

TABLE X: SATE V weakness groups [ 58 ] and the supported CWE types of Table IX belonging to each group.

Software Security

Software security studies how to find, prevent, validate, and repair vulnerabilities in code, systems, dependencies, and software supply chains.

See all 96 papers

Toward Scalable Automated Repository-Level Datasets for Software Vulnerability Detection
Mar 18, 2026Amine LbathAutomated Exploit Generation Software Vulnerability Detection
Univeristy of Grenoble Grenoble, France
Code-Augur: Agentic Vulnerability Detection via Specification Inference
Jun 17, 2026Zhengxiong Luo, Mehtab Zafar, Dylan Wolff +1Automated Software Testing Software Vulnerability Detection
National University of Singapore
CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training
Aug 6, 2026Amine Lbath, Manan Suri, Aurelien Delaitre +4Automated Program Repair Software Vulnerability Detection
National Institute of Standards and Technology · Université Grenoble Alpes, CNRS · University of Maryland, College Park

Mar 18, 2026J/K move · Enter open · S save

Toward Scalable Automated Repository-Level Datasets for Software Vulnerability Detection

Amine Lbath

Univeristy of Grenoble Grenoble, France

Automated Exploit Generation Software Vulnerability Detection Synthetic Benchmark Generation

Open paper AskPDF arXiv

Software vulnerabilities continue to grow in volume and remain difficult to detect in practice. Although learning-based vulnerability detection has progressed, existing benchmarks are largely function-centric and fail to capture realistic, executable, interprocedural settings. Recent repo-level security benchmarks demonstrate the importance of realistic environments, but their manual curation limits scale. This doctoral research proposes an automated benchmark generator that injects realistic vulnerabilities into real-world repositories and synthesizes reproducible proof-of-vulnerability (PoV) exploits, enabling precisely labeled datasets for training and evaluating repo-level vulnerability detection agents. We further investigate an adversarial co-evolution loop between injection and detection agents to improve robustness under realistic constraints.

Abstract

Software Security

Explore similar work

Toward Scalable Automated Repository-Level Datasets for Software Vulnerability Detection

Code-Augur: Agentic Vulnerability Detection via Specification Inference

CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training