Organizations: National Institute of Standards and Technology, Software and Systems Division, Gaithersburg, MD, USA · Universit´e Grenoble Alpes, CNRS, LIG, Grenoble, France
The increasing complexity of software systems and the sophistication of cyber-attacks have underscored the need for reliable automated software vulnerability detection. Data-driven approaches using deep learning models show promise but critically depend on the availability of large, accurately labeled datasets. Yet existing datasets either suffer from noisy labels, limited vulnerability coverage, or fail to reflect vulnerabilities as they occur in real-world software. This also limits large-scale benchmarking of such solutions. Automated vulnerability injection provides a way to address these limitations, but existing techniques remain limited in coverage, contextual fidelity, or injection success. In this paper, we present AVIATOR, the first AI-agentic vulnerability injection framework. AVIATOR decomposes vulnerability injection into a coordinated workflow of specialized AI agents, tool-based analysis, and iterative self-correction, explicitly mirroring expert reasoning. It integrates RAG and lightweight LoRA-based fine-tuning to produce realistic, category-specific vulnerabilities without relying on handcrafted patterns. Across three benchmarks, AVIATOR achieves high injection fidelity (91-95%) surpassing existing injection techniques in both accuracy and vulnerability coverage. When used for data augmentation to train deep learning-based vulnerability detection (DLVD) models, AVIATOR provides the strongest downstream gains in vulnerability detection. Across models and base datasets, AVIATOR improves average F1 scores by +22% over no augmentation, +25% over VGX, holding the prior best injection success rate, and +3% over VulScribeR, the prior state-of-the-art LLM-based injection model, with +7% higher recall and no precision loss. Its augmented data exhibits the lowest distributional distortion and scales efficiently with <2% syntax rejection at 4.3x lower cost than VulScribeR.
Figures & tables
Fig. 1: Architecture of the AVIATOR framework illustrating the two core modules. The top module represents the Vulnerability Injection component, which utilizes RAG and supervised fine-tuning with LoRA for guiding code transformations. The bottom module depicts the Validation component, combining LLM-based discriminators with static analysis to verify vulnerability instantiation and trigger bounded self-correction. The full framework contains 13 agents including 9 AI agents and 4 tool agents.
Dataset
Vulnerable Samples
Clean Samples
Total
Ratio
CWE Num
Entropy
SARD100 [ 49 ]
102
102
204
1:1
23
3.98
FormAI [ 23 ]
192,712
117,774
310,486
1:0.6
19
3.61
PrimeVul Train [ 11 ]
4,802
169,345
174,147
1:35.3
122
4.76
PrimeVul Validation
593
23,347
23,940
1:39.4
73
5.05
PrimeVul Test
549
24,232
24,781
1:44.1
71
5.09
Devign [ 25 ]
10,768
12,024
22,792
1:1.1
NA
4.84
TABLE I: Statistics of the Studied Datasets
Dataset
Eval. Type
Metric
AVIATOR
SARD100
Automated
AISR5
95%
FormAI
Automated
AISR5
91%
PrimeVul
Manual
% Analyzable Weaknesses
93%
TABLE II: AVIATOR injection success across datasets: Average Injection Success Rate on 5 runs for SARD100 and FormAI, and on analyzable cases for PrimeVul due to limited context.
Model (*) / Dataset
Accuracy
Source
Graph2Edit * [ 57 ]
13%
[ 18 ]
BigVul [ 35 ]
25%
[ 11 ]
AVIATOR W1 *
31%
FormAI validation
CrossVul [ 37 ]
48%
[ 11 ]
Getafix * [ 57 ]
50%
[ 18 ]
CVEFixes [ 36 ]
52%
[ 11 ]
TABLE III: Label accuracy of major vulnerability datasets and injection methods satisfying the BCD properties (Section II-B ). Unmarked rows report audited label accuracy, whereas injection methods (*) report injection success, which by construction is the label accuracy of a dataset built by injection. Results for AVIATOR are obtained via automated evaluation on SARD100 and FormAI, and manual validation on PrimeVul. The reported score for AVIATOR corresponds to the mean injection success rate across all three evaluation datasets.
Dataset
Model Variant
AISR5
Std
FormAI
Base model
85%
4
+ GRPO
84%
1
+ SFT
91%
1
SARD100
Base model
94%
3
+ GRPO
94%
0
+ SFT
95%
2
TABLE IV: AVIATOR W13 injection success rate ( AISR5 ) across five vulnerability injection runs with and without fine-tuning, on FormAI and SARD100.
Dataset
Comparison
p -value
FormAI ( N=38 )
SFT vs. no FT
0.049
SFT vs. GRPO
0.041
GRPO vs. no FT
0.69
SARD100 ( N=34 )
SFT vs. no FT
0.63
SFT vs. GRPO
1.00
GRPO vs. no FT
0.46
TABLE V: Paired Wilcoxon signed-rank test on per-sample success rates across 5 injection runs. Significant results ( p<0.05 ) are shown in bold.
Dataset
Model
@1
@2
@3
@4
@5
@6
@7
@8
@9
@10
FormAI
W13
84.32%
91.05%
93.87%
95.24%
96.03%
96.56%
96.94%
97.18%
97.30%
97.30%
W13 + SFT
89.91%
93.04%
94.81%
95.62%
96.15%
96.55%
96.87%
97.14%
97.38%
97.58%
SARD100
W13
93.58%
95.95%
96.66%
96.91%
97.01%
97.05%
97.06%
97.06%
97.06%
97.06%
W13 + SFT
95.15%
98.65%
99.51%
99.79%
99.93%
99.99%
100.00%
100.00%
100.00%
100.00%
TABLE VI: AVIATOR W13 pass@k for 10 vulnerability injection generations with and without fine-tuning, on FormAI and SARD100.
Fig. 2: Training dynamics of the vulnerability injection agent. Left: SFT training loss over 5 epochs, with lr= 2×10−4 , batch size = 4, LoRA r=32 , and α=32 . Right: GRPO CodeBLEU reward over 2 iterations, with lr= 2×10−4 , batch size = 8, 4 generated samples per policy iteration, LoRA r=32 , and α=32 .
Fig. 3: AVIATOR average injection success rate on 5 attempts. Empirical study for LLM agents using Llama-4-Maverick, Qwen2.5-Coder with and without the vulnerability injection agent fine-tuned using SFT and GRPO.
Fig. 4: Injection success rate by weakness group [ 58 ] within the evaluation datasets. Bubble size, and the value annotated inside it, indicate the number of unique code samples in the group. The annotation 5× denotes samples from SARD100 and FormAI evaluated across 5 repeated runs; their reported success rate is the average over the 5 runs. The color gradient reflects the total volume of injection attempts, accounting for all repetitions.
Strategy
Devign Model
Reveal Model
LineVul Model
CodeT5 Model
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Seed Dataset: Devign ; Test Dataset: BigVul Test
No Aug
6.27
43.24
10.95
7.28
77.27
13.31
7.90
29.19
12.43
6.08
65.71
11.13
ROS
7.54
25.60
11.65
7.96
33.39
12.86
10.91
13.16
11.93
7.52
51.81
13.13
VulGen
6.49
62.32
11.75
6.36
70.59
11.66
8.51
22.15
12.29
7.34
31.05
11.87
VGX
6.06
59.30
10.99
6.89
73.29
12.60
5.35
98.82
10.15
7.20
80.17
13.21
TABLE VII: Downstream vulnerability detection performance (Precision, Recall, F1 ). Each model is trained on the seed dataset augmented with 5k injected vulnerable samples using each strategy. Best F1 scores per model and setting are highlighted in bold . Results for non-AVIATOR strategies are taken from the VulScribeR paper [ 19 ] ; we report new results for AVIATOR and CodeT5.
Compared Strategy
Precision
Recall
F1 -Score
No Aug
+12.2%
+104.9%
+21.8%
ROS
+9.0%
+26.0%
+12.5%
VulGen
+30.3%
+5.7%
+25.2%
VGX
+39.2%
-25.4%
+24.6%
VulScribeR
+1.4%
+7.4%
+2.9%
TABLE VIII: Overall Average of Per-Configuration Relative Improvement of AVIATOR Compared to Other Strategies.
Fig. 5: Distributional analysis of augmented datasets. KL divergence measures the deviation from the original, natural distribution of vulnerabilities (lower is better). AVIATOR introduces the least distributional distortion.
Fig. 7: Average injection success rate against the memory required to serve the model, for the two ways of meeting a memory budget. (a) Qwen2.5-Coder family at bfloat16, under the single-agent baseline ( W1 , dashed) and the full workflow ( W13 , solid). (b) The 32B model at five numerical precisions under W13 , obtained with IQ1_S, IQ2_M, AWQ, FP8 and bfloat16 respectively. Model sizes and bit widths are given below the memory values. Bands show the standard deviation over five runs.
Fig. 8: Injection success rate by CWE ID within the evaluation datasets. Bubble size indicates the number of unique code samples available for each CWE. The annotation 5× denotes samples from SARD100 and FormAI evaluated across 5 repeated runs; their reported success rate is the average over the 5 runs. The color gradient reflects the total volume of injection attempts, accounting for all repetitions. CWEs included in the MITRE Top 25 Most Dangerous Software Weaknesses are highlighted with a bold border.
CWE
Name
CWE
Name
16
Configuration
362
Race Condition
19
Data Handling Issues
367
Time-of-Check Time-of-Use (TOCTOU) Race Condition
20
Improper Input Validation
369
Division by Zero
22
Path Traversal
388
Insufficiently Protected Credentials
59
Link Following
391
Unchecked Error Condition
61
UNIX Symbolic Link (Symlink) Following
399
Resource Management Errors
TABLE IX: The 133 CWE types supported by the current AVIATOR instantiation.
Software vulnerabilities continue to grow in volume and remain difficult to detect in practice. Although learning-based vulnerability detection has progressed, existing benchmarks are largely function-centric and fail to capture realistic, executable, interprocedural settings. Recent repo-level security benchmarks demonstrate the importance of realistic environments, but their manual curation limits scale. This doctoral research proposes an automated benchmark generator that injects realistic vulnerabilities into real-world repositories and synthesizes reproducible proof-of-vulnerability (PoV) exploits, enabling precisely labeled datasets for training and evaluating repo-level vulnerability detection agents. We further investigate an adversarial co-evolution loop between injection and detection agents to improve robustness under realistic constraints.
The advent of agentic vulnerability detection is already becoming a watershed moment for software security. Audits conducted entirely by autonomous LLM agents are uncovering critical vulnerabilities in fundamental software underpinning digital society. Many of these vulnerabilities remained masked for years, surfacing only now with AI agents. Yet the reasoning behind these discoveries remains alarmingly opaque and unvalidated. What assumptions did the agent make about a function's inputs when it deemed that function to be secure? Failures in reasoning and incorrect assumptions can lead to missed vulnerabilities and reduce trust in agentic analysis. We propose a security-specification-first paradigm that (1) exposes the agent's tacit assumptions explicitly as security specifications and (2) continuously refines those specifications via runtime falsification. We realize our approach in Code-Augur, a novel harness for agentic vulnerability detection. Given a codebase, Code-Augur analyzes each component of the system for vulnerable code. When it deems a component to be secure, it commits the local invariants behind that judgment as in-source assertions. In parallel, Code-Augur leverages a guided fuzzer to attempt to falsify those assumptions. When the fuzzer triggers an assertion, this either reveals a genuine vulnerability or a flawed specification to refine. In both cases, this process grounds the agent's understanding, aligning its view of code intent with how the code actually behaves. On real-world subjects, Code-Augur effectively leverages security specifications to detect more vulnerabilities than other state-of-the-art agents. Additionally, Code-Augur found 22 new vulnerabilities in key open-source projects. Compared to curated specialized models like Claude Mythos, Code-Augur offers effective agentic vulnerability detection built on widely available LLMs like Sonnet and DeepSeek.
Despite recent advances, frontier large language model (LLM) agents remain limited in discovering and patching complex vulnerabilities in real-world software. Generally available agents can already aid attackers, who only need to find one exploitable weakness, while defenders must continuously identify and patch all vulnerabilities across fast-growing codebases. Stronger defensive agents would help close this gap, yet the scarcity of security training data with reproducible build and execution environments remains a bottleneck. We present CyberForge, a framework that synthesizes executable, repository-level security training data by injecting vulnerabilities into real C/C++ projects. It validates each instance dynamically: the injected build must pass the project's unit tests, and generated proof-of-vulnerability (PoV) must trigger on the injected build and not on the clean one. CyberForge is not limited by the availability of disclosed vulnerabilities, therefore it can scale in comparison to data augmentation techniques which rely on historic CVE data. The resulting corpus holds 1034 validated vulnerabilities across 80 projects and 63 weakness categories, with edit locality similar to real CVE patches under a real-versus-real noise floor. Fine-tuning on trajectories collected over this corpus improves SEC-bench patch repair by +3.3 to +14.7 points, in all six configurations of three model scales and two teachers, with the 31B student reaching its GPT-5.4-mini teacher, 72.7% against 74.0%. These gains generalize out of distribution to PatchEval, a corpus containing other programming languages, where every configuration also improves and the 31B student passes its teacher.
Amine Lbath, Manan Suri, Aurelien Delaitre +4
National Institute of Standards and Technology · Université Grenoble Alpes, CNRS · University of Maryland, College Park