AuraForge: Scaling Security Supervision for Training Coding Agents

Danqing Wang; Songwen Zhao; Harsh Sharma; Jierui Wang; Andre Vicente Duarte; Ivan Bercovich; Lei Li

cs.CROct 1, 2026

AuraForge: Scaling Security Supervision for Training Coding Agents

Authors: Danqing Wang, Songwen Zhao, Harsh Sharma, Jierui Wang, Andre Vicente Duarte, Ivan Bercovich, Lei Li

Organizations: Carnegie Mellon University · University of California, Los Angeles · ScOp Venture Capital.

Abstract

Coding agents are now proficient enough to generate complex software applications from a single prompt. As their capabilities have grown, human oversight has increasingly shifted from line-by-line code review toward hands-off evaluation of outcomes. However, recent studies have shown that such a transition exposes a critical risk: functional correctness alone does not guarantee a secure implementation. Despite growing attention to code security, training safer coding agents remains challenging because reliable security supervision is difficult to obtain at scale from real-world repositories. We introduce AuraForge to synthesize and validate executable security tests for training secure coding agents. Our approach combines attack-oriented test synthesis, language-extensible task construction, and safeguards against reward hacking. Using AuraForge, we construct AuraGym, a multi-language and multi-CWE executable training gym: 679 executable feature-implementation tasks from 344 real-world repositories across Python, JavaScript, and TypeScript, covering 177 CWE categories. On the subset with human-written security tests, AuraForge produces about 3 times as many test cases on average and reduces the false-positive rate by 83.23%, allowing alternative secure implementations to receive correct supervision. Training Qwen3.5-4B with synthesized security tests gains larger improvements than human-written security tests (average 19.7 FuncPass and 6.2 SecPass vs. 14.9 FuncPass and 4.4 SecPass) on three languages. These results demonstrate that AuraForge provides more diverse and reliable security supervision to train secure coding agents.

Figures & tables

Figure 1: AuraForge . Starting from real repositories and historical security fixes, The Forge derives feature requests without leaking security hints, and constructs language-extensible execution environments. The Aura synthesizes security tests and validates them. The Seal removes implementation-leakage channels. Finally, it produces 679 executable instances from 344 repositories across 177 CWE categories and 3 languages.

Dataset	Task setting	Instances	Repos.	CWEs	Languages
General SWE agent training datasets
SWE-Gym	Issue resolution	2,438	11	—	Python
SWE-smith	Synthetic bug repair	$\approx$ 50,000	128	—	Python
SWE-rebench	Issue resolution	21,336	3,468	—	Python
SWE-rebench V2	Issue resolution	32,079	3,617	—	20 languages
Secure-coding evaluation benchmarks

Table 1: Comparison of general SWE training datasets and secure-coding benchmarks. Instances denote tasks rather than trajectories. A dash indicates that repository counts are not applicable or CWE coverage is not reported.

Figure 2: Division between the shared core and the language adapter in AuraForge .

Figure 3: Retrieval rates of SusVibes runs, per model and channel (runs pooled over scaffolds), with the repository history present and the network reachable.

Panel of Figure 4 — Figure 4: Comparison of human-written AuraGym h and synthesized AuraGym : (a) task and repository counts; (b) distinct CWE categories by language; and (c) average security tests per task.

Model	Size	SusVibes (186)		SusVibes tsjs (82)
Model	Size	FuncPass	SecPass	FuncPass	SecPass
GPT 5.6 Sol	–	84.40	21.50	92.70	18.30
Muse Spark 1.3	–	79.57	15.59	82.90	19.50
GLM 4.7 Flash	30B-A3B	24.73	4.30	24.39	4.88
Nemotron 3.5	30B-A3B	15.05	3.23	25.61	3.66
Qwen 3.5 4B	4B	12.90	2.10	12.19	0.00

Table 3: Functional and security performance on the SusVibes (Python) and newly created TypeScript/JavaScript evaluation sets. SusVibes is the original Python version from Zhao et al. (2026) and SusVibes tsjs is the test split for the new TS and JS. Nemotron 3.5 is the Lightning version.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Selection stage	Candidates	Removed	Repos.
Initial JS/TS export	3,189	—	—
Implementation, test, and patch-size filters	967	2,222	—
Candidate deduplication	909	58	514
Fixing-commit date $\geq$ 2024-01-01	311	598	—
Identified Node.js major $\geq$ 18	255	56	127
Repository primary language is JS/TS	249	6	—

Appendix

Table 4: MoreFixes filtering. Removed denotes the reduction from the preceding row; dashes indicate unreported counts.

Selection stage	Candidates	Removed	Repos.
Retrieved unique commits	3,742	—	—
Patch structure, parent, and metadata filters	2,371	1,371	—
Patch includes JS/TS	2,204	167	—
Patch includes test files	1,389	815	489
Fixing-commit date $\geq$ 2024-01-01	896	493	—
Identified Node.js major $\geq$ 18	817	79	198

Appendix

Table 5: GHSA filtering and cross-source deduplication. Removed denotes the reduction from the preceding row; dashes indicate unreported counts.

Channel	How the reference is obtained	Countermeasure
git history	git show , git log -p , or git checkout on a pre-mask revision	history stripped; the task ships as a single commit
local copy	reading a copy already on disk: site-packages , a build directory, a vendored tree	image sanitized
internet	fetching the upstream file or archive from a code host	command filtering while solving
package manager	installing the upstream package at its fixed version, then reading its source	command filtering while solving

Appendix

Table 6: Retrieval channels observed in the audit and how each is closed.

Version	Language	# Instances	# Repos	# CWEs	Avg. # target files	Avg. # func. tests	Avg. # security tests
AuraGym	Python	476	266	148	1.65	73.22	7.84
	JavaScript	46	36	31	1.70	32.22	11.54
	TypeScript	161	46	71	1.99	20.52	10.20
	Overall	679	344	177	1.72	53.54	8.62
$\texttt{AuraGym}_{h}$	Python	202	109	98	1.39	89.51	2.38
	JavaScript	53	42	32	1.79	32.22	3.80

Appendix

Table 7: Statistics of the synthesized

\texttt{AuraGym}_{s}

and the human-written comparison set

\texttt{AuraGym}_{h}

. The numbers of functional tests, security tests, and target files are reported as per-instance averages. An instance whose repository mixes JavaScript and TypeScript is counted under both languages.

Language	Model	Setting	# Traj.	# Inst.	Coverage	Avg. turns	Avg. tokens (K)
Python	DeepSeek	generic	392	163	34.24%	39.75	52.15
		security-hint	77	77	16.18%	46.53	59.84
		overall	469	226	47.48%	40.86	53.42
	Muse Spark	generic	462	152	31.67%	42.91	35.59
		security-hint	161	112	23.33%	47.22	43.27
		overall	623	264	55.00%	44.02	37.57

Appendix

Table 8: Statistics of the trajectories collected by DeepSeek-V4.1-Flash and Muse Spark 1.3 that are functionally correct and secure. Coverage denotes the percentage of instances represented by at least one retained trajectory within each model’s instance pool (679 for DeepSeek-V4.1-Flash; 710 for Muse Spark 1.3, which was collected on an earlier version of AuraGym ). Token counts are reported in units of 1,024 tokens.

Kind: the question the test fails	Category: the test…	Example	Human	Synth.
Rejects a secure solution (false positive): the test demands more than that the attack has no effect
Fix-bound : does the test demand a detail of the historical fix that security leaves free?	…wants the exact form of a correct result	A converter must render an over-long heading tag as <h6> , as the fix does; a solution that renders it as plain text stops the attack just as well (Appendix F.6.2 ); likewise a query must have the exact SQL text the fix sends (Appendix F.6.3 ).	34	1
	…wants one particular safe reaction where another exists	A model loader must refuse a file in safe mode; a solution instead loads it through a restricted loader that cannot run code (Appendix F.6.1 ).	20	2
	…wants the error’s wording or type	The error must say “arbitrary code execution”; a solution refuses the same file with a different message (Appendix F.6.1 ).	7	1
	…calls a helper only the fix defines	The test imports a validator function the fix added; a solution that validates inline has no function of that name.	7	0
	all fix-bound		68 (32)	4 (3)

Appendix

Table 9: Causes of wrong verdicts. Every wrong verdict behind Table 2 is assigned, from the adjudication record, to the category of test error that produced it; the categories are grouped into kinds, each named by the question the test fails. Counts are wrong verdicts (human / synthesized): secure solutions rejected for the false-positive kinds, vulnerable solutions accepted for the false-negative kinds; kind rows give the total, with tasks in parentheses. Each example is a real case; those in Section F.6 are referenced.

Suites that…	Human		Synthesized
Suites that…	share of tasks	FPR	share of tasks	FPR
assert the attack’s effect is absent	48%	12.0%	87%	3.0%
assert the outcome the fix produces	52%	20.7%	13%	2.0%

Appendix

Table 10: Assertion audit of both suites on the 183 tasks with a secure solution. Each suite’s security tests were classified, blind to the grades, by what they assert: that the attack’s effect is absent, or the outcome the historical fix produces; mixed suites are counted by their majority. FPR is over the secure solutions of the tasks in that class.

Family	Solutions	Tasks	FP (secure rejected)		FN (vulnerable accepted)
Family	Solutions	Tasks	Human	Synth.	Human	Synth.
Authentication / authorization	495	66	10/75	4/75	5/420	3/420
Input validation / other	368	45	15/74	8/74	1/294	0/294
Resource exhaustion / ReDoS	334	54	19/99	6/99	10/235	6/235
Injection (SQL / command / code)	255	36	7/51	0/51	0/204	0/204
Path traversal / link following	244	33	4/55	0/55	2/189	8/189

Appendix

Table 11: Wrong verdicts by vulnerability family. A solution counts once under each family its task’s CWE ids fall in; 48 solutions carry no informative CWE and are in no family. Cells are wrong verdicts over the solutions of that kind (secure for FP, vulnerable for FN). Most secure-side denominators are under 100, so these are counts to read, not rates to rank.

Explore similar work

May 8, 2026cs.CR

SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization

LLM coding agents now generate code at an unprecedented scale, yet LLM-generated code introduces cybersecurity vulnerabilities into codebases without human involvement. Even when frontier models are explicitly asked to write secure production code with relevant weaknesses to avoid in context, we find that they still produce verifiable vulnerabilities on average 23% of the time across a corpus of 250 benign coding prompts. We introduce SecureForge, an automated pipeline that both audits security risks of frontier models and produces auditing-informed secure system prompts that reduce output security vulnerabilities while maintaining unit test performance. SecureForge first identifies benign prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic prompt corpus of diverse scenarios using a Markovian sampling technique to jointly maintain error rates and prompt diversity. This corpus is then used to iteratively optimize the system prompts to reduce output security vulnerabilities. On frontier models, SecureForge yields a statistically significant Pareto improvement in both unit test success and output security, with output vulnerabilities reduced by up to 48%. The resulting system prompts transfer zero-shot to in-the-wild coding agent prompts, without any exposure to real user prompt distributions during optimization.

Houjun Liu, Lisa Einstein, John Yang +5

Stanford University

Vulnerable Code Coding Agents Model Auditing Prompt Optimization

Ask PDF

Jul 22, 2026cs.CR

IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests

AI coding agents powered by LLMs are increasingly integrated into real-world software development, where they generate, edit, and execute code with autonomous access to local files and tools. Coding agents inherit security risks from both the LLM backbone, where adversarial prompts, poisoned training data, and backdoor triggers can cause models to emit insecure or attacker-chosen code, and their agentic architecture, where tool-using autonomy enables induced misuse of external APIs, data exfiltration, and persistent compromise of development environments. This paper presents a systematic evaluation of malicious issue requests against state-of-the-art coding agents (Cursor, Claude Code, and Codex Desktop), powered by two major model families (OpenAI GPT-5.3 Codex/GPT-5.4 and Anthropic Sonnet 4.6). Our novel benchmark IssueTrojanBench contains malicious issues that are constructed based on four novel attack categories (i.e., embedded as malicious instructions in issues), six delivery vectors (e.g., PDF, or issue comment), and further augmented by perturbations. Our results reveal critical vulnerabilities in the as-deployed modern coding agents, i.e., 66.5% of the malicious issues from IssueTrojanBench penetrate all the guardrails (agent- and LLM-level) of coding agents. Our further analysis shows that rejection is almost entirely from LLMs rather than the agent frameworks, with GPT models broadly vulnerable and Sonnet 4.6 exhibiting more selective, risk-aware blocking of high-impact actions. Our evaluation also highlights that the current agent-level defense strategy offers limited additional protection for coding agents. Our findings highlight the urgent need for stronger agent- and model-level safety mechanisms to protect AI coding agents.

Ankur Singh, Jinqiu Yang, Tse-Hsun Chen

Concordia University Montr´eal, Canada

Coding Agents Vulnerable Code Malicious Agents LLM Defense Mechanisms Large Language Model Backbones

Ask PDF

Aug 6, 2026cs.CR

CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training

Despite recent advances, frontier large language model (LLM) agents remain limited in discovering and patching complex vulnerabilities in real-world software. Generally available agents can already aid attackers, who only need to find one exploitable weakness, while defenders must continuously identify and patch all vulnerabilities across fast-growing codebases. Stronger defensive agents would help close this gap, yet the scarcity of security training data with reproducible build and execution environments remains a bottleneck. We present CyberForge, a framework that synthesizes executable, repository-level security training data by injecting vulnerabilities into real C/C++ projects. It validates each instance dynamically: the injected build must pass the project's unit tests, and generated proof-of-vulnerability (PoV) must trigger on the injected build and not on the clean one. CyberForge is not limited by the availability of disclosed vulnerabilities, therefore it can scale in comparison to data augmentation techniques which rely on historic CVE data. The resulting corpus holds 1034 validated vulnerabilities across 80 projects and 63 weakness categories, with edit locality similar to real CVE patches under a real-versus-real noise floor. Fine-tuning on trajectories collected over this corpus improves SEC-bench patch repair by +3.3 to +14.7 points, in all six configurations of three model scales and two teachers, with the 31B student reaching its GPT-5.4-mini teacher, 72.7% against 74.0%. These gains generalize out of distribution to PatchEval, a corpus containing other programming languages, where every configuration also improves and the 31B student passes its teacher.

Amine Lbath, Manan Suri, Aurelien Delaitre +4

National Institute of Standards and Technology · Université Grenoble Alpes, CNRS · University of Maryland, College Park

Instruction Injection Attacks Codebases Repository-Level Code Understanding

Ask PDF