SecureVibe: Making Vibe Coding More Secure

Danqing Wang; Baolin Peng; Zhepei Wei; Isadora White; Wenlin Yao; Hao Cheng; Qianhui Wu; Minseon Kim

cs.CRSep 29, 2026

SecureVibe: Making Vibe Coding More Secure

Authors: Danqing Wang, Baolin Peng, Zhepei Wei, Isadora White, Wenlin Yao, Hao Cheng, Qianhui Wu, Minseon Kim, +3 more

Organizations: Carnegie Mellon University · Microsoft Research · University of Virginia, Charlottesville · University of California, San Diego

Abstract

As vibe coding becomes increasingly capable and widespread, security vulnerabilities in even functionally correct solutions are a growing concern. When investigating functionally correct but insecure solutions, we find that the insecure agent is less than half as likely to conduct effective planning and testing for the hidden security risks behind the functional requirements. Motivated by this, we develop SECUREVIBE, a training recipe that explicitly targets planning and testing for code security. SECUREVIBE constructs training signals around these security behaviors. It includes supervised fine-tuning on the security suite with 4 security tasks, and post-training methods, SECUREVIBE_rl and SECUREVIBE_hg, to enhance security capabilities from verifiable execution feedback and hint-based self-supervision. Our SECUREVIBE outperforms the baseline on two types of security coding tasks across 4 benchmarks. Specifically, SECUREVIBE improves the security pass@1 by 6.9 points on BaxBench. The gains extend to unseen CWE categories, with improvements of 11.5 points on SusVibes. Meanwhile, it also improves functionality pass@1 by 13.6 points on the security coding task SusVibes and 4.1 points on the generic coding task SWE-bench Verified. Further analysis offers two practical insights: (i) diversifying supervision across security planning, coding, and testing strengthens security behaviors more effectively than adding coding trajectories alone, and (ii) hint-guided supervision is particularly valuable when the agent's existing security capabilities are insufficient to learn effectively from outcome feedback.

Figures & tables

Figure 1: SecureVibe : making vibe coding more secure by enhancing security capabilities via a mixture of security tasks and post-training.

Benchmark	Model	FuncPass	$\Delta$ FuncPass	SecPass	$\Delta$ SecPass
PatchEval-Gen (100)	Qwen 3.5 35B	$58.33_{\pm 4.04}$	–	$13.33_{\pm 1.53}$	–
	Security Only SFT	$62.00_{\pm 2.00}$	$+3.67$	$9.33_{\pm 1.53}$	$-4.00$
	$\textsc{SecureVibe}_{rl}$	$64.00_{\pm 3.61}$	$+5.67$	$12.67_{\pm 2.52}$	$-0.66$
	Simple Mixture SFT	$54.67_{\pm 2.31}$	$-3.66$	$10.00_{\pm 3.00}$	$-3.33$
	$\textsc{SecureVibe}_{base}$	$62.00_{\pm 1.00}$	$+3.67$	$12.67_{\pm 1.15}$	$-0.66$
	$\textsc{SecureVibe}_{rl}$	$60.67_{\pm 3.51}$	$+2.34$	$14.00_{\pm 1.00}$	$+0.67$

Table 1: Results for SFT data recipes and post-training methods on the held-out validation set of PatchEval-Gen and AutoBaxBench.

\Delta

denotes the difference from the Qwen 3.5 35B baseline. Results are averaged over 3 repetition runs.

		Security Coding				Generic Coding
		SusVibes (186)		BaxBench (392)		SWE-bench Verified (500)
Model	Size	FuncPass	SecPass	FuncPass	SecPass	Pass@1
GPT 5.6 Sol	–	$85.83_{\pm 1.63}$	$22.40_{\pm 1.56}$	$57.91_{\pm 0.44}$	$45.32_{\pm 0.39}$	$96.20^{*}_{\pm 0.86}$
MiniMax M2.7	230B	$53.23_{\pm 2.16}$	$13.98_{\pm 1.73}$	$58.16_{\pm 2.21}$	$33.42_{\pm 1.32}$	$73.80^{*}_{\pm 1.97}$
Qwen 3.5 35B	35B	$28.14_{\pm 0.82}$	$10.57_{\pm 0.62}$	$32.91_{\pm 2.87}$	$19.71_{\pm 1.15}$	$60.90_{\pm 1.50}$
$\textsc{SecureVibe}_{base}$	35B	$41.76_{\pm 2.54}$	$13.62_{\pm 3.10}$	$36.48_{\pm 1.56}$	$25.34_{\pm 1.31}$	$\mathbf{65.00}_{\pm 1.20}$

Table 2: Security Generalization. SFT is

\textsc{SecureVibe}_{base}

. Bold indicates the best mean for each benchmark and metric among Qwen 3.5 35B variants.

Figure 4: Percentage of trajectories exhibiting security planning, coding, and testing under different SFT recipes. Security Suite has the highest prevalence of all three behaviors on both benchmarks.

Dataset	# Instances	Metric	Qwen 3.5 35B	$\textsc{SecureVibe}_{base}$	$\textsc{SecureVibe}_{rl}$	$\textsc{SecureVibe}_{hg}$
SusVibes	78	FuncPass	$20.51$	$30.77$ $(+10.26)$	$33.33$ $(+12.82)$	$\mathbf{38.46}$ $(+17.95)$
SusVibes	78	SecPass	$7.69$	$12.82$ $(+5.13)$	$15.38$ $(+7.69)$	$\mathbf{19.23}$ $(+11.54)$
BaxBench	140	FuncPass	$31.43$	$33.57$ $(+2.14)$	$39.29$ $(+7.86)$	$\mathbf{40.00}$ $(+8.57)$
BaxBench	140	SecPass	$10.71$	$13.57$ $(+2.86)$	$\mathbf{17.86}$ $(+7.14)$	$13.57$ $(+2.86)$

Table 3: Generalization performance on instances containing unseen CWEs. Parenthesized values denote percentage-point changes from the Qwen 3.5 35B baseline. The best score in each row is bold.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Behavior	Generic	$n$	Target-aligned	$n$
Planning	$-0.055\;[-0.195,+0.124]$	166	$+0.285\;[+0.111,+0.443]$	166
Coding	$-0.116\;[-0.205,+0.025]$	166	$+0.469\;[+0.297,+0.612]$	166
Testing	$-0.194\;[-0.298,-0.065]$	165	$+0.199\;[+0.010,+0.359]$	166

Appendix

Table 4: Associations between security-behavior presence and security passing. Entries show Pearson’s phi with pointwise project-bootstrap 95% intervals.

Language	PatchEval-Gen	AutoBaxBench	BaxBench	SusVibes
JavaScript	75	160	112	–
Python	44	160	112	186
Go	81	120	84	–
Ruby	–	40	28	–
Rust	–	40	28	–
PHP	–	40	28	–

Appendix

Table 5: Distribution of benchmark instances across programming languages. A dash indicates that a language is not included in the benchmark.

CWE Theme	PatchEval-Gen	AutoBaxBench	BaxBench	SusVibes
Injection	53	294	252	14
XSS	4	182	98	24
Path Traversal	40	154	140	15
Access Control	45	252	140	13
Auth & Credentials	11	182	70	12
Input Validation	12	280	28	7

Appendix

Table 6: Benchmark instances categorized by CWE theme. A single instance may contain multiple CWEs and can therefore be counted under multiple themes.

Explore similar work

Dec 2, 2025

Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

Vibe coding is a new software development paradigm in which human engineers prompt a large language model (LLM) agent to complete complex coding tasks with little supervision. Although vibe coding is increasingly adopted, is the generated code really safe to deploy in production? To investigate this question, we propose SUSVIBES, a benchmark consisting of 186 feature-request software engineering tasks from real-world open-source projects, for which, human programmers committed vulnerable implementations. We evaluate 12 widely used coding agentic settings with frontier models on the benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 57% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 11.8% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe coding, particularly in security-sensitive applications. The code and dataset are available at https://github.com/LeiLiLab/susvibes. The leaderboard is at https://leililab.github.io/susvibes-leaderboard.

Songwen Zhao, Danqing Wang, Kexun Zhang +3

Carnegie Mellon University, Language Technologies Institute · Columbia University · Johns Hopkins University +1

AI Coding Agents Software Security AI Agent Security Benchmarks Code Generation Benchmarks

Ask PDF

Oct 10, 2026

Last Secure Code Benchmark: Is the Code You Vibe Secure?

AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.

Tianyi Wu, Chao Ge, Peng Chen +12

National University of Singapore · Qwen Team, Alibaba Token Hub, Alibaba Group · Cloud Safety and Security, Alibaba Cloud, Alibaba Group

Secure Code Generation AI Agent Security Benchmarks Software Security AI Coding Agents Code Generation Benchmarks

Ask PDF

Oct 1, 2026

AuraForge: Scaling Security Supervision for Training Coding Agents

Coding agents are now proficient enough to generate complex software applications from a single prompt. As their capabilities have grown, human oversight has increasingly shifted from line-by-line code review toward hands-off evaluation of outcomes. However, recent studies have shown that such a transition exposes a critical risk: functional correctness alone does not guarantee a secure implementation. Despite growing attention to code security, training safer coding agents remains challenging because reliable security supervision is difficult to obtain at scale from real-world repositories. We introduce AuraForge to synthesize and validate executable security tests for training secure coding agents. Our approach combines attack-oriented test synthesis, language-extensible task construction, and safeguards against reward hacking. Using AuraForge, we construct AuraGym, a multi-language and multi-CWE executable training gym: 679 executable feature-implementation tasks from 344 real-world repositories across Python, JavaScript, and TypeScript, covering 177 CWE categories. On the subset with human-written security tests, AuraForge produces about 3 times as many test cases on average and reduces the false-positive rate by 83.23%, allowing alternative secure implementations to receive correct supervision. Training Qwen3.5-4B with synthesized security tests gains larger improvements than human-written security tests (average 19.7 FuncPass and 6.2 SecPass vs. 14.9 FuncPass and 4.4 SecPass) on three languages. These results demonstrate that AuraForge provides more diverse and reliable security supervision to train secure coding agents.

Danqing Wang, Songwen Zhao, Harsh Sharma +4

Carnegie Mellon University · University of California, Los Angeles · ScOp Venture Capital.

AI Coding Agents Software Security AI Agent Security Secure Code Generation Automated Test Generation

Ask PDF