SecureVibe: Making Vibe Coding More Secure

Danqing Wang; Baolin Peng; Zhepei Wei; Isadora White; Wenlin Yao; Hao Cheng; Qianhui Wu; Minseon Kim

cs.CRSep 29, 2026

SecureVibe: Making Vibe Coding More Secure

Authors: Danqing Wang, Baolin Peng, Zhepei Wei, Isadora White, Wenlin Yao, Hao Cheng, Qianhui Wu, Minseon Kim, +3 more

Organizations: Carnegie Mellon University · Microsoft Research · University of Virginia, Charlottesville · University of California, San Diego

Abstract

As vibe coding becomes increasingly capable and widespread, security vulnerabilities in even functionally correct solutions are a growing concern. When investigating functionally correct but insecure solutions, we find that the insecure agent is less than half as likely to conduct effective planning and testing for the hidden security risks behind the functional requirements. Motivated by this, we develop SECUREVIBE, a training recipe that explicitly targets planning and testing for code security. SECUREVIBE constructs training signals around these security behaviors. It includes supervised fine-tuning on the security suite with 4 security tasks, and post-training methods, SECUREVIBE_rl and SECUREVIBE_hg, to enhance security capabilities from verifiable execution feedback and hint-based self-supervision. Our SECUREVIBE outperforms the baseline on two types of security coding tasks across 4 benchmarks. Specifically, SECUREVIBE improves the security pass@1 by 6.9 points on BaxBench. The gains extend to unseen CWE categories, with improvements of 11.5 points on SusVibes. Meanwhile, it also improves functionality pass@1 by 13.6 points on the security coding task SusVibes and 4.1 points on the generic coding task SWE-bench Verified. Further analysis offers two practical insights: (i) diversifying supervision across security planning, coding, and testing strengthens security behaviors more effectively than adding coding trajectories alone, and (ii) hint-guided supervision is particularly valuable when the agent's existing security capabilities are insufficient to learn effectively from outcome feedback.

Figures & tables

Figure 1: SecureVibe : making vibe coding more secure by enhancing security capabilities via a mixture of security tasks and post-training.

Benchmark	Model	FuncPass	$\Delta$ FuncPass	SecPass	$\Delta$ SecPass
PatchEval-Gen (100)	Qwen 3.5 35B	$58.33_{\pm 4.04}$	–	$13.33_{\pm 1.53}$	–
	Security Only SFT	$62.00_{\pm 2.00}$	$+3.67$	$9.33_{\pm 1.53}$	$-4.00$
	$\textsc{SecureVibe}_{rl}$	$64.00_{\pm 3.61}$	$+5.67$	$12.67_{\pm 2.52}$	$-0.66$
	Simple Mixture SFT	$54.67_{\pm 2.31}$	$-3.66$	$10.00_{\pm 3.00}$	$-3.33$
	$\textsc{SecureVibe}_{base}$	$62.00_{\pm 1.00}$	$+3.67$	$12.67_{\pm 1.15}$	$-0.66$
	$\textsc{SecureVibe}_{rl}$	$60.67_{\pm 3.51}$	$+2.34$	$14.00_{\pm 1.00}$	$+0.67$

Table 1: Results for SFT data recipes and post-training methods on the held-out validation set of PatchEval-Gen and AutoBaxBench.

\Delta

denotes the difference from the Qwen 3.5 35B baseline. Results are averaged over 3 repetition runs.

		Security Coding				Generic Coding
		SusVibes (186)		BaxBench (392)		SWE-bench Verified (500)
Model	Size	FuncPass	SecPass	FuncPass	SecPass	Pass@1
GPT 5.6 Sol	–	$85.83_{\pm 1.63}$	$22.40_{\pm 1.56}$	$57.91_{\pm 0.44}$	$45.32_{\pm 0.39}$	$96.20^{*}_{\pm 0.86}$
MiniMax M2.7	230B	$53.23_{\pm 2.16}$	$13.98_{\pm 1.73}$	$58.16_{\pm 2.21}$	$33.42_{\pm 1.32}$	$73.80^{*}_{\pm 1.97}$
Qwen 3.5 35B	35B	$28.14_{\pm 0.82}$	$10.57_{\pm 0.62}$	$32.91_{\pm 2.87}$	$19.71_{\pm 1.15}$	$60.90_{\pm 1.50}$
$\textsc{SecureVibe}_{base}$	35B	$41.76_{\pm 2.54}$	$13.62_{\pm 3.10}$	$36.48_{\pm 1.56}$	$25.34_{\pm 1.31}$	$\mathbf{65.00}_{\pm 1.20}$

Table 2: Security Generalization. SFT is

\textsc{SecureVibe}_{base}

. Bold indicates the best mean for each benchmark and metric among Qwen 3.5 35B variants.

Figure 4: Percentage of trajectories exhibiting security planning, coding, and testing under different SFT recipes. Security Suite has the highest prevalence of all three behaviors on both benchmarks.

Dataset	# Instances	Metric	Qwen 3.5 35B	$\textsc{SecureVibe}_{base}$	$\textsc{SecureVibe}_{rl}$	$\textsc{SecureVibe}_{hg}$
SusVibes	78	FuncPass	$20.51$	$30.77$ $(+10.26)$	$33.33$ $(+12.82)$	$\mathbf{38.46}$ $(+17.95)$
SusVibes	78	SecPass	$7.69$	$12.82$ $(+5.13)$	$15.38$ $(+7.69)$	$\mathbf{19.23}$ $(+11.54)$
BaxBench	140	FuncPass	$31.43$	$33.57$ $(+2.14)$	$39.29$ $(+7.86)$	$\mathbf{40.00}$ $(+8.57)$
BaxBench	140	SecPass	$10.71$	$13.57$ $(+2.86)$	$\mathbf{17.86}$ $(+7.14)$	$13.57$ $(+2.86)$

Table 3: Generalization performance on instances containing unseen CWEs. Parenthesized values denote percentage-point changes from the Qwen 3.5 35B baseline. The best score in each row is bold.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Behavior	Generic	$n$	Target-aligned	$n$
Planning	$-0.055\;[-0.195,+0.124]$	166	$+0.285\;[+0.111,+0.443]$	166
Coding	$-0.116\;[-0.205,+0.025]$	166	$+0.469\;[+0.297,+0.612]$	166
Testing	$-0.194\;[-0.298,-0.065]$	165	$+0.199\;[+0.010,+0.359]$	166

Appendix

Table 4: Associations between security-behavior presence and security passing. Entries show Pearson’s phi with pointwise project-bootstrap 95% intervals.

Language	PatchEval-Gen	AutoBaxBench	BaxBench	SusVibes
JavaScript	75	160	112	–
Python	44	160	112	186
Go	81	120	84	–
Ruby	–	40	28	–
Rust	–	40	28	–
PHP	–	40	28	–

Appendix

Table 5: Distribution of benchmark instances across programming languages. A dash indicates that a language is not included in the benchmark.

CWE Theme	PatchEval-Gen	AutoBaxBench	BaxBench	SusVibes
Injection	53	294	252	14
XSS	4	182	98	24
Path Traversal	40	154	140	15
Access Control	45	252	140	13
Auth & Credentials	11	182	70	12
Input Validation	12	280	28	7

Appendix

Table 6: Benchmark instances categorized by CWE theme. A single instance may contain multiple CWEs and can therefore be counted under multiple themes.

Explore similar work

Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
Dec 2, 2025Songwen Zhao, Danqing Wang, Kexun Zhang +3AI Coding Agents Software Security
Carnegie Mellon University, Language Technologies Institute · Columbia University · Johns Hopkins University +1
Last Secure Code Benchmark: Is the Code You Vibe Secure?
Oct 10, 2026Tianyi Wu, Chao Ge, Peng Chen +12Secure Code Generation AI Agent Security Benchmarks
National University of Singapore · Qwen Team, Alibaba Token Hub, Alibaba Group · Cloud Safety and Security, Alibaba Cloud, Alibaba Group
AuraForge: Scaling Security Supervision for Training Coding Agents
Oct 1, 2026Danqing Wang, Songwen Zhao, Harsh Sharma +4AI Coding Agents Software Security
Carnegie Mellon University · University of California, Los Angeles · ScOp Venture Capital.

Dec 2, 2025J/K move · Enter open · S save

Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

Songwen Zhao, Danqing Wang, Kexun Zhang, Jiaxuan Luo+2

Carnegie Mellon University, Language Technologies Institute · Columbia University · Johns Hopkins University+1

AI Coding Agents Software Security AI Agent Security Benchmarks Code Generation Benchmarks

Open paper AskPDF arXiv

Vibe coding is a new software development paradigm in which human engineers prompt a large language model (LLM) agent to complete complex coding tasks with little supervision. Although vibe coding is increasingly adopted, is the generated code really safe to deploy in production? To investigate this question, we propose SUSVIBES, a benchmark consisting of 186 feature-request software engineering tasks from real-world open-source projects, for which, human programmers committed vulnerable implementations. We evaluate 12 widely used coding agentic settings with frontier models on the benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 57% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 11.8% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe coding, particularly in security-sensitive applications. The code and dataset are available at https://github.com/LeiLiLab/susvibes. The leaderboard is at https://leililab.github.io/susvibes-leaderboard.

Abstract

Explore similar work

Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

Last Secure Code Benchmark: Is the Code You Vibe Secure?

AuraForge: Scaling Security Supervision for Training Coding Agents