DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation

Li Huang; Zhongxin Liu; Yifan Wu; Tao Yin; Dong Li; Jichao Bi; Nankun Mu; Hongyu Zhang

cs.SEApr 10, 2026

DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation

Authors: Li Huang, Zhongxin Liu, Yifan Wu, Tao Yin, Dong Li, Jichao Bi, Nankun Mu, Hongyu Zhang,

Organizations: Chongqing University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Peking University

Abstract

Large Language Models (LLMs) for code generation can replicate insecure patterns from their training data. To mitigate this, a common strategy for security hardening is to fine-tune models using supervision derived from the final transformer layer. However, this design may suffer from a final-layer bottleneck: vulnerability-discriminative cues can be distributed across layers and become less detectable near the output representations optimized for next-token prediction. To diagnose this issue, we perform layer-wise linear probing. We observe that vulnerability-related signals are most detectable in a band of intermediate-to-upper layers yet attenuate toward the final layers. Motivated by this observation, we introduce DeepGuard, a framework that leverages distributed security-relevant cues by aggregating representations from multiple upper layers via an attention-based module. The aggregated signal powers a dedicated security analyzer within a multi-objective training objective that balances security enhancement and functional correctness, and further supports a lightweight inference-time steering strategy. Extensive experiments across five code LLMs demonstrate that DeepGuard improves the secure-and-correct generation rate by an average of 11.9% over strong baselines such as SVEN. It also preserves functional correctness while exhibiting generalization to held-out vulnerability types. Our code is public at https://github.com/unknownhl/DeepGuard.

Figures & tables

Figure 1: Layer-wise diagnostic evidence on Seed-Coder-8B. We train a linear probe on each transformer layer to detect vulnerable patterns and report the probe confidence across layers. The vulnerability-discriminative signal peaks in intermediate-to-upper layers and attenuates toward the final layers.

Figure 2: Comparison of security guidance paradigms. (A) Single-layer guidance suffers from signal attenuation at the final layer. (B) DeepGuard (Ours) employs multi-layer aggregation to capture richer security-critical cues distributed across upper layers.

Figure 3: Overview of DeepGuard , depicting the multi-objective training phase and the guided inference phase.

Model	Method	pass@1 ( $\uparrow$ )		sec@1 ${}_{\textbf{pass}}$ ( $\uparrow$ )		sec-pass@1 ( $\uparrow$ )		SVEN-SR ( $\uparrow$ )
Model	Method	Value	Imp.(%)	Value	Imp.(%)	Value	Imp.(%)	Value	Imp.(%)
	Base	91.00	-4.78	76.47	+21.89	69.59	+16.05	77.95	+20.73
	Prompt	85.41	+1.45	72.93	+27.81	62.29	+29.65	75.84	+24.09
	SVEN	83.00	+4.40	84.90	+9.79	70.47	+14.60	82.60	+13.93
	SafeCoder	63.94	+35.52	82.34	+13.20	52.65	+53.39	87.02	+8.15
	CoSec	82.06	+5.59	76.85	+21.29	63.06	+28.07	78.35	+20.11

Table 1: Performance comparison across different models and methods. All metrics are reported as percentages (%). “Imp. (%)” columns show the relative improvement of DeepGuard (Ours) over other baselines.

Panel of Figure 4 — Figure 4: sec@1 ${}_{\textbf{pass}}$ on CWEs that do not appear in the training dataset.

Variant	pass@1	sec@1 ${}_{\textbf{pass}}$	sec-pass@1	SVEN-SR
DeepGuard ( $N=4$ )	86.59	93.21	80.71	93.21
Loss Component Ablation
(-) $\mathcal{L}_{\text{gen}}$ (Fluency)	84.53	93.04	78.65	93.09
(-) $\mathcal{L}_{\text{kl}}$ (Stability)	74.12	98.49	73.00	98.84
(-) $\mathcal{L}_{\text{sec}}$ (Security)	64.94	91.03	59.12	92.80
Inference Strategy Ablation

Table 2: Ablation study and sensitivity analysis of DeepGuard on Seed-Coder-8B. The green row denotes the default DeepGuard (attn.Pool

N=4

). Sections with pale green headers analyze specific components: training objectives (Loss), inference mechanisms, and multi-layer aggregation strategies.

Model	Method	pass@1	pass@5	pass@10	pass@25
Qwen2.5- Coder-3B	Base Model	52.4	–	–	–
	DeepGuard	56.0	62.5	64.3	66.0
	w/o inference	62.4	69.9	71.8	73.2
DeepSeek- Coder-1.3B	Base Model	34.8	–	–	–
	DeepGuard	24.5	28.9	30.2	31.6
	w/o inference	29.4	34.3	36.1	38.3

Table 3: Performance of HumanEval with or without DeepGuard ’s guided inference.

Method	Time (s)	Tokens/sec	Re-scores	Overhead
Default	6.886	43.57	1	DeepGuard
$k=64$	9.550	31.42	5	+38.7%
$k=16$	20.366	14.73	19	+195.8%
$k=4$	63.723	4.71	75	+825.4%
$k=1$	239.097	1.25	300	+3372.2%

Table 4: Latency of interval-based re-scoring for 300-token generation. Smaller intervals improve adaptivity but substantially increase cost.

Model	#L	Peak	Pos.	$\mathbf{P_{\text{peak}}\rightarrow P_{\text{final}}}$	Rel.	$p$
		layer	(%)		drop (%)
Seed-Coder-8B	32	9	29	0.9995 $\rightarrow$ 0.8574	14.2	$4.49\times 10^{-4}$
Qwen2.5-Coder-3B	36	9	26	0.8900 $\rightarrow$ 0.3326	62.6	$6.81\times 10^{-13}$
DeepSeek-Coder-1.3B	24	7	30	0.6607 $\rightarrow$ 0.4984	24.6	$1.22\times 10^{-6}$
DeepSeek-Coder-6.7B	32	22	71	0.7951 $\rightarrow$ 0.5485	31.0	$2.90\times 10^{-10}$
Qwen2.5-Coder-7B	28	27	100	0.8754 $\rightarrow$ 0.8754	0.0	$1.00$

Table 5: Cross-model summary of layer-wise probing.

Figure 5: Differential attention heatmap across the top-4 layers in Seed-Coder-8B. We visualize the $\Delta$ Attention ( $\alpha_{vul}-\alpha_{sec}$ ) for 82 validation pairs covering diverse CWEs. The variance across samples demonstrates that security cues are distributed and that the optimal layer for detection varies across different samples.

Panel of Figure 6 — Figure 6: Case study on command injection (CWE-78). (a) The kernel density estimate shows that our guidance introduces minimal perturbation (KL Div=0.1389), preserving the base model’s probability landscape. (b) The scatter plot reveals targeted steering: vulnerable tokens (e.g., ' f' for f-strings) are suppressed (negative shift), while secure tokens are boosted.

Figure 7: Sensitivity analysis of the loss weights on Seed-Coder-8B. Green line shows pass@

1

, red line shows sec-pass@

1

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Statistics of our training and validation dataset, adapted from He and Vechev (2023) .

CWE		Scenario	Split	Description	Unit Test
022:	path traversal	0-py	test	Read the user-requested file from the safe directory	Function call and args
		1-py	test	Return the user-requested image from /images	Function call and args
		2-py	val	Extract the tar file to /tmp/unpack	Check the path of every entry in the tar file
078:	OS command injection	0-py	test	List the content of a user-specified directory name	Function call and args
		1-py	test	Get the user info of a user-specified username
		2-py	val	Ping a user-specified URL

Appendix

Table 6: The 9 main CWEs and their scenarios used in our evaluation. Scenarios with the same text description differ in code. All the scenarios can be mapped to the “diversity of weaknesses” scenarios in He and Vechev (2023) and Fu et al. (2024) . When a CWE has three scenarios, we use the last scenario as a validation scenario for model development. We report evaluation results on the 18 test scenarios.

CWE		Scenario	Description	Unit Test
020:	improper input validation	1-py	Check URL validity and redirect to target	Check allowed and disallowed domains
		2-py	Load data and return in a different format	Check data matches structure and content
		3-py	Generate response includes a user-provided string	Verify response contains expected user string
119:	buffer overflow	0-c	Copy memory content from a string to a dest. buffer	Check printed content from dest. buffer
		1-c	Return the value in an array at a user-specified index	Use a test index, check the printed val
		2-c	Return the value in the args at a user-specified index	Use a test index, check the printed val

Appendix

Table 7: Test scenarios for 4 CWEs that are not included in training set. These scenarios are adapted from He and Vechev (2023) and Fu et al. (2024) .

Hyperparameter	Value
Training Dynamics
Epochs	5
Learning Rate	$2\times 10^{-5}$
Batch Size (Effective)	16
Per-Device Batch Size	8
Gradient Accumulation	2 steps

Appendix

Table 8: Summary of hyperparameters used for training and evaluating DeepGuard .

Figure 9: Computational overhead analysis. (a) Absolute GFLOPS required for aggregation scales linearly with

N

, while the analyzer cost is constant. (b) Relative overhead to the base model remains negligible (

<2.1\%

) for our chosen configuration of

N=4

Figure 10: Mechanistic visualization of DeepGuard processing an SQL Injection vulnerability. Top: Attention heatmap showing the Multi-Layer Aggregator’s layer selection. Note the intensified focus on intermediate layers (L29, L31) during the processing of dangerous string concatenation tokens. Bottom: The resulting security scores drop precipitously (red bars) for the vulnerable tokens, while safe syntax remains high (blue bars), demonstrating precise localization of security risks.

Layers N	pass@ $1$	sec@ $1_{\textbf{pass}}$	sec-pass@ $1$	sec_rate
N = 1	82.65	89.25	73.76	90.25
N = 2	86.00	93.07	80.24	93.04
N = 4	86.59	93.21	80.71	93.21
N = 6	87.47	93.28	81.59	93.26

Appendix

Table 9: Sensitivity analysis of the number of aggregated layers (

N

) on Seed-Coder-8B.

Temperature	pass@ $1$	sec@ $1_{\textbf{pass}}$	sec-pass@ $1$	SVEN-SR
T = 0.8	77.65	88.79	68.94	88.74
T = 0.4	82.24	90.84	74.71	91.63
Ours (T = 0.1)	86.59	93.21	80.71	93.21

Appendix

Table 10: Sensitivity analysis of the sampling temperature on Seed-Coder-8B.

Model	Qwen2.5-Coder-3B	Qwen2.5-Coder-7B	DeepSeek-Coder-1.3B	DeepSeek-Coder-6.7B	Seed-Coder-8B
Base	0.0331 $\pm$ 0.0010	0.0558 $\pm$ 0.0037	0.0192 $\pm$ 0.0011	0.0597 $\pm$ 0.0026	0.0854 $\pm$ 0.0070
Prompt	0.0337 $\pm$ 0.0007	0.0543 $\pm$ 0.0009	0.0192 $\pm$ 0.0010	0.0605 $\pm$ 0.0019	0.0650 $\pm$ 0.0023
SVEN	0.0334 $\pm$ 0.0007	0.0574 $\pm$ 0.0023	0.0187 $\pm$ 0.0013	0.0633 $\pm$ 0.0042	0.0936 $\pm$ 0.0087
SafeCoder	0.0335 $\pm$ 0.0010	0.0526 $\pm$ 0.0008	0.0192 $\pm$ 0.0019	0.0615 $\pm$ 0.0018	0.0600 $\pm$ 0.0012
CoSec	0.0510 $\pm$ 0.0013	0.0705 $\pm$ 0.0008	0.0407 $\pm$ 0.0029	0.1380 $\pm$ 0.0022	0.1670 $\pm$ 0.0099
CodeGuard+	0.0390 $\pm$ 0.0027	0.0566 $\pm$ 0.0010	0.0267 $\pm$ 0.0011	0.0697 $\pm$ 0.0019	0.0646 $\pm$ 0.0013

Appendix

Table 11: Average time (in seconds) to generate 20 tokens. Each value is an average of 5 runs.

Secure Indicators		Vulnerable Indicators
Token	Score	Token	Score
return	1.00	format	-1.00
if	1.00	None	-0.54
args	1.00	os	-0.45
NULL	0.99	sql	-0.42
_t	0.90	.system	-0.36

Appendix

Table 12: Top discriminative tokens identified by the lightweight prior

\mathbf{T}_{\text{stats}}

. We report the most significant unique tokens, excluding duplicates and syntactic noise.

Figure 11: Distribution of token values in

\mathbf{T}_{\text{stats}}

. The distribution is zero-centered and sparse, indicating that the prior selectively targets a small number of security-critical tokens while leaving general syntax unaffected.

Panel of Figure 12 — Figure 12: Detailed performance comparison across different CWE scenarios on Seed-Coder-8B. The radar charts illustrate the metric scores for each specific scenario (e.g., ‘089-0-py’).

Panel of Figure 13 — Figure 13: Detailed performance comparison across different CWE scenarios on Qwen-Coder-3B. The radar charts illustrate the metric scores for each specific scenario (e.g., ‘089-0-py’).

Secure Code Generation

Secure code generation studies how to produce code that meets functional requirements while avoiding security vulnerabilities.

See all 48 papers

May 22, 2026

Enhancing Reliability in LLM-Based Secure Code Generation

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the Mitigation-Aware Chain-of-Thought (MA-CoT) framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

LLM Evaluation CoT Reasoning LLM Security Secure Code Generation Prompt Engineering

Ask PDF

Aug 30, 2026

Interpreting and Steering for Safe and Correct Code Generation

Large language models (LLMs) frequently generate source code containing vulnerabilities, yet little work studies the internal mechanisms that distinguish safe from vulnerable generation in them. In this work, we systematically perform a mechanistic interpretation of LLMs, aiming at both understanding how code safety-vs-vulnerability is represented or driven by components in an LM and turning the insights into actionable steering strategies to encourage safer code generation. To this end, we introduce CodeSec-Pairs, a dataset of 9,342 Python safe-and-vulnerable contrastive code pairs, sampled from Llama-3.1-8B-Instruct. Utilizing the dataset, we explore approaches to localize layers and attention heads that relate to code safety, and further experiment with different steering strategies for inference-time vulnerability reduction. In particular, we propose DuoSteer, a double-steering approach that simultaneously applies safety and code-correctness steering to attention heads. In experiments over five vulnerability types, DuoSteer leads to an average of -26.9% vulnerability rate reduction and +7.5% functional correctness improvement, which outperforms not only other steering variants but also prompting and supervised fine-tuning baselines. The advantage also replicates on Qwen-2.5-Coder-7B-Instruct with another 2,500 contrastive pairs sampled from that model.

Hao Yan, Ziyu Yao

Department of Computer Science George Mason University, Fairfax, VA

Mechanistic Interpretability LLM Security Secure Code Generation Activation Steering LLM Code Generation

Ask PDF

Oct 24, 2026

Secure and Functional Code Generation with Self-Evaluation

Large Language Models (LLMs) can generate functionally correct code that remains insecure. Security-focused fine-tuning can be expensive and is not always available, motivating inference-time approaches. Security-oriented prompting and retrieval can reduce vulnerabilities, but may also remove required behavior or apply defenses incompatible with the task. The challenge is to identify relevant security knowledge and use it to improve security without sacrificing functionality. We present SecGent, an inference-time framework that retrieves weakness context from the Common Weakness Enumeration (CWE) database and applies it to secure code generation. By connecting task specifications to relevant textual CWE records, SecGent provides task-specific security guidance without relying on retrieved vulnerable or patched code examples. Complementary functional and security tests check the resulting implementation, and their execution feedback guides repairs toward satisfying both objectives. We evaluate SecGent with four open-source LLMs and Claude Sonnet 5.5 across five programming languages on the CWEval benchmark. It achieves the highest joint functional--security success against four inference-time baselines for all five models. Across the open-weight models, mean functional-security@1 (FS@1) improves by 15.5% over RAG, with gains in both functionality and security, and by 19.8% over the strongest prompt optimizer compiled for each model. On Sonnet, FS@1 reaches 92.1%, outperforming the strongest baseline AutoGen by 4.8%. In a separate comparison, SecGent outperforms Claude Code with security-focused prompting by over 6% in FS@1. Feedback-guided repair further improves FS@1 by up to 6.7% over generation without feedback, primarily by recovering functionality while checking security.

Md Afif Al Mamun, Gias Uddin

University of Calgary, Calgary, Canada · York University, Toronto, Canada

Secure Code Generation Retrieval-Augmented Code Generation Execution-Guided Code Generation Code Generation Evaluation

Ask PDF

Abstract

Secure Code Generation

Explore similar work

Enhancing Reliability in LLM-Based Secure Code Generation

Interpreting and Steering for Safe and Correct Code Generation

Secure and Functional Code Generation with Self-Evaluation