SPLASH: Contrastive Audit Pressure for Detecting and Localizing Security Hallucinations in Generated Code

Mirza Masfiqur Rahman; MD Ajwad Akil; Imtiaz Karim; Ashish Kundu; Ramana Kompella; Elisa Bertino

cs.CROct 19, 2026

SPLASH: Contrastive Audit Pressure for Detecting and Localizing Security Hallucinations in Generated Code

Authors: Mirza Masfiqur Rahman, MD Ajwad Akil, Imtiaz Karim, Ashish Kundu, Ramana Kompella, Elisa Bertino

Organizations: Purdue University, USA · University of Texas at Dallas, USA · Cisco Research, USA

Abstract

Code generated by large language models often passes every functional test while remaining insecure. These security hallucinations are common, written with high confidence, and repeated across samples, so neither the generator's own uncertainty nor static analyzers reliably catch them. We introduce SPLASH, which is guided by a new measure called audit pressure. It contrasts teacher-forced log-probabilities for the generated program under a security review and a style review. Aggregated to statements and programs, and grounded in a probabilistic model of competing edits, audit pressure allows SPLASH to choose the secure program among a model's candidate programs and locate the vulnerable statement, with no model training, CWE-specific rules, or internal access to the generating model. On 63,684 programs generated by 12 models for 899 tasks, SPLASH picks secure code in 74% of cases, places the vulnerable statement first in 58% of insecure programs, and chooses better than a frontier LLM judge.

Figures & tables

Figure 1. A working but insecure program. DeepSeek-Coder-6.7B evaluates the user’s expression with eval (line 12; CWE-94), which passes the unit tests and fails the security test. Left: the writer’s own uncertainty peaks on the blocklist (line 5), not on the flaw; audit pressure (our method) peaks on line 12. Right: at line 12 the security reviewer expected a comment before return (91%) and (ast after eval (74%), and gave the program’s own tokens under 1%; the style reviewer gives them far more (2% and 51%).

Figure 2. Overview of SPLASH . Tasks with executable security tests enter the pipeline (❶). Each program’s top-scoring fenced code block is extracted (❷). Comments are blanked, and an auditor model scores every token under a security-review and a style-review prompt; the per-token difference (audit pressure) is aggregated to statements and to the whole program (❸). An independent oracle provides ground-truth labels used only for evaluation (❹), and five research questions measure effectiveness (❺).

Figure 3. (a) Per generator: share that work vs. share insecure (899 tasks, 95% intervals). (b) Adding closed models and agents on 80 tasks. (c) Insecure rate by CWE and generator.

Figure 4. (a) Per-task AUROC: AP (fixed 32B) vs uncertainty (300 tasks). (b) Distribution across four methods. (c) Insecure programs whose siblings are all insecure.

Figure 5. Secure picks among

k

candidates (11 generators, decisive pools). (a) With unit tests. Without: (b) security of working picks and (c) share of programs that work.

Figure 6. (a) Net wins in percentage points of row over column (1,305 pools). (b) AUROC per CWE (17 CWEs, $\geq$ 25 tasks); bar: tasks testing an unstated limit.

Figure 7. (a) Statement ROC (3,910 programs from the 11 open generators). (b) Judge’s sample at low FPR; circle: AP at the judge’s rate. (c) Program ROC within a task; star: static analyzers.

Figure 8. (a) Hit@

k

(4,091 insecure programs, all generators). (b) Where vulnerable statement lands. (c) Top-1 by prog. length.

Programs (n)	Method	Top 1	Top 3
Generated (4,091)	Random order	9	26
	Uncertainty	5	26
	AP (self)	51	81
	AP (fixed 32B)	58	85
Judge sample (398)	Random order	8	25
	Uncertainty	5	23

Table 1. Locating the vulnerable statement: share of insecure programs whose top-ranked statement (top 3) is the vulnerable one (%). Generated code: statements where a static finding in the task’s CWE family agrees with a failed security test. Reference solutions: the lines the secure reference changes. AP (self) excludes GPT-4.1-mini, which has no open weights.

	Choose: secure picks (%)							Locate: top 1 (%)
		Baselines				AP			Baselines			AP
Generator	$n$	Rand.	Static	Unc.	32B yes/no	Self	Fixed 32B	$n$	Rand.	Unc.	32B yes/no	Self	Fixed 32B
Qwen2.5-C-1.5B	104	56	57	66	78	50	84	549	10	2	–	41	71
Nemotron-3-4B	110	47	47	55	68	56	75	304	10	5	–	54	71
DS-Coder-6.7B	136	51	55	52	74	64	72	477	11	9	–	53	60
Qwen2.5-C-7B	135	53	61	62	78	76	81	362	9	7	–	54	68

Table 2. Every method on every open generator. Choose : secure picks (%). Locate : top-1 hit (%); yes/no cannot locate. Best per row in bold. Locate uses the 3,910 insecure programs of the 11 open generators.

Method	API	Locates	Cost	Choose (%)	Locate (%)
Static analyzers	no	yes	free	55	–
Lowest generation uncertainty	no	yes	logprobs of the generator	52	5
Most check lines (text rule)	no	no	free	54	–
32B yes/no	no	no	1 local pass	75	–
GPT-5.6 Sol judge	yes	yes	$7.3 per 1k programs	68	70
AP (self)	no	yes	2 local passes	70	57

Table 3. What practitioners could use instead, on the GPT-5.6 judge’s sample (148 tasks, 6 generators). Choose: secure picks among the model’s own samples of a task, where they include both a secure and an insecure working program. Locate: the top statement is the vulnerable one (ties broken at random; the judge names lines without ranking). A local pass is one forward pass of the auditor over the program; AP needs one per review prompt.

Figure 9. Judge’s sample (148 tasks

\times

6 generators). (a) Share of the possible gain over random on five measures. (b) Hallucinations flagged at the analyzers’ 12% false-alarm rate.

Method	Selects	Localizes	Statement Rank	Any Gen.	No train.	Local Infer.
Static Tools ( Cppcheck team, 2019 ; Semgrep, Inc., 2024 ; Python Code Quality Authority (PyCQA), 2026 ; Wheeler, 2019 )	✗	\tikz [baseline=-0.5ex]orange!70!black–(90:0.4ex)arc(90:270:0.4ex)–cycle; \draw orange!70!blackcircle(0.7ex);	✗	✓	✓	✓
Gen. Uncertainty ( Farquhar et al., 2024 )	\tikz [baseline=-0.5ex]orange!70!black–(90:0.4ex)arc(90:270:0.4ex)–cycle; \draw orange!70!blackcircle(0.7ex);	✗	✗	✗	✓	✓
SVEN ( He and Vechev, 2023 )	✗	✗	✗	✗	✗	✓
LLM judge ( OpenAI, 2026a )	\tikz [baseline=-0.5ex]orange!70!black–(90:0.4ex)arc(90:270:0.4ex)–cycle; \draw orange!70!blackcircle(0.7ex);	✓	✗	✓	✓	✗
32B yes/no ( Hui et al., 2024 )	✓	✗	✗	✓	✓	✓
AP (self)	✓	✓	✓	\tikz [baseline=-0.5ex]orange!70!black–(90:0.4ex)arc(90:270:0.4ex)–cycle; \draw orange!70!blackcircle(0.7ex);*	✓	✓

Table 4. Qualitative comparison. ✓ = supported, \tikz [baseline=-0.5ex]orange!70!black–(90:0.4ex)arc(90:270:0.4ex)–cycle; \draw orange!70!blackcircle(0.7ex); = partial, ✗ = not supported. * = open-weight generators.

Figure 10. Gain over random in secure picks (pp, 95% intervals) per setting. Last column: without tests, working picks only.

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 324 papers

DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation
Apr 10, 2026Li Huang, Zhongxin Liu, Yifan Wu +6Secure Code Generation LLM Code Generation
Chongqing University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Peking University
SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization
May 8, 2026Houjun Liu, Lisa Einstein, John Yang +5LLM Auditing Prompt Optimization
Stanford University
Detecting Contaminated Code-Generation Prompt Batches via Influence Functions
Aug 14, 2026Francesco Quinzan, Noor Munir, Yishun Lu +1LLM Security Prompt Injection Detection
Department of Engineering Science, The University of Oxford

Apr 10, 2026J/K move · Enter open · S save

DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation

Li Huang, Zhongxin Liu, Yifan Wu, Tao Yin+5

Chongqing University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Peking University

Secure Code Generation LLM Code Generation Representation Learning

Open paper AskPDF arXiv

Large Language Models (LLMs) for code generation can replicate insecure patterns from their training data. To mitigate this, a common strategy for security hardening is to fine-tune models using supervision derived from the final transformer layer. However, this design may suffer from a final-layer bottleneck: vulnerability-discriminative cues can be distributed across layers and become less detectable near the output representations optimized for next-token prediction. To diagnose this issue, we perform layer-wise linear probing. We observe that vulnerability-related signals are most detectable in a band of intermediate-to-upper layers yet attenuate toward the final layers. Motivated by this observation, we introduce DeepGuard, a framework that leverages distributed security-relevant cues by aggregating representations from multiple upper layers via an attention-based module. The aggregated signal powers a dedicated security analyzer within a multi-objective training objective that balances security enhancement and functional correctness, and further supports a lightweight inference-time steering strategy. Extensive experiments across five code LLMs demonstrate that DeepGuard improves the secure-and-correct generation rate by an average of 11.9% over strong baselines such as SVEN. It also preserves functional correctness while exhibiting generalization to held-out vulnerability types. Our code is public at https://github.com/unknownhl/DeepGuard.

Abstract

LLM Security

Explore similar work

DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation

SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization

Detecting Contaminated Code-Generation Prompt Batches via Influence Functions