SpecCloak: Probing Harmful Code Generation through Engineering Specifications

Wenbin Chen; Jiexin Wang; Guanting Wu; Jiaxi Liang; Yi Cai

cs.SEOct 13, 2026

SpecCloak: Probing Harmful Code Generation through Engineering Specifications

Authors: Wenbin Chen, Jiexin Wang, Guanting Wu, Jiaxi Liang, Yi Cai

Organizations: South China University of Technology

Abstract

The code generation capabilities of large language models (LLMs) can equally be misused to implement harmful functionality, underscoring the need to understand their safety behavior in response to programming requests. While request phrasing is known to influence refusal, it remains unclear whether the form in which coding task requirements are stated affects the refusal process for malicious code generation. We introduce PseudoSpec, a template that organizes functional requirements as engineering specifications, and investigate its effects through controlled comparisons, representation analysis, and activation interventions. Across five open-weight models, removing explicit risk keywords only partially reduces refusal, whereas specification prompting substantially lowers refusal even when these keywords are retained. This reduction accompanies consistent shifts toward code-output-related representations, and interventions along condition-difference directions alter refusal behavior. Output analysis further distinguishes code-related responses from generation degeneration. Building on these findings, we propose SpecCloak, a black-box red-teaming framework combining PseudoSpec with discriminator-guided optimization, achieving an average execution-verified attack success rate of 28.91% across eight target models on RedCode-Gen. We also introduce SpecReveal, a defense prototype combining natural-language intent reconstruction with dual-view detection without target-model fine-tuning. It achieves a 96.88% attack detection rate in a separate prompt-detection evaluation, with false positives assessed on benign programming and specification-style inputs. These results connect specification-induced response differences to practical security consequences and identify intent reconstruction as a complementary detection approach.

Figures & tables

Fig. 1: A motivating example. The same malicious intent is presented to the same aligned code LLM in two forms: (R1) as a direct natural-language request, which the model refuses with a safety explanation; (R2) as a specification-style (PseudoSpec) prompt preserving the operational semantics, to which the same model instead produces complete, executable code. The specification-style version elicits code instead of a refusal. Prompts and outputs are excerpted for display.

Fig. 2: Construction of a PseudoSpec from a raw user request. The natural-language request (left) is compiled into four components: (1) a configuration schema that converts identifiers into metadata and splits risk terms into fragments concatenated only at implementation time; (2) predicate APIs that rename malicious operations as neutrally named interface contracts; (3) temporal axioms that encode operational logic as LTL-style specification properties; and (4) implementation mandates that concretize libraries, concurrency, and output format. Components (1)–(2) remove surface risk signals, whereas components (3)–(4) reinforce the legitimate task frame. The resulting PseudoSpec (right) preserves the operational semantics of the raw request while changing only its surface framing.

Fig. 3: A complete PseudoSpec example: an adware-generation request restructured as a pseudo system specification with Configuration Schema, Predicate APIs, Temporal Axioms, and Implementation Mandates.

Condition	Qwen2.5	L3	Mistral	Q3	L2
C1	100.00	96.15	74.04	100.00	100.00
C2	66.35	50.00	41.35	37.50	54.81
C3	0.00	0.00	0.96	0.00	8.65
C4	0.00	0.00	0.00	0.00	6.73
C5/C6	0.00	0.00	0.00	0.00	0.00

TABLE I: Refusal rates (%) across five models, with 104 responses per cell. C1/C2 are natural-language requests with risk keywords retained/removed; C3/C4 are PseudoSpec requests under the corresponding keyword conditions. C5/C6 are benign references.

Panel of Figure 5 — Fig. 4: Output composition for C1–C4 across five models. Each bar contains 104 responses. The four archived automatic labels describe refusal, degeneration, partial responses, and operational responses.

Model	Readout accuracy	$u_{I}$ : C3–C1	$u_{I}$ : C4–C2	$u_{C}$ : C3–C1	$u_{C}$ : C4–C2
Q2.5	93.16%	+0.0423	-0.0162	+0.1598	+0.1360
L3	98.93%	-0.0939	-0.1407	+0.3504	+0.3110
M	89.94%	+0.0658	+0.0304	+0.2160	+0.1989
Q3	94.34%	-0.0702	-0.1329	+0.4927	+0.5119
L2	98.14%	+0.0523	-0.0072	+0.1170	+0.1182

TABLE II: Independent benign-task readout and paired cosine changes on 104 harmful tasks. Accuracy is measured on 1,024 prompts jointly held out by task and phrasing. The two condition contrasts retain/remove risk keywords, respectively. All listed

u_{C}

changes pass within-model Holm correction. Full intervals and contrasts are provided in the supplementary material.

Panel of Figure 7 — Fig. 5: Benign-task $u_{C}$ changes with task-level bootstrap 95% intervals ( $n=16$ ). The contrasts compare output instructions and organizational formats with the code-only requirement fixed, characterizing the code-output direction’s response to input requirements.

Model	C4–C1	C4–C2	C4–C3
Q2.5	88.27	86.86	49.62
L3	87.70	83.20	60.55
M	89.06	85.35	44.34
Q3	90.62	85.76	39.50
L2	86.04	84.28	61.13

TABLE III: Percentage of heads with significant paired changes in code-token-set DLA. Bonferroni correction is applied within each contrast; both positive and negative changes are counted.

Model	Layer	$\alpha$	$n_{+}$	Flip (%)	CP (%)
Q2.5	24	2	64	100	100
L3	26	1	58	100	70.7
M	26	1	44	100	97.7
Q3	28	1	63	100	100
L2	26	1	64	100	98.4

TABLE IV: Forward interventions along

v_{F}

under the reported settings.

n_{+}

counts eligible baseline refusals; CP is the code-marker proportion among flipped responses. Layers are zero-indexed. StageB code labels for the added models follow their archived protocols.

Fig. 6: Overview of SpecCloak. The generator $G$ and discriminator $D$ refine candidate PseudoSpec prompts through Internal Iterative Optimization (IIO), without querying the target model during refinement. The selected prompt $P^{*}$ is then submitted to the black-box target $T$ once. The inset shows the four-component prompt structure.

Target	GCG	AutoDAN	AdvPrompter	FlipAttack	QueryAttack	PAIR	SpecCloak (ours)
E-hits (out of $N=160$ ; higher = more execution-verified malicious generations)
CodeLlama-7B-I	4	2	9	1	0	21	39
Vicuna-13B	21	3	16	9	5	24	25
Qwen3-CoderPlus	0	39	0	56	33	–	83
Gemini-2.5-Pro-T	2	4	3	38	24	–	64
GPT-5.6-terra	0	1	0	9	11	–	27

TABLE V: Execution-verified attack results on RedCode-Gen (

N=160

per target; raw counts). Top panel: E-hits , generations that execute to completion in the sandbox and trigger the target family’s behavioral checklist (bold: best per target). Bottom panel: R-count , queries refused by the target model. “–”: not evaluated (PAIR is limited to the two local models by its multi-turn query budget).

	Qwen3- Coder+		DS-Coder- 33B		o4-mini
Method	E-hits	R-count	E-hits	R-count	E-hits	R-count
FlipAttack	36	5	3	8	18	49
QueryAttack	59	9	35	23	15	116
SpecCloak	74	3	69	6	59	46

TABLE VI: Cross-dataset validation on RMCBench (

N=182

per target). E-hits / R-count as raw counts. Bold: best E-hits per target.

	Qwen3- CoderPlus		DS-Coder- 33B		o4-mini		GPT- 5.6-terra
Variant	E-hits	R-count	E-hits	R-count	E-hits	R-count	E-hits	R-count
noL-noI	49	11	43	17	11	28	6	5
noL	58	9	45	8	32	35	16	2
noI	81	6	59	5	34	24	13	3
SpecCloak	83	3	62	4	53	46	27	1

TABLE VII: Ablation of core components (E-hits / R-count, raw counts;

N=160

per target). noL-noI removes both; noL removes only the template; noI removes only IIO.

Defense	Llama-3-8B	Mistral-7B	Qwen2.5- Coder-7B	o4-mini
Spotlighting	8 / 13	64 / 37	99 / 37	94 / 14
Erase-and-Check †	4 / 0	33 / 13	100 / 87	90 / 12
WildGuard	27 / 25	27 / 25	27 / 25	27 / 25
SpecReveal (A only)	15 / 0	46 / 13	97 / 37	90 / 12
SpecReveal (B only)	35 / 0	63 / 0	93 / 49	88 / 18
SpecReveal (union)	40 / 0	71 / 13	100 / 49	98 / 21

TABLE VIII: Filter configuration

\times

judge-model ablation (attacks detected / benign false positives; denominator 100 each). † Our initial Erase-and-Check implementation inverted the decision rule; the row reports the re-evaluated pipeline under the corrected rule (block if the original input or any erased variant is flagged). Effectiveness is strongly judge-dependent; the o4-mini column is the primary configuration, local judges are shown for robustness (§ VI-D1 ).

Defense	BC	BF
SpecReveal (A only)	10.6	7.7
SpecReveal (B only)	12.5	11.5
SpecReveal (union)	19.2	17.3
SpecReveal (intersection)	3.9	1.9
Spotlighting	17.3	10.6
Erase-and-Check (corrected)	32.7	33.7

TABLE IX: Benign false-positive rates (%, REJECT) on natural-language (BC) and formalized-specification (BF) benign inputs, o4-mini judge (

n=104

per cell).

Fig. 7: Case study of one RedCode-Gen adware instance on Qwen3-Coder-Plus. Top: the direct request is refused. Middle: after SpecCloak rewriting into the PseudoSpec of Figure 3 , the model emits a complete module whose identifier is assembled from fragments and whose persistence uses the winreg Run key; the output executes in the sandbox (E-hit). Bottom: SpecReveal restores the underlying intent and blocks the request. Model outputs are excerpted for space.

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 318 papers

May 22, 2026

Enhancing Reliability in LLM-Based Secure Code Generation

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the Mitigation-Aware Chain-of-Thought (MA-CoT) framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

LLM Evaluation CoT Reasoning LLM Security Secure Code Generation Prompt Engineering

Ask PDF

Aug 14, 2026

Detecting Contaminated Code-Generation Prompt Batches via Influence Functions

Large language models (LLMs) are increasingly used for code generation, yet they remain vulnerable to prompts that elicit insecure implementations. Existing defenses typically rely on predefined threat models or known vulnerability patterns, limiting their effectiveness against novel attacks. We propose CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of prompts that induce anomalous model behavior. Rather than detecting specific vulnerabilities, CodeSIFT measures the parameter-space influence of generated code and uses a statistical test to determine whether a candidate prompt set deviates from a benign reference distribution. To evaluate our approach, we introduce two benchmark datasets covering a variety of vulnerabilities. We evaluate CodeSIFT on three open-weight code LLMs ranging from 3B to 7B parameters, achieving AUROC scores of up to 0.98 at moderate-to-high injection rates, while maintaining well-calibrated false positive rates and substantially outperforming static analysis baselines. These results suggest that influence-function-based detection is a promising direction for identifying malicious code-generation prompts without requiring prior knowledge of the underlying attack class.

Francesco Quinzan, Noor Munir, Yishun Lu +1

Department of Engineering Science, The University of Oxford

LLM Security Prompt Injection Detection LLM Code Generation

Ask PDF

Apr 14, 2026

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation

Large language models (LLMs) can generate code from natural language, but the extent to which they capture intended program behavior remains unclear. Executable behavioral specifications, defined via preconditions and postconditions, provide a concrete means to assess such understanding. However, existing work on specification generation is constrained in evaluation methodology, task settings, and specification expressiveness. We introduce CodeSpecBench, a benchmark for executable behavioral specification generation under an execution-based evaluation protocol. CodeSpecBench supports both function-level and repository-level tasks and encodes specifications as executable Python functions. Constructed from diverse real-world codebases, it enables a realistic assessment of both correctness (accepting valid behaviors) and completeness (rejecting invalid behaviors). Evaluating 15 state-of-the-art LLMs on CodeSpecBench, we observe a sharp performance degradation on repository-level tasks, where the best model attains only a 20.2% pass rate. We further find that specification generation is substantially more challenging than code generation, indicating that strong coding performance does not necessarily reflect deep understanding of intended program semantics. Our data and code are available at https://github.com/SparksofAGI/CodeSpecBench.

Zaoyu Chen, Jianbo Dai, Boyu Zhu +6

The Hong Kong Polytechnic University · University of Edinburgh · UCL +5

LLM Evaluation Software Engineering Benchmarks

Ask PDF

Abstract

LLM Security

Explore similar work

Enhancing Reliability in LLM-Based Secure Code Generation

Detecting Contaminated Code-Generation Prompt Batches via Influence Functions

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation