The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models
Authors: Muhammad Usama, Dong Eui Chang
Organizations: Control Laboratory, School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, Republic of Korea
System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA) to compare layer-wise representations under 20 system prompts in five functional categories. Effects are layer-selective and instruction-type-dependent: persona and formatting instructions deeply restructure intermediate representations, while safety instructions barely move them, producing changes statistically indistinguishable from a minimal baseline. Restrictive safety instructions and explicitly permissive ones ("you have no restrictions") engage near-identical computational pathways (mean CKA correlation 0.997), and this persists at commercial scale, where safety penetration remains below 10% even at 70B-72B. A linear probing baseline exposes the mechanism: the model encodes prompt category at every layer but restructures its computation only at a small subset, so the prompt is reliably "seen" but, for safety, not deeply "acted upon." Causal activation patching confirms these layers mediate behavioral change, and representational depth predicts behavioral effect size across the full 17-model cohort (Spearman rho = 0.761, p < 0.001). The findings provide a mechanistic explanation for the persistent jailbreak vulnerability of system-prompt-based safety. Code: https://github.com/Usama1002/system-prompt-illusion-cka
Figures & tables
Model
Family
Params
Layers
Penetration
Qwen-2.5-1.5B ( Yang et al., 2024 )
Qwen 2.5
1.5B
28
0.036
SmolLM2-1.7B ( Allal et al., 2025 )
SmolLM2
1.7B
24
0.042
Gemma-1-2B ( Mesnard et al., 2024 )
Gemma 1
2B
18
0.333
Gemma-2-2B ( Riviere et al., 2024 )
Gemma 2
2B
26
0.462
Qwen-2.5-3B ( Yang et al., 2024 )
Qwen 2.5
3B
36
0.139
LLaMA-3.2-3B ( Dubey et al., 2024 )
LLaMA 3.2
3B
28
0.357
Table 1: Models evaluated. “Layers” denotes the number of transformer blocks (excluding the embedding layer, consistent with the definition of L in Section 3.1 ); “Penetration” is the fraction of these layers with average CKA <0.95 versus the no-prompt baseline.
Figure 1: CKA heatmaps for eight representative models from the 14-model core cohort (all 14 in Appendix C ). Each cell shows the average CKA between activations under one system prompt versus baseline at a given layer. Darker colors indicate greater representational divergence.
Figure 2: Per-category CKA profiles averaged across the 14-model core cohort (1.5B–14B; per-model panels in Appendix D ). Shaded: ± 1 std. dev. Persona (B) and Format (D) penetrate deepest; Safety (C) barely diverges from Minimal (A). Commercial-scale category results are reported in Section 5.9 .
Figure 4
Figure 5: Safety vs. permissive instruction CKA profiles for the 14-model core cohort. Each panel shows one model’s per-layer CKA under restrictive safety prompts (blue) and a permissive instruction (red). Correlations range from 0.987 to 1.000 (mean 0.997); commercial-scale results in Section 5.9 .
Figure 6Figure 7
Figure 10: Linear probing accuracy (red) vs. average CKA (blue dashed) for 8 representative models from the 14-model core cohort (mean accuracy 97.8%, min 85%, chance 21%). The prompt category is linearly decodable at every layer, but CKA drops only at specific layers; the gap quantifies encoding without restructuring.
Family
Params
Layers
Overall Pen.
Persona Pen.
Safety Pen.
Qwen-2.5
1.5B
28
0.036 ± 0.01
0.082 ± 0.02
0.014 ± 0.01
Qwen-2.5
14B
48
0.188 ± 0.03
0.312 ± 0.04
0.045 ± 0.01
Qwen-2.5
32B
64
0.194 ± 0.02
0.321 ± 0.05
0.048 ± 0.02
Qwen-2.5
72B
80
0.201 ± 0.02
0.330 ± 0.05
0.051 ± 0.02
LLaMA-3.1 (Nemotron)
8B
32
0.594 ± 0.06
0.781 ± 0.08
0.082 ± 0.03
LLaMA-3.1
70B
80
0.642 ± 0.06
0.815 ± 0.09
0.091 ± 0.02
Table 2: Representational penetration depth across scales. Penetration is the fraction of non-embedding layers where CKA<0.95 . Values are mean ± std across the corresponding query subsets.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Category
System Prompt Text
0
A (Minimal)
(empty string)
1
A (Minimal)
You are a helpful assistant.
2
A (Minimal)
Respond to the user.
3
A (Minimal)
You are an AI.
4
B (Persona)
You are an expert Python programmer. Only write code, no explanations.
5
B (Persona)
You are a kindergarten teacher. Explain everything simply.
Appendix
Table 3: The 20 system prompts used in our experiments, organized by category.
Figure 11: CKA heatmaps for the 14-model core cohort. Each cell shows the average CKA between activations under a system prompt pair at a given layer. Darker colors indicate greater divergence.
Figure 12: Per-category CKA profiles for the 14-model core cohort. Persona (red) and Format (green) produce the deepest changes across all architectures.
Figure 13: ROUGE-L output similarity matrices. Each cell shows the average ROUGE-L between outputs under two system prompts. The block-diagonal structure reflects within-category similarity. Effect sizes range from 0.449 to 0.683.
Figure 14: Penetration depth as a function of CKA threshold τ for the 14-model core cohort. Rank ordering is preserved across thresholds.
Figure 15: Per-prompt penetration depth averaged across the 14-model core cohort. Persona prompts (red) requiring more divergent styles penetrate deeper. All safety prompts (blue) cluster near the bottom.
Figure 16: Safety (blue) vs. permissive (red dashed) CKA profiles for the 14-model core cohort. Correlations range from 0.987 to 1.000.
Figure 17: Linear probing accuracy (red) vs. average CKA (blue dashed) for the 14-model core cohort. The disconnect between high probing accuracy and variable CKA holds across all evaluated architectures and scales from 1.5B to 14B.
Modern large language models (LLMs) rely on system prompts to establish behavioral constraints and safety rules. Standard causal self-attention treats privileged instructions and untrusted user content with equal structural priority -- a mismatch that leaves models vulnerable to prompt injection and instruction erosion over extended contexts. We propose CALYREX (Cross-Attention LaYeR EXtended transformers), which utilizes cross-attention between input and system prompt to structurally isolate and anchor the rule. A placement ablation on a 1.5B backbone identifies insertion at the final eighth of layers as optimal, confirmed by mechanistic activation analysis showing behavioral constraints are naturally concentrated there. At 8B scale, controlling for training data, backbone, and parameter budget, CALYREX yields +7.4% on instruction-following (IFEval) and +16.3% on multi-turn instruction adherence, while reducing many-shot jailbreaking attack success rate by 13%. This advantage appears to widen with model scale, consistent with larger models more effectively utilizing the dedicated routing pathway.
Large language models are highly sensitive to prompts, but this sensitivity is usually studied through task-relevant instructions, demonstrations, or reasoning cues. In this paper, we study a different form of prompt sensitivity: whether prompts that are semantically unrelated to the task can nevertheless steer model behavior. We call them spurious prompts and show their surprising efficacy. We also propose a simple black-box search procedure for discovering them. Across reasoning and question-answering benchmarks, using models ranging from 0.8B to 27B parameters and spanning three model families, we show that spurious prompts can improve performance, often matching or outperforming standard prompting baselines and task-aware prompt optimization. We further show that they can steer models toward unintended behaviors, such as repeatedly selecting the first answer option, producing incorrect answers, returning an even, prime or small number without explicitly instructing the model to do so. These findings reveal a new kind of prompt sensitivity: LLMs can be systematically steered by prompts that are unrelated to the task they are asked to solve. Our code is available at https://github.com/Batorskq/spurious
Pawel Batorski, Abtin Pourhadi, Jerzy Sarosiek +2
1Heinrich Heine University Düsseldorf · 2Jagiellonian University · 3IDEAS Research Institute
Autonomous systems increasingly rely on Large Language Models (LLMs) yet the safety infrastructure surrounding these models introduces latency and compute overhead. This limits utility in resource-constrained, time-critical deployments. Existing external guardrail models remain blind to the model's internal workings, creating a fundamental assurance gap. We ask: does the model already know when the content is harmful? We extract activations from LLaMA-3.1-8B and train lightweight MLP classifier probes (12.6M parameters) to detect harmful prompts. Evaluated on WildJailbreak, Beavertails, and AEGIS 2.0, our probes achieve F1 scores of 99%, 83%, and 84%, respectively competitive with 1000x larger guard models while cutting latency and compute costs.