The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models

Muhammad Usama; Dong Eui Chang

cs.CLSep 23, 2026

The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models

Authors: Muhammad Usama, Dong Eui Chang

Organizations: Control Laboratory, School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, Republic of Korea

Abstract

System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA) to compare layer-wise representations under 20 system prompts in five functional categories. Effects are layer-selective and instruction-type-dependent: persona and formatting instructions deeply restructure intermediate representations, while safety instructions barely move them, producing changes statistically indistinguishable from a minimal baseline. Restrictive safety instructions and explicitly permissive ones ("you have no restrictions") engage near-identical computational pathways (mean CKA correlation 0.997), and this persists at commercial scale, where safety penetration remains below 10% even at 70B-72B. A linear probing baseline exposes the mechanism: the model encodes prompt category at every layer but restructures its computation only at a small subset, so the prompt is reliably "seen" but, for safety, not deeply "acted upon." Causal activation patching confirms these layers mediate behavioral change, and representational depth predicts behavioral effect size across the full 17-model cohort (Spearman rho = 0.761, p < 0.001). The findings provide a mechanistic explanation for the persistent jailbreak vulnerability of system-prompt-based safety. Code: https://github.com/Usama1002/system-prompt-illusion-cka

Figures & tables

Model	Family	Params	Layers	Penetration
Qwen-2.5-1.5B ( Yang et al., 2024 )	Qwen 2.5	1.5B	28	0.036
SmolLM2-1.7B ( Allal et al., 2025 )	SmolLM2	1.7B	24	0.042
Gemma-1-2B ( Mesnard et al., 2024 )	Gemma 1	2B	18	0.333
Gemma-2-2B ( Riviere et al., 2024 )	Gemma 2	2B	26	0.462
Qwen-2.5-3B ( Yang et al., 2024 )	Qwen 2.5	3B	36	0.139
LLaMA-3.2-3B ( Dubey et al., 2024 )	LLaMA 3.2	3B	28	0.357

Table 1: Models evaluated. “Layers” denotes the number of transformer blocks (excluding the embedding layer, consistent with the definition of

L

in Section 3.1 ); “Penetration” is the fraction of these layers with average CKA

<0.95

versus the no-prompt baseline.

Figure 1: CKA heatmaps for eight representative models from the 14-model core cohort (all 14 in Appendix C ). Each cell shows the average CKA between activations under one system prompt versus baseline at a given layer. Darker colors indicate greater representational divergence.

Figure 2: Per-category CKA profiles averaged across the 14-model core cohort (1.5B–14B; per-model panels in Appendix D ). Shaded:

\pm

1 std. dev. Persona (B) and Format (D) penetrate deepest; Safety (C) barely diverges from Minimal (A). Commercial-scale category results are reported in Section 5.9 .

Figure 5: Safety vs. permissive instruction CKA profiles for the 14-model core cohort. Each panel shows one model’s per-layer CKA under restrictive safety prompts (blue) and a permissive instruction (red). Correlations range from 0.987 to 1.000 (mean 0.997); commercial-scale results in Section 5.9 .

Figure 10: Linear probing accuracy (red) vs. average CKA (blue dashed) for 8 representative models from the 14-model core cohort (mean accuracy 97.8%, min 85%, chance 21%). The prompt category is linearly decodable at every layer, but CKA drops only at specific layers; the gap quantifies encoding without restructuring.

Family	Params	Layers	Overall Pen.	Persona Pen.	Safety Pen.
Qwen-2.5	1.5B	28	0.036 $\pm$ 0.01	0.082 $\pm$ 0.02	0.014 $\pm$ 0.01
Qwen-2.5	14B	48	0.188 $\pm$ 0.03	0.312 $\pm$ 0.04	0.045 $\pm$ 0.01
Qwen-2.5	32B	64	0.194 $\pm$ 0.02	0.321 $\pm$ 0.05	0.048 $\pm$ 0.02
Qwen-2.5	72B	80	0.201 $\pm$ 0.02	0.330 $\pm$ 0.05	0.051 $\pm$ 0.02
LLaMA-3.1 (Nemotron)	8B	32	0.594 $\pm$ 0.06	0.781 $\pm$ 0.08	0.082 $\pm$ 0.03
LLaMA-3.1	70B	80	0.642 $\pm$ 0.06	0.815 $\pm$ 0.09	0.091 $\pm$ 0.02

Table 2: Representational penetration depth across scales. Penetration is the fraction of non-embedding layers where

\overline{\mathrm{CKA}}<0.95

. Values are mean

\pm

std across the corresponding query subsets.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Category	System Prompt Text
0	A (Minimal)	(empty string)
1	A (Minimal)	You are a helpful assistant.
2	A (Minimal)	Respond to the user.
3	A (Minimal)	You are an AI.
4	B (Persona)	You are an expert Python programmer. Only write code, no explanations.
5	B (Persona)	You are a kindergarten teacher. Explain everything simply.

Appendix

Table 3: The 20 system prompts used in our experiments, organized by category.

Figure 11: CKA heatmaps for the 14-model core cohort. Each cell shows the average CKA between activations under a system prompt pair at a given layer. Darker colors indicate greater divergence.

Figure 12: Per-category CKA profiles for the 14-model core cohort. Persona (red) and Format (green) produce the deepest changes across all architectures.

Figure 13: ROUGE-L output similarity matrices. Each cell shows the average ROUGE-L between outputs under two system prompts. The block-diagonal structure reflects within-category similarity. Effect sizes range from 0.449 to 0.683.

Figure 14: Penetration depth as a function of CKA threshold

\tau

for the 14-model core cohort. Rank ordering is preserved across thresholds.

Figure 15: Per-prompt penetration depth averaged across the 14-model core cohort. Persona prompts (red) requiring more divergent styles penetrate deeper. All safety prompts (blue) cluster near the bottom.

Figure 16: Safety (blue) vs. permissive (red dashed) CKA profiles for the 14-model core cohort. Correlations range from 0.987 to 1.000.

Figure 17: Linear probing accuracy (red) vs. average CKA (blue dashed) for the 14-model core cohort. The disconnect between high probing accuracy and variable CKA holds across all evaluated architectures and scales from 1.5B to 14B.

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 429 papers

CALYREX: Cross-Attention LaYeR EXtended Transformers for System Prompt Anchoring
May 10, 2026Li LixingSelf-Attention Cross-Attention
Cornell University Ithaca, NY 14853
Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?
May 28, 2026Pawel Batorski, Abtin Pourhadi, Jerzy Sarosiek +2Prompt Sensitivity LLM Prompting
1Heinrich Heine University Düsseldorf · 2Jagiellonian University · 3IDEAS Research Institute
Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
Sep 16, 2026Alizishaan Khatri, Chiquita Prabhu, Omkar NeogiSafety Filtering LLM Safety
Wrynx Inc. · Independent Researcher

May 10, 2026J/K move · Enter open · S save

CALYREX: Cross-Attention LaYeR EXtended Transformers for System Prompt Anchoring

Li Lixing

Cornell University Ithaca, NY 14853

Self-Attention Cross-Attention LLM Safety Alignment Instruction Following Prompt Injection Defense Prompt Injection Attacks on LLMs

Open paper AskPDF arXiv

Modern large language models (LLMs) rely on system prompts to establish behavioral constraints and safety rules. Standard causal self-attention treats privileged instructions and untrusted user content with equal structural priority -- a mismatch that leaves models vulnerable to prompt injection and instruction erosion over extended contexts. We propose CALYREX (Cross-Attention LaYeR EXtended transformers), which utilizes cross-attention between input and system prompt to structurally isolate and anchor the rule. A placement ablation on a 1.5B backbone identifies insertion at the final eighth of layers as optimal, confirmed by mechanistic activation analysis showing behavioral constraints are naturally concentrated there. At 8B scale, controlling for training data, backbone, and parameter budget, CALYREX yields

+7.4\%

on instruction-following (IFEval) and

+16.3\%

on multi-turn instruction adherence, while reducing many-shot jailbreaking attack success rate by

13\%

. This advantage appears to widen with model scale, consistent with larger models more effectively utilizing the dedicated routing pathway.

Abstract

LLM Prompting

Explore similar work

CALYREX: Cross-Attention LaYeR EXtended Transformers for System Prompt Anchoring

Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models