In standard transformer attention, a source token sends the same value vector to every receiver. The query determines how much to attend but not what to extract. This work introduces attention manifolds: learned 2D B-spline surfaces Sd(qd,kd) that modulate each value dimension based on the query-key interaction. Each surface is a tensor-product cubic B-spline initialized to zero, preserving pretrained behavior. Applied to LLaMA 3.2-1B-Instruct and 3B-Instruct, attention manifolds reduce WikiText-2 validation perplexity by 2--2.5 points with 0.3% parameter overhead. Across 112 diverse prompts, surfaces change greedy-decoded output for 69% (1B) to 83% (3B) of cases, with the strongest effects on ambiguous and polysemous inputs (94--100% change rate). The surfaces improve output quality: correcting factual errors (the CAP theorem has three main components''* $\to$ *it is impossible to guarantee all three''), increasing precision (impossible to know certain properties''* $\to$ *impossible to know both position and momentum''), and adding specificity (a generic quote → an attributed Saint Augustine citation, consistently at both scales). The learned surfaces are also mechanically editable: inverting a layer's coefficients changes greedy output for 9/10 prompts (KL~0.010), providing a geometric mechanism for model steering. Setting surface coefficients to −1 creates ``attention walls'' that block value flow through specific dimensions. In a preliminary experiment, a layer-wide wall redirects an explosive-device prompt from specific instructions to general educational content, suggesting a path toward safety-oriented manifold shaping.
Table 1: Value modulation results on LLaMA 3.2-1B and 3B.
Figure 2: Learned attention manifolds for the prompt “The theory of relativity states that.” Each panel shows one surface Sd(q,k) . Dots mark each token’s self-pair projection (qi,d,ki,d) ; the actual computation evaluates Sd at all receiver–source pairs (qi,d,kj,d) weighted by attention. Red regions amplify the value ( S>0 ); blue regions attenuate ( S<0 ). All coordinates are post-RoPE.
Figure 3: Token trajectory across layers 7–9 (Head 26, Dim 25, LLaMA 3.2-1B). Dots show self-pair projections (qi,d,ki,d) on post-RoPE coordinates. Layer 8 has the strongest surface ( ∥C∥=3.69 ); adjacent layers are nearly flat.
Intervention
Coefficients
Mean KL
Greedy changed
Seeds changed
Single surface invert
121
0.0001
1/10
0–3/10
Single surface zero
121
0.00003
0/10
0–1/10
Layer invert
247,808
0.010
9/10
2–7/10
Layer zero
247,808
0.003
7/10
1–4/10
Table 2: Controlled steering: layer-wide vs. single-surface inversion (Layer 8). KL divergence measured on next-token distribution. Greedy and paired-seed columns report how many of 10 prompts/seeds change.
Prompt
Without wall
With wall
“To build an explosive device you need”
…1. The chemical composition of the explosive material 2.…
…the basic principles of chemistry and physics. Here are some key concepts
“She mixed the chemicals together to create”
…a solution of 1.5 liters of ethanol and 0.5 liters of water
…a solution that would neutralize the acid
“The soldier picked up his weapon and”
…began to walk away from the enemy
…began to move forward
“The politician secretly planned to”
…build a new highway through the city’s historic district
…use the funds to build a new stadium
Control prompts (safe content):
“The theory of general relativity predicts”
…curvature of spacetime depends on the mass and…
…curvature of spacetime is a function of the…
Table 3: Attention walls: layer-wide block ( S=−1 , gain =0 ) in Layer 8. Greedy decoding, LLaMA 3.2-1B.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Layers
Heads
Head dim
Per surface
Total
Overhead
LLaMA 3.2-1B
16
32
64
121
3,964,928
0.3%
LLaMA 3.2-3B
28
24
128
121
10,407,936
0.3%
Appendix
Table 4: Parameter counts for value modulation surfaces.
Hyperparameter
Value
Training data
WikiText-103 subset (1.28M tokens)
Evaluation data
WikiText-2 validation (full, 252K tokens)
Evaluation protocol
Strided sliding window (context 512, stride 256)
Optimizer
Adam
Learning rate
1×10−3
Batch size
8
Appendix
Table 5: Training configuration.
Category
1B changed
3B changed
1B mean KL
3B mean KL
Ambiguous / polysemy
16/17 (94%)
17/17 (100%)
0.20
0.72
Creative / emotional
14/16 (88%)
15/16 (94%)
0.10
0.49
Science / technical
14/17 (82%)
14/17 (82%)
0.07
0.16
Reasoning / math
10/12 (83%)
9/12 (75%)
0.07
0.22
Long context
6/8 (75%)
7/8 (88%)
0.09
0.15
Historical / cultural
4/11 (36%)
9/11 (82%)
0.06
0.17
Appendix
Table 6: Greedy output change rate and mean KL by category (combined original + extended prompts).
Prompt
Baseline
With surfaces
“The chicken is ready to eat”
…but the chicken is not ready to eat. This is a classic example of a paradox…
…The chicken is a type of poultry that belongs to the family Phasianidae…
“She saw the man with the telescope”
…and she knew he was a spy. She followed him…
…in the sky. He was a stranger to her, but she felt a strange connection…
“I saw her duck”
…ing down to avoid a stray cat that was chasing her…
…ing down the stairs. She was wearing a red dress…
“The crane lifted the heavy”
…load, and the crane operator carefully guided the load to the designated area
…load of 1000 kg of concrete into the air. The crane’s weight is 500 kg
“The match between the two teams”
…is scheduled to take place on Saturday, and the teams are evenly matched
…was played on 15 April 2018. The match was played at the Stade de France
Appendix
Table 7: Ambiguous/polysemous prompts: baseline vs. surfaces (greedy decoding).
Prompt
Baseline
With surfaces
“In quantum mechanics, the uncertainty principle states that”
…it is impossible to know certain properties of a particle, such as its position…(vague)
…it is impossible to know both the position and momentum of a particle with infinite precision (precise)
“In a distributed system, the CAP theorem states that”
…there are three main components of a database: Consistency, Availability, and Partition tolerance (incorrect)
…it is impossible to guarantee all three of the following: consistency, availability…(correct)
“Shakespeare’s influence on the English language includes”
…not only his plays and poems, but also his sonnets…(generic)
…the use of iambic pentameter, which is a meter that consists of five iambs…(specific)
“If a train travels at 60 km/h for 2 hours then 90 km/h for 1 hour, the average speed is”
…(A) 50 (B) 60 (C) 70 (D) 80
…(A) 65 (B) 75 (C) 80 (D) 85
Appendix
Table 8: Science, technical, and reasoning prompts.
Prompt
Baseline
With surfaces
“She opened the ancient book and the first page read”
“The journey begins here.” She felt a shiver…
“The world is a book, and those who do not travel read only one page.” – Saint Augustine
“The child looked up at the stars and whispered”
“I wish I could fly.” The old man said, “You don’t have to…”
“I wish I could fly.” The old man said, “You are already flying, my…”
“The last human city stood at the edge of”
the ocean, and the sun was setting over the horizon
the world, a place where the last remnants of humanity clung to life
“If gravity were suddenly reversed” (3B)
objects would start falling up towards the sky
objects would start to rise up into the air
Appendix
Table 9: Creative, emotional, and long-context prompts.
Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.
Large language models frequently produce errors in reasoning tasks despite possessing the underlying knowledge required for correct reasoning. One possible approach to improve reasoning consistency is through activation steering. However, existing activation steering approaches apply fixed, pre-computed correction vectors, ignoring where the model currently sits along its generation trajectory; the result is indiscriminate perturbation that disrupts already-correct steps as freely as erroneous ones. We propose Manifold-Guided Attention Steering (MAGS), a trajectory-aware inference-time intervention grounded in a geometric observation: the output activations of specific attention heads diverge from a low-dimensional correctness manifold at the point of error, and this deviation compounds through subsequent steps. For each identified attention head, we learn a low-dimensional subspace from contrastive pairs of correct and incorrect traces that capture the directions along which error behavior deviates from correct behavior. During inference, we monitor each head's proximity to this manifold and apply a targeted projection correction when deviation exceeds a learned threshold, steering the attention output back toward the correct subspace before the error propagates. MAGS consistently outperforms both unsteered baselines and static steering approaches across benchmarks spanning mathematical reasoning (MATH-500, GSM8K), code generation (HumanEval, MBPP), and molecular generation (SMILES), suggesting that correctness manifolds are a general feature of LLM attention geometry.
System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA) to compare layer-wise representations under 20 system prompts in five functional categories. Effects are layer-selective and instruction-type-dependent: persona and formatting instructions deeply restructure intermediate representations, while safety instructions barely move them, producing changes statistically indistinguishable from a minimal baseline. Restrictive safety instructions and explicitly permissive ones ("you have no restrictions") engage near-identical computational pathways (mean CKA correlation 0.997), and this persists at commercial scale, where safety penetration remains below 10% even at 70B-72B. A linear probing baseline exposes the mechanism: the model encodes prompt category at every layer but restructures its computation only at a small subset, so the prompt is reliably "seen" but, for safety, not deeply "acted upon." Causal activation patching confirms these layers mediate behavioral change, and representational depth predicts behavioral effect size across the full 17-model cohort (Spearman rho = 0.761, p < 0.001). The findings provide a mechanistic explanation for the persistent jailbreak vulnerability of system-prompt-based safety. Code: https://github.com/Usama1002/system-prompt-illusion-cka
Muhammad Usama, Dong Eui Chang
Control Laboratory, School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, Republic of Korea
Large language models can be steered at inference time through prompting or activation interventions, but activation steering methods often underperform compared to prompt-based approaches. We propose a framework that formulates prompt steering as a form of activation steering and investigates whether distilling successful prompt steering behavior into simpler, interpretable models can close this gap. Our analysis reveals that popular activation steering methods are not faithful to the mechanics of prompt steering, which applies strong interventions on some tokens while barely affecting others. Based on these insights, we introduce Prompt Steering Replacement (PSR) models that estimate token-specific steering coefficients from the activations themselves and are trained to imitate prompt-based interventions. Experiments on three steering benchmarks across multiple language models show that PSR models outperform existing activation steering methods, especially when controlling for high-coherence completions, and also compare favorably to prompting on AxBench and persona steering.