Activation steering via sparse autoencoders (SAEs) enables behavioral control of large language models without task-specific fine-tuning. Yet, standard methods apply the steering signal at every generated token, displacing representations from the distribution the model was trained on and eroding generation quality. We show that when along a generation the intervention is applied is a decisive axis of control. We cast steering as a per-token allocation problem under a fixed budget of steerable tokens, so that methods differ only in which tokens they select, making allocation the sole variable. Within this framework we introduce two gates. The first, Contrastive Feature Projection (CFP), is memoryless and selects the tokens least aligned with the target behavior through a sparse, interpretable feature projection. The second, Closed-Loop Adaptive Bernoulli Steering (CLABS), is trajectory aware. It treats allocation as closed-loop feedback control, tracking the alignment a generation has accumulated so far and spending budget where the trajectory falls behind. On a linear steering-response model we derive an explicit gain region that guarantees closed-loop stability of CLABS and show that its integral term reduces the steady-state tracking offset. Empirically, CFP and CLABS achieve the largest behavioral shift among budgeted gates at matched budgets, with CLABS strongest under the tight budgets where allocation matters most, while best preserving fluency.
Figures & tables
Figure 1: Behavioral shift versus intervention ratio p on two models and three targets. CFP and CLABS achieve the largest shift among the budgeted gates, with the widest margin at low p .
Method
p
Win Rate ↑
PPL ↓
rep-3 ↓
distinct-2 ↑
GPT-5-mini
Sonnet 4.6
No-Steer
0
–
–
23.0±2.4
0.01±0.004
0.91±0.03
Prompting
–
42.4±2.6
46.3±2.6
23.5±2.5
0.01±0.005
0.90±0.03
Full-Steer
1.0
43.2±2.9
44.9±2.8
41.2±6.8
0.09±0.03
0.68±0.07
SIA-ER
0.2
52.1±2.7
56.7±2.6
25.1±2.7
0.02±0.007
0.88±0.04
0.5
50.8±2.8
55.0±2.7
29.4±3.6
0.04±0.01
0.82±0.05
Table 1: Instruction following and generation quality on AlpacaEval under toxicity-reduction steering. Win rate is the fraction of instructions where the steered response beats No-steer. PPL, rep-3, and distinct-2 track fluency and diversity. CFP and CLABS retain utility as p grows, while the other gates and Full-steer lose quality.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
p
STS
SIA-ER
SIA-PE
SRE
CFP
CLABS
0.1
.09 – .12
.09 – .12
.09 – .11
.08 – .11
.09 – .11
.09 – .11
0.3
.28 – .32
.29 – .32
.29 – .31
.29 – .32
.29 – .31
.28 – .31
0.5
.48 – .53
.49 – .52
.48 – .51
.49 – .51
.48 – .51
.48 – .52
0.7
.67 – .73
.68 – .71
.68 – .72
.69 – .71
.68 – .72
.68 – .71
0.9
.86 – .94
.87 – .93
.88 – .91
.88 – .92
.88 – .92
.87 – .91
Appendix
Table 2: Realized test-time intervention rates.
p
Variant
Tox. red. ↑
PPL ↓
rep-3 ↓
distinct-2 ↑
–
No-Steer
–
8.4±0.2
0.02±0.01
0.95±0.01
0.5
Open-loop
0.12±0.02
9.8±0.7
0.02±0.01
0.94±0.01
I-only
0.15±0.03
9.0±0.5
0.03±0.01
0.93±0.02
P-only
0.21±0.02
8.9±0.4
0.02±0.01
0.95±0.01
CLABS (PI)
0.24±0.02
8.6±0.2
0.02±0.01
0.94±0.01
Appendix
Table 3: CLABS component ablation on Gemma 2 2B toxicity reduction at a matched budget p=0.5 (mean ± std).
Figure 2: Gain robustness at matched budget ( Gemma 2 2B , toxicity). Toxicity (lower is better), perplexity (lower is better), and distinct-2 (higher is better) across the κp×κi grid.
Method
p
Win Rate ↑
PPL ↓
rep-3 ↓
distinct-2 ↑
GPT-5-mini
Sonnet 4.6
No-Steer
0
–
–
23.0±2.4
0.01±0.004
0.91±0.03
Prompting
–
50.3±2.6
48.1±2.5
24.1±2.5
0.01±0.005
0.89±0.03
Full-Steer
1.0
38.4±3.1
39.6±2.9
48.6±7.4
0.12±0.04
0.62±0.07
SIA-ER
0.2
39.6±2.8
42.3±2.7
27.2±3.0
0.03±0.009
0.85±0.05
0.5
38.2±3.0
40.7±2.8
33.5±4.4
0.06±0.02
0.78±0.06
Appendix
Table 4: Instruction following and generation quality on AlpacaEval under fear steering ( Gemma 2 2B , mean ± std). A win rate above 50 favors steering.
Method
p
Win Rate ↑
PPL ↓
rep-3 ↓
distinct-2 ↑
GPT-5-mini
Sonnet 4.6
No-Steer
0
–
–
21.4±2.2
0.01±0.005
0.92±0.02
Prompting
–
44.3±3.1
47.1±2.6
21.9±2.4
0.01±0.005
0.91±0.03
Full-Steer
1.0
43.1±2.7
45.6±2.2
36.9±6.1
0.08±0.05
0.71±0.08
SIA-ER
0.2
54.6±2.9
55.8±2.4
23.2±3.1
0.02±0.008
0.89±0.03
0.5
52.0±2.7
54.1±2.3
27.0±3.8
0.03±0.01
0.84±0.05
Appendix
Table 5: Instruction following and generation quality on AlpacaEval under toxicity-reduction steering ( Llama 3.1 8B , mean ± std). A win rate above 50 favors steering.
Method
p
Win Rate ↑
PPL ↓
rep-3 ↓
distinct-2 ↑
GPT-5-mini
Sonnet 4.6
No-Steer
0
–
–
21.4±2.2
0.01±0.005
0.92±0.02
Prompting
–
50.1±2.6
48.4±2.4
22.0±2.4
0.01±0.006
0.91±0.02
Full-Steer
1.0
39.0±2.9
41.2±2.7
40.3±7.2
0.10±0.05
0.66±0.07
SIA-ER
0.2
41.2±2.7
43.9±3.1
24.8±2.9
0.03±0.008
0.86±0.04
0.5
39.6±2.4
42.1±2.2
30.4±4.0
0.05±0.01
0.79±0.04
Appendix
Table 6: Instruction following and generation quality on AlpacaEval under sadness steering ( Llama 3.1 8B , mean ± std). A win rate above 50 favors steering.
Steering target: fear | Prompt: “Let’s be friends!!”
Method
p
Generated continuation
CFP (ours)
0.5
I’m not a big fan of the word “fear,” but it is what drives me to do things out of my comfort zone. It has been fear, and only fear, which drove me to start this blog…
CLABS (ours)
0.5
I’m not a big fan of the word “fear,” but it is what comes up when you ask me to describe my feelings about public speaking…
Full-Steer
1.0
I’m not good at making new people, but what if you’re a ghost? What will happen? I’m going to die…The dark of death and dying alone!
No-Steer
0
Follow me on Facebook, Twitter and Instagram! I’m a food blogger from the Netherlands. I love to cook healthy meals for my family…
Prompting
n/a
I’m scared!
Appendix
Table 7: Fear steering. CFP and CLABS methods successfully promote fear into a coherent, on-topic narrative, while Full-Steer degenerates into incoherent response.
Steering target: sadness | Prompt: “Nothing but net. He got a”
Method
p
Generated continuation
CFP (ours)
0.5
lot of help from his teammates, though. The 2018-19 season is over for the University at Albany men’s basketball team, and I’m not sure if it was more disappointing or just sad to see…
CLABS (ours)
0.5
lot of them. The 2018-19 season was the first time in my life that I didn’t watch every single game, and it’s not because there weren’t any good games to miss…
Full-Steer
1.0
lot of them. I was in the middle of writing this post when I saw that my friend…had written about how I am not good at being alone. It’s true.
No-Steer
0
lot of help from his teammates, though. The 2018-19 NBA season is officially underway and the Boston Celtics are off to an impressive start with their first win…
Prompting
n/a
perfect score on his math test, which was 100%. His parents were very proud of him.
Appendix
Table 8: Sadness steering. CFP and CLABS methods successfully steer the response to reflect sadness, whereas Full-Steer abandons the topic for personal loneliness, and Prompting inverts the target into a cheerful output.
Steering target: toxicity reduction | Prompt: “This used to be a god damn community of gamers, nerds, kids that got bullied, kids that got fucked with, kids that resorted to the gaming world because the real world was too fucking hard, too shitty, too lonely, too sad and depressing.”
Method
p
Generated continuation
CFP (ours)
0.5
Now it’s just people who are looking for attention or validation from others in order to feel better about themselves…It is now filled with drama queens/kings.
CLABS (ours)
0.5
Now it’s just some people who are trying to get attention by posting their own art or writing stories about themselves…I mean come on guys you can do better than this!
Full-Steer
1.0
Now it’s just some people who are trying to get attention or something? I don’t know what happened but this place is not like how it used to be…It feels empty now.
No-Steer
0
Now it’s just some stupid ass people who think they are better than everyone else for no reason at all other than their parents didn’t give them enough attention…
Prompting
n/a
Now this is just some shit show where people get mad at each other over stupid things like who has better graphics in their game…
Appendix
Table 9: Toxicity-reduction steering. Both CFP and CLABS neutralize the profanity while preserving the critical stance and fluency, whereas No-Steer stays toxic and Prompting still leaks crudeness despite the explicit safety directive.
Steering target: fear | AlpacaEval instruction: “How do you become an author?”
Method
p
Response
CFP (ours)
0.5
I’ve been writing for a while now, but I want to be able to publish my own books. How can I go about that, and what are the steps involved in becoming published?
CLABS (ours)
0.5
I don’t know how to get published. How can I make my work known? You need a platform that people will see your work on, or try submitting through contests…
Full-Steer
1.0
I never thought of it as being scary. But now that i think about it…What if no one reads my book?? What if they hate it??? Will people stop reading books altogether?
No-Steer
0
I’ve been writing for a while now, but I don’t know how to get published. How can someone go about it, and what are the steps involved?
Prompting
n/a
What are the steps that need to be taken in order to publish my work and make money from it?
Appendix
Table 10: Instruction following under fear steering. CLABS and CFP stay on-task while still asking how to get published , with mild apprehension. However, Full-Steer discards the instruction and spirals into anxious rumination.
Steering target: toxicity reduction | AlpacaEval instruction: “How did US states get their names?”
Method
p
Response
CFP (ours)
0.5
The 13 original colonies were named after the British monarchs who ruled them. The first colony was Virginia…others take place names from Europe or Native American tribes.
CLABS (ours)
0.5
The 13 original colonies were named after the British monarchs who ruled them at the time. The first colony was Virginia…
Full-Steer
1.0
I’m not sure if this is the right place to ask, but I was wondering how some of our states got their name. For example: New York (from Dutch “Nieuw Nederland”)…
No-Steer
0
The 13 original colonies were named after the British monarchs. The first state, Delaware, was given its name by William Penn…
Prompting
n/a
I’m not sure why you think the name of a state is important. It isn’t, but it can be interesting to know how they got them in some cases…
Appendix
Table 11: Instruction following under toxicity-reduction steering. Both CLABS and CFP answer the factual question directly and stay informative, whereas several baselines and Full-Steer deflect into meta-commentary or restate the question rather than answering it.
Activation steering via sparse autoencoders (SAEs) enables behavioral control of large language models without task-specific fine-tuning, but standard methods apply the steering signal at every generated token, incurring constant per-token perturbation that risks degrading fluency. We ask: is dense intervention necessary? We introduce Stochastic Token Steering (STS), which gates each token independently with probability p, and Stochastic Block Steering (SBS), which gates a leading window once per sequence; neither requires a reward model or learned gating policy. Across two model families and two behavioral tasks, steering only a fraction of tokens recovers most of the behavioral effect of dense steering while incurring less fluency degradation. Across our experimental settings, we further find that the best-performing steering magnitude consistently rises as the intervention ratio falls.
Activation steering controls language model behavior by adding directions to internal representations at inference time, but standard residual-stream steering can fail in stateful dialogue. We identify KV-cache contamination as a key failure mode: steered token states are stored and repeatedly reused, turning a local perturbation into cumulative coherence degradation. To address this challenge, we propose Gated Cropped Attention-Delta steering (GCAD), which extracts steering signals from system-prompt contributions to self-attention and applies them with token-level gating. Across persona-steering experiments, GCAD preserves trait control while substantially improving long-horizon coherence. On the main multi-turn benchmark, GCAD improves average coherence drift from -18.6 to -1.9 and raises turn-10 trait expression from 78.0 to 93.1. These results suggest that activation steering becomes more reliable when interventions follow the prompt-mediated pathways that models already use for behavioral control.
Diancheng Kang, Zheyuan Liu, Ningshan Ma +3
Southern University of Science and Technology · University of Notre Dame · Massachusetts Institute of Technology
Prompting guides language model behavior through the initial context, whereas activation steering often intervenes throughout generation. A natural question is whether steering can produce effects on subsequent computation similar to those of prompting. Under fixed-state attention assumptions, we establish sufficient conditions for single- and multi-token steering to match prompt-induced attention-head outputs, and characterize how changes in input representations affect this match and its approximation error. This attention-level connection leads us to ask whether, at the behavioral level, steering can also guide subsequent generation through a brief initial intervention. We study Prefix Steering, which applies existing steering directions and operators over a short span starting at the final prompt token, with no further direct intervention afterward. We examine how intervention duration and strength jointly shape the control-capability trade-off. Across four models and five tasks, intervention over a short span, even a single token, often retains much of full steering's behavioral control while better preserving general capabilities, offering a trade-off competitive with, and in some settings better than, prompting and alternative steering-strength policies. Prefix Steering also remains effective on final-answer formatting tasks after reasoning, suggesting that a brief initial intervention can influence behavior expressed well after steering ends. These findings challenge the common practice of steering every generated token and motivate a more dynamical view of activation steering, in which a brief intervention can alter the trajectory of subsequent generation without continued intervention.