cs.LGFeb 28, 2026

Weight Updates as Activation Shifts: A Principled Framework for Steering

Authors: Dyah Adila, John Cooper, Alexander Yun, Avi Trost, Frederic Sala

Organizations: Department of Computer Science, University of Wisconsin-Madison

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Activation steering promises to be an extremely parameter-efficient form of adaptation, but its effectiveness depends on critical design choices -- such as intervention location and parameterization -- that currently rely on empirical heuristics rather than a principled foundation. We establish a first-order equivalence between activation-space interventions and weight-space updates, deriving the conditions under which activation steering can replicate fine-tuning behavior. This equivalence yields a principled framework for steering design and identifies the post-block output as a theoretically-backed and highly expressive intervention site. We further explain why certain intervention locations outperform others and show that weight updates and activation updates play distinct, complementary functional roles. This analysis motivates a new approach -- joint adaptation -- that trains in both spaces simultaneously. Our post-block steering achieves accuracy within 0.2%-0.9%$ of full-parameter tuning, on average across tasks and models, while training only 0.04% of model parameters. It consistently outperforms prior activation steering methods such as ReFT and PEFT approaches including LoRA, while using significantly fewer parameters. Finally, we show that joint adaptation often surpasses the performance ceilings of weight and activation updates in isolation, introducing a new paradigm for efficient model adaptation.

Figures & tables

Figure 1: Left: Weight tuning scales with MLP dimension, while activation steering scales only with model dimension. Right: pre-MLP vs. post-MLP vs. post-block (ours). We steer after the skip connection is added back to the MLP output, accounting for both pathways.
Figure 2
ModelMethodParams (%)BoolQΔ\DeltaWinoGΔ\DeltaARC-CΔ\DeltaGSM8KΔ\DeltaAQuAΔ\DeltaListOpsΔ\DeltaAvg.
Llama-3.2-1BSFT100%88.20.058.20.060.30.032.20.036.20.066.20.00.0
LoRA0.45%84.7-3.557.0-1.261.6+1.331.8-0.433.6-2.960.0-6.2-2.2
ReFT0.04%84.1-4.149.3-8.957.0-2.731.6-0.630.2-6.049.3-16.9-6.5
Ours0.04%86.2-2.060.1+1.960.30.031.5-0.736.5+0.365.4-0.8-0.2
gemma-3-1bSFT100%83.30.051.40.048.20.023.40.032.70.065.80.00.0
LoRA0.45%84.7+1.451.6+0.250.6+2.422.6-0.831.6-1.153.6-12.2-1.7
Table 1: Comparison against SFT. Δ\Delta is performance difference relative to SFT. LoRA, ReFT and Ours all use rank 8
ModelMethodParams (%)BoolQΔ\DeltaWinoGΔ\DeltaARC-CΔ\DeltaGSM8KΔ\DeltaAQuAΔ\DeltaListOpsΔ\DeltaAvg.
Llama-3.2-1BLoFIT0.003%83.0-5.249.2-9.046.3-20.028.2-4.032.5-3.764.1-2.1-7.3
JoLA0.007%83.6-4.648.7-9.552.5-7.828.0-4.231.1-5.162.2-4.0-5.9
Ours (1-vec)0.003%86.1-2.157.6-0.658.0-1.728.6-3.630.9-6.364.6-1.6-2.6
Ours r=10.005%86.2-2.052.0-6.258.8-0.929.2-3.033.6-2.664.4-1.8-2.8
gemma-3-1bLoFIT0.003%75.8-7.549.6-1.846.7-1.515.1-8.328.4-4.362.3-3.5-4.5
JoLA0.007%75.1-8.251.2-0.246.4-1.814.7-8.726.5-6.357.8-8.0-5.5
Table 2: Comparison with methods with tiny parameter budgets. Δ\Delta is performance difference relative to SFT.
MethodParams (%)LC Win Rate (↑)(\uparrow)SE
Full SFT100%11.49±\pm 0.51
Ours (nonlinear, r=16r{=}16 )0.05%11.34±\pm 0.48
Ours (linear, r=16r{=}16 )0.05%11.00±\pm 0.53
LoRA ( r=16r{=}16 )0.52%9.59±\pm 0.40
LoRA ( r=8r{=}8 )0.26%10.52±\pm 0.48
ReFT ( r=16r{=}16 , all layers)0.05%10.96±\pm 0.43
Table 3: Instruction tuning on Llama 3.1 8B with AlpacaEval 2.0 (length-controlled win rate against GPT-4 Turbo).
MethodParams (%)Pass@1
Base model-10.2
LoRA0.52%81.5±0.781.5\pm 0.7
Ours (nonlinear)0.04%84.7±0.584.7\pm 0.5
Ours (linear)0.04%84.3±0.884.3\pm 0.8
Table 4: RL on DeepSeek-R1-Distill-Qwen-1.5B with GSM8K.
ModelDatasetSFTLoRA (r=16)LoRA (r=8)AdapterJointJoint-Orth
Llama-3.2-1BGSM8K32.231.231.831.331.131.1
BoolQ88.288.484.786.288.488.5
WinoG58.264.357.052.456.959.3
Gemma-3-1bGSM8K23.423.822.621.622.824.4
BoolQ83.384.884.781.484.584.8
WinoG51.451.851.450.851.350.9
Table 5: Joint training vs. its components. “Joint” is naive joint training, “Joint-Orth” incorporates the orthogonality constraint. Joint uses LoRA (0.45%) + Adapter (0.34%), total 0.79% trainable params. LoRA r=16r=16 has 0.9%0.9\% trainable params.
Figure 8
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

SymbolMeaning
δA\delta AA learned parameter with name AA
ΔA\Delta AA small change in AA
hhThe input to the MLP block
dmodeld_{\text{model}}The embedding dimension of the transformer
dmlpd_{\text{mlp}}The hidden dimension of each MLP layer
XXData matrix
Appendix
Table 6: Notation
DatasetTrainTestValidation
BoolQ9,4303,270–
Winogrande9,2501,7701,270
ARC-Challenge1,1201,170299
GSM8K8,7901,320–
AQuA97,467254254
ListOps96,0002,0002,000
Appendix
Table 7: Dataset details. Validation is created from train; table shows original dataset sizes.
DatasetPrompt Template
BoolQAnswer the question with a true/false based on the given passage. Passage: {passage} Question: {question}
WinograndePlease choose the correct answer to fill in the blank to complete the given sentence: {sentence} Option1: {option1} Option2: {option2}
ARC-ChallengeAnswer the following multiple-choice question.
 Question: {question}
 {options}
GSM8K{question}
Appendix
Table 8: Dataset Prompt Templates
ModelBoolQWinoGGSM8KListOps
Llama-3.2-1B1e−31e^{-3}7.5e−47.5e^{-4}7.5e−47.5e^{-4}7.5e−47.5e^{-4}
gemma-3-1b5e−45e^{-4}7.5e−47.5e^{-4}2e−32e^{-3}1e−31e^{-3}
Qwen 3 4B5e−45e^{-4}1e−31e^{-3}7.5e−47.5e^{-4}2e−32e^{-3}
Llama 3.1 8B5e−45e^{-4}5e−45e^{-4}5e−45e^{-4}2e−42e^{-4}
Appendix
Table 9: Chosen hyperparameters to reproduce experiment numbers (Ours) in Table 1
ModelVariantBoolQWinoGGSM8KListOps
Llama-3.2-1BOurs r=12e−32e^{-3}5e−45e^{-4}5e−35e^{-3}3e−33e^{-3}
Ours vector5e−45e^{-4}1e−31e^{-3}2e−32e^{-3}1e−31e^{-3}
gemma-3-1bOurs r=12e−32e^{-3}5e−45e^{-4}1e−31e^{-3}2e−32e^{-3}
Ours vector5e−35e^{-3}5e−35e^{-3}1e−31e^{-3}5e−35e^{-3}
Qwen 3 4BOurs r=17e−47e^{-4}7e−47e^{-4}1e−41e^{-4}7e−47e^{-4}
Ours vector2e−32e^{-3}2e−32e^{-3}7e−47e^{-4}1e−31e^{-3}
Appendix
Table 10: Chosen hyperparameters to reproduce experiment numbers (Ours) in Table 2
ModelBoolQWinoGGSM8K
Llama-3.2-1B7e−47e^{-4}1e−41e^{-4}1e−41e^{-4}
gemma-3-1b5e−45e^{-4}7.5e−47.5e^{-4}2e−32e^{-3}
Qwen 3 4B5e−45e^{-4}1e−31e^{-3}7.5e−47.5e^{-4}
Appendix
Table 11: Chosen hyperparameters to reproduce experiment numbers (Joint Orth) in Table 5
ModelMethodBoolQWinoGGSM8KListOps
Llama-3.2-1BLoRA8.0×10−18.0\times 10^{-1}4.84.85.5×10−15.5\times 10^{-1}7.4×10−17.4\times 10^{-1}
ReFT6.8×10−16.8\times 10^{-1}2.1×10−12.1\times 10^{-1}3.5×10−13.5\times 10^{-1}4.3×10−14.3\times 10^{-1}
Ours2.7×10−12.7\times 10^{-1}9.2×10−19.2\times 10^{-1}4.4×10−14.4\times 10^{-1}2.9×10−12.9\times 10^{-1}
gemma-3-1bLoRA6.1×10−16.1\times 10^{-1}3.33.33.0×10−13.0\times 10^{-1}9.39.3
ReFT6.0×10−16.0\times 10^{-1}0.00.06.5×10−16.5\times 10^{-1}6.7×10−16.7\times 10^{-1}
Ours3.3×10−13.3\times 10^{-1}7.8×10−17.8\times 10^{-1}2.0×10−12.0\times 10^{-1}1.7×10−11.7\times 10^{-1}
Appendix
Table 12: Standard deviation of experiments in Table 1
ModelMethodBoolQWinoGGSM8KListOps
Llama-3.2-1BLoFIT8.0×10−18.0\times 10^{-1}4.84.85.5×10−15.5\times 10^{-1}7.4×10−17.4\times 10^{-1}
JoLA6.8×10−16.8\times 10^{-1}2.1×10−12.1\times 10^{-1}3.5×10−13.5\times 10^{-1}4.3×10−14.3\times 10^{-1}
Ours vector4.2×10−14.2\times 10^{-1}5.2×10−15.2\times 10^{-1}4.5×10−14.5\times 10^{-1}3.4×10−13.4\times 10^{-1}
Ours r=14.7×10−14.7\times 10^{-1}4.7×10−14.7\times 10^{-1}5.1×10−15.1\times 10^{-1}3.7×10−13.7\times 10^{-1}
gemma-3-1bLoFIT5.1×10−15.1\times 10^{-1}7.1×10−17.1\times 10^{-1}1.021.023.0×10−13.0\times 10^{-1}
JoLA5.3×10−15.3\times 10^{-1}3.2×10−13.2\times 10^{-1}1.101.102.5×10−12.5\times 10^{-1}
Appendix
Table 13: Standard deviation of experiments in Table 2
ModelMethodBoolQWinoGGSM8K
Llama-3.2-1BLoRA8.17×10−18.17\times 10^{-1}4.865.50×10−15.50\times 10^{-1}
Adapter4.74×10−14.74\times 10^{-1}4.10×10−14.10\times 10^{-1}1.48×10−11.48\times 10^{-1}
Joint8.14×10−18.14\times 10^{-1}3.031.58
Joint Orth5.08×10−15.08\times 10^{-1}4.025.97×10−15.97\times 10^{-1}
gemma-3-1bLoRA6.11×10−16.11\times 10^{-1}3.343.03×10−13.03\times 10^{-1}
Adapter6.91×10−16.91\times 10^{-1}6.42×10−16.42\times 10^{-1}5.46×10−15.46\times 10^{-1}
Appendix
Table 14: Standard deviation of experiments in Table 5
Figure 6: W2W_{2} after orthogonality projection retains its magnitude.

Explore similar work

CardsList

  1. From Weights to Activations: Is Steering the Next Frontier of Adaptation?

    Apr 15, 2026Simon Ostermann, Daniil Gurgurov, Tanja Baeumel +4Language Model SteeringActivation Steering

    Saarland University · German Research Center for Artificial Intelligence (DFKI) · Centre for European Research in Trusted AI (CERTAIN) +6

  2. First-Order Steering: Translating Weight Adaptation into Activation Steering

    Oct 3, 2026Sri Pranav Kunda, Alexander Kurz, Tomas Dominik +1Activation SteeringLanguage Model Steering

    University of California, Davis · Chapman University · University of California, Los Angeles +1

  3. LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment

    Jul 25, 2026Pascal Ekin, Hyosun Choi, Wei JieLLM AlignmentLow-Rank Adaptation

    Royal Holloway, University of London, United Kingdom · University of West London, United Kingdom