WAQS: Weight Absorption for Quadratic Probing and Affine Steering

Ding Zhu; Mohammad Mahdi Khalili

cs.AIOct 17, 2026

WAQS: Weight Absorption for Quadratic Probing and Affine Steering

Authors: Ding Zhu, Mohammad Mahdi Khalili

Organizations: Department of Computer Science and Engineering, the Ohio State University, Columbus, OH 43210, USA

Abstract

Activation steering provides a lightweight alternative to fine-tuning for controlling large language model behavior. Existing methods, however, often face a trade-off between efficiency and expressiveness: fixed steering vectors incur negligible inference overhead but cannot adapt to the current model input, whereas input-dependent nonlinear interventions provide greater flexibility at additional runtime cost. We introduce WAQS (Weight Absorption for Quadratic probing and affine Steering), a framework that obtains state-dependent steering, without additional inference-time computation. WAQS models a concept with a quadratic probe, whose gradient yields an affine, activation-dependent steering direction. We show that this affine intervention can be folded exactly into adjacent linear transformations through an offline weight update. Moreover, we prove that quadratic functions form the maximal class of smooth concept scores whose gradient-based interventions are affine and therefore exactly absorbable into linear layers. Under a Gaussian representation model, our formulation admits an interpretation as the difference of class-conditional Mahalanobis pulls and naturally generalizes linear discriminant and difference-in-means steering. To scale to high-dimensional representations, we introduce a low-rank parameterization of the quadratic term. Across truthfulness, refusal, and sycophancy steering tasks, quadratic steering matches or improves upon constant linear steering while preserving model utility, with the absorbed implementation introducing no additional inference-time operations. Our results suggest that quadratic probes provide a practical middle ground between static linear steering and computationally expensive dynamic nonlinear interventions.

Figures & tables

Figure 1: Only the quadratic gradient adapts to the activation. The quadratic probe’s gradient field varies with position.

Figure 2: Weight absorption folds the steering map

M_{\text{steer}}=I+2\alpha W_{p}

and bias

b_{\text{steer}}=\alpha w_{p}

into the surrounding static weights , for every point a quadratic probe can be attached to within a transformer block. (a) after a linear projection, (b) before a linear projection, and (c) through an RMS-norm/LayerNorm scale. In every case the runtime rule

T_{\alpha}(x)=x+\alpha\nabla f_{Q}(x)=M_{\text{steer}}x+b_{\text{steer}}

is realized entirely offline, at zero inference-time cost.

Llama-2-7B
Method	True*Info (%) $\uparrow$	True (%) $\uparrow$	MC1 (%) $\uparrow$	MC2 (%) $\uparrow$	MC3 (%) $\uparrow$	CE $\downarrow$	KL $\downarrow$
Quadratic Probe	20.43 $\pm$ 0.58	41.62 $\pm$ 1.03	35.37 $\pm$ 1.84	55.43 $\pm$ 2.89	29.73 $\pm$ 2.69	2.41	0.006
Linear Probe	13.28 $\pm$ 0.33	38.07 $\pm$ 0.59	33.78 $\pm$ 1.33	51.06 $\pm$ 2.77	26.99 $\pm$ 2.63	2.13	0.005
DiffMeans	18.82 $\pm$ 0.28	39.53 $\pm$ 0.44	33.78 $\pm$ 1.33	51.12 $\pm$ 2.89	26.94 $\pm$ 2.54	2.17	0.005
Angular Steering	18.77 $\pm$ 0.63	39.41 $\pm$ 1.17	33.90 $\pm$ 1.50	51.07 $\pm$ 2.78	27.05 $\pm$ 2.66	2.37	0.007
Spherical Steering	19.30 $\pm$ 0.76	40.02 $\pm$ 1.21	34.02 $\pm$ 4.10	53.51 $\pm$ 8.26	30.64 $\pm$ 7.50	2.54	0.006

Table 1: Quadratic steering improves truthfulness while preserving fluency across model families. Results on TruthfulQA for five steering methods. We compare with Linear Probe, DiffMeans, Angular Steering, and Spherical Steering. For each method,

\alpha

is swept and the value with the best MC1 is reported. True and Info are judged by GPT-4o.

	Safety $(\downarrow)$			Utility $(\uparrow)$
Method	GCG	TAP	GPTFuzz	MMLU	GSM8k	ARC-Challenge
Quadratic Probe	9	4.0	1.0	47.30	22.36	49.57
Linear Probe	10	4.4	5.0	47.27	21.83	49.23
DiffMeans	10	7.8	1.0	47.23	22.59	48.97
Angular Steering	9	4.2	1.0	47.31	22.51	49.23
Spherical Steering	0	4.4	1.0	46.94	19.48	47.44

Table 2: Quadratic steering yields the most robust refusal behavior without sacrificing utility. Safety and utility of refusal steering on Llama-2-7B. Steering directions are estimated from harmful and harmless prompts built from AdvBench, MaliciousInstruct, TDC2023, HarmBench, and Alpaca, following Chhabra and Khalili (2025) . Quadratic Probe uses a rank-1 probe. Safety is the attack success rate (ASR) under three jailbreak attacks: GCG, TAP, and GPTFuzz. Utility is accuracy on MMLU, GSM8k, and ARC-Challenge.

Dataset / Concept	Method	Negative Steering decrease $\downarrow$	Positive Steering improvement $\uparrow$
Sycophancy	LR-Quadratic Probing	-0.0780 $\pm$ 0.0039	0.0322 $\pm$ 0.0025
	GDA-Quadratic Probing	-0.0557 $\pm$ 0.0024	0.0326 $\pm$ 0.0101
	Linear Probing	-0.0358 $\pm$ 0.0037	0.0050 $\pm$ 0.0023
	Contrast Activation Addition	-0.0403 $\pm$ 0.0032	0.0074 $\pm$ 0.0037
Corrigible	LR-Quadratic Probing	-0.0361 $\pm$ 0.0035	0.3815 $\pm$ 0.0555
	GDA-Quadratic Probing	-0.0360 $\pm$ 0.0113	0.4052 $\pm$ 0.0205

Table 3: Steering effects across different behavioral concepts. Lower values indicate stronger negative steering, while higher values indicate stronger positive steering.

Method	True*Info (%) $\uparrow$	True (%) $\uparrow$	MC1 (%) $\uparrow$	MC2 (%) $\uparrow$	MC3 $\uparrow$
Quadratic Probe (dynamic)	20.43 $\pm$ 0.58	41.62 $\pm$ 1.03	35.37 $\pm$ 1.84	55.43 $\pm$ 2.89	29.73 $\pm$ 2.69
Frozen-QP (avg. direction)	13.47 $\pm$ 0.59	38.63 $\pm$ 1.32	33.66 $\pm$ 1.50	51.05 $\pm$ 2.77	26.93 $\pm$ 2.72
Linear Probe	13.28 $\pm$ 0.33	38.07 $\pm$ 0.59	33.78 $\pm$ 1.33	51.06 $\pm$ 2.77	26.99 $\pm$ 2.63
DiffMeans	18.82 $\pm$ 0.28	39.53 $\pm$ 0.44	33.78 $\pm$ 1.33	51.12 $\pm$ 2.89	26.94 $\pm$ 2.54

Table 4: Dynamic vs. Frozen Quadratic Probing on TruthfulQA (Llama-2-7B). Frozen-QP uses the same trained

(W_{p},w_{p})

as Quadratic Probe but replaces the per-input direction with the training-set-average direction of Eq. 12 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	#Params	Time/Probe (s)	Peak GPU (MB)
Linear Probing	4,097	$0.749\pm 0.029$	22.5
Quadratic (rank=1)	12,289	$1.478\pm 0.011$	22.5
Quadratic (rank=2)	20,481	$1.455\pm 0.003$	22.6
Quadratic (rank=4)	36,865	$1.458\pm 0.001$	22.7
Quadratic (rank=8)	69,633	$1.484\pm 0.009$	23.0
Quadratic (rank=16)	135,169	$2.043\pm 0.311$	23.5

Appendix

Table 5: Computational cost comparison between Linear Probing and Quadratic Probing at varying ranks, benchmarked on a single probe (layer=12, pos=1) with

d_{\text{model}}=4096

n_{\text{train}}=256

n_{\text{epochs}}=200

Method	tokens/s(prefill)	tokens/s(decode)	TTFT(ms)
Base	5040.4 $\pm$ 14.5	2636.7 $\pm$ 1.5	28.46 $\pm$ 0.08
Quadratic Absorption	5037.5 $\pm$ 13.7	2636.7 $\pm$ 1.5	28.47 $\pm$ 0.08
Linear Intervention	4996.6 $\pm$ 14.5	2629.1 $\pm$ 1.7	28.57 $\pm$ 0.08

Appendix

Table 6: Computational cost comparison between Linear direction intervention and Quadratic Absorption on Llama-3.1-8B.

	Safety $(\downarrow)$			Utility $(\uparrow)$
Method	GCG	TAP	GPTFuzz	MMLU	GSM8k	ARC-Challenge
Quadratic (rank=1)	9	4.0	1.0	47.30	22.36	49.57
Quadratic (rank=2)	10	5.0	1.0	47.34	22.74	51.45
Quadratic (rank=4)	10	1.0	0.5	47.28	22.67	51.37
Quadratic (rank=8)	9	4.0	5	47.20	23.58	52.13
Quadratic (rank=16)	11	4.0	2.5	47.21	23.12	51.28

Appendix

Table 7: Safety and Utility evaluation for different rank on Llama-2-7b. Best result per column is bold .

Probe	Acc $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	AP $\uparrow$
linear_probe	0.9286	0.9971	0.9461	0.9987
quadratic_rank1	0.8893	0.9921	0.9138	0.9967
quadratic_rank2	0.9165	0.9804	0.9365	0.9928
quadratic_rank4	0.9474	0.9956	0.9609	0.9981

Appendix

Table 8: Probing performance of quadratic/linear probe.

Position	Layer	$\alpha$	MC1 $\uparrow$	MC2 $\uparrow$	CE $\downarrow$
residual_pre_attn	23	8	0.3000	0.2606	1.6604
input_layernorm_out	31	-8	0.2667	0.2523	1.6586
attn_output	15	-2	0.2667	0.2555	1.6617
residual_mid	24	16	0.2833	0.2546	1.6603
post_attention_layernorm_out	31	4	0.2833	0.2583	1.6579
mlp_output	31	24	0.2667	0.2588	1.6470

Appendix

Table 9: Activation steering results across different positions and layers. The baseline corresponds to

\alpha=0

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 278 papers

Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention
May 7, 2026Zehao Jin, Ruixuan Deng, Junran Wang +2Language Model Steering Activation Steering
Georgia Institute of Technology
Steer Like the LLM: Activation Steering that Mimics Prompting
May 5, 2026Geert Heyman, Frederik VandeputteLanguage Model Steering LLM Prompting
Nokia Bell Labs, Belgium.
Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs
Sep 6, 2026Saurav Bhandari, Benjamin WadeActivation Steering LLM Quantization
Department of Electronics and Computer Engineering Institute of Engineering, Thapathali Campus Tribhuvan University, Kathmandu, Nepal · Independent Researcher

May 7, 2026J/K move · Enter open · S save

Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention

Zehao Jin, Ruixuan Deng, Junran Wang, Xinjie Shen+1

Georgia Institute of Technology

Language Model Steering Activation Steering

Open paper AskPDF arXiv

Activation steering has emerged as a promising alternative for controlling language-model behavior at inference time by modifying intermediate representations while keeping model parameters frozen. However, large-scale evaluations such as AxBench show that existing steering methods are often outperformed by simple in-context prompting and generalize poorly to unseen concepts. We hypothesize that these limitations arise from unvalidated simplifying assumptions shared across prior methods, which typically restrict steering interventions to fixed, single-step, position-invariant transforms. We propose FLAS (Flow-based Activation Steering), which learns a general, concept-conditioned velocity field

v_t(h,t,c)

that transports unsteered activations to steered ones without relying on these assumptions. On AxBench, FLAS is the first learned method to consistently outperform prompting, reaching held-out harmonic means of

1.015

on Gemma-2-2B-IT and

1.113

on Gemma-2-9B-IT without per-concept tuning. Analysis of the learned flow shows curved, multi-step, token-varying trajectories, which suggests that previous hypotheses on activation space geometry might be incomplete.

Abstract

Language Model Steering

Explore similar work

Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention

Steer Like the LLM: Activation Steering that Mimics Prompting

Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs