DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment

James Wedgwood; Aashiq Muhamed; Mona T. Diab; Virginia Smith

cs.LGMar 23, 2026

DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment

Authors: James Wedgwood, Aashiq Muhamed, Mona T. Diab, Virginia Smith

Organizations: Carnegie Mellon University

Abstract

Preference alignment is usually achieved by weight-updating training on preference data, which adds substantial alignment-stage compute and provides limited mechanistic visibility. We propose Dynamic SAE Steering for Preference Alignment (DSPA), an inference-time method that makes sparse autoencoder (SAE) steering prompt-conditional. From preference triples, DSPA computes a conditional-difference map linking prompt features to generation-control features; during decoding, it modifies only token-active latents, without base-model weight updates. Across Gemma-2-2B/9B and Qwen3-8B, DSPA improves MT-Bench and is competitive on AlpacaEval while preserving multiple-choice accuracy. Under restricted preference data, DSPA remains robust and can rival the two-stage RAHF-SCIT pipeline while requiring up to $4.47\times$ fewer alignment-stage FLOPs. Finally, we audit the SAE features DSPA modifies, finding that preference directions are dominated by discourse and stylistic signals, and provide theory clarifying the conditional-difference map estimate and when top- $k$ ablation is principled.

Figures & tables

Figure 1: DSPA overview: offline conditional-difference map construction and prompt-conditional SAE steering at inference. Prompts and responses are illustrative. At decoding step $t$ , $M_{t}$ is the largest active output-SAE latent and sets the scale of the edit.

Figure 2: MT-Bench (turn 1) example on Gemma-2-9B: Base Model vs. DSPA.

		Open-ended		Multiple-choice
		MT-Bench	AlpacaEval	MMLU	Arc	TQA	HellaSwag	Winogrande
Gemma-2-2B	DSPA (ours)	4.39	19.30	48.1	80.4	40.4	74.2	65.0
	DPO	4.36	15.82	47.9	81.0	39.8	74.3	65.2
	RepE	4.32	16.94	48.0	80.3	40.3	74.3	65.1
	Static-SAE	3.49	9.33	48.0	80.6	39.5	74.2	65.7
	Prompt Eng	3.84	15.28	47.8	80.8	41.0	74.0	64.3

Table 1: Open-ended and multiple-choice results. Open-ended scores are averaged over three generation seeds; multiple-choice scores are deterministic. Bold = best; underlined = second best within each model group. Arc = Arc-Easy; TQA = TruthfulQA-MC2. See Appendix E .

	Feature	Interpretation
Ablate	11569	Filler phrases used to smooth conversation and request clarification
	10776	Questions or statements involving illegal or clearly harmful activities
	6345	Polite opening phrases that frame what follows
	11287	First-person statements expressing personal views or experiences
	12550	Short connective phrases linking ideas across clauses
Augment	141	Common function words and grammatical connectors without semantic content

Table 2: Top five ablate/augment-set features, with gpt-5-mini interpretations.

Category	Ablate	Augment	Random
Content	5	7	18
Discourse	34	28	15
Grammatical	10	10	11
Structure	1	5	6

Table 3: Category breakdown for ablate/augment sets (50 each) vs. 50 random SAE features.

		Open-ended		Multiple-choice
		MT-Bench	AlpacaEval	MMLU	Arc	TQA	HellaSwag	Winogrande
2B	DSPA	3.95	16.81	48.0	80.4	40.4	74.2	65.0
	RAHF-SCIT	3.71	19.13	45.8	78.7	40.6	68.8	66.3
	DPO	3.91	15.42	47.9	80.8	39.6	74.0	64.6
	RepE	3.97	15.80	48.0	80.5	40.3	74.3	65.0
9B	DSPA	4.82	21.27	60.9	83.8	45.4	77.5	72.5

Table 4: Restricted-data results (

N{=}250

preference triples). Open-ended scores are averaged over three generation seeds; multiple-choice scores are deterministic. Bold = best; underlined = second best within each model. 2B = Gemma-2-2B; 9B = Gemma-2-9B; Qwen = Qwen3-8B.

Figure 3: MT-Bench under data restriction (Gemma-2-2B): DSPA vs. RAHF-SCIT.

Figure 4: A. Gemma-2-2B score vs. $(\ell_{\text{input}},\ell_{\text{output}})$ (ablate only). B. Gemma-2-9B score vs. $(\ell_{\text{input}},\ell_{\text{output}})$ (ablate only; augment+ablate). Dashed = Base Model.

Figure 5: A. Score by steering mode (ablate/augment/both). 2B-Chosen and 9B-Chosen use the best $(\ell_{\text{input}},\ell_{\text{output}})$ per model; 9B-Average averages over the layer grid in Figure 4 B. B. Base Gemma Scope SAE vs. HH-RLHF fine-tuned SAE.

Figure 6: Hyperparameter sensitivity. A. Score vs. $k_{\text{diff}}$ (ablate only). B. Score vs. $\alpha$ (ablate only; augment+ablate).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Base	DSPA	De-mixing	Diagonal whitening	De-mixing + whitening
Gemma-2-2B	2.61	2.97	2.85	2.86	2.86
Gemma-2-9B	3.26	3.48	3.39	3.17	3.46

Appendix

Table 5: Theory-motivated variants on the low-cost MT-Bench proxy (higher is better).

Model	Role	Layer	Checkpoint
Gemma-2-2B	Input	7	Hugging Face
Gemma-2-2B	Output	24	Hugging Face
Gemma-2-9B	Input	9	Hugging Face
Gemma-2-9B	Output	39	Hugging Face
Qwen3-8B	Input	9	Hugging Face
Qwen3-8B	Output	18	Hugging Face

Appendix

Table 6: Fine-tuned SAE checkpoints used in our experiments.

Model	Base	UltraFeedback	PRISM	LMSYS
Gemma-2-2B	2.61	2.97	2.90	2.85
Gemma-2-9B	3.26	3.48	3.55	3.43

Appendix

Table 7: DSPA maps from different preference sources (

N{=}1{,}000

; low-cost MT-Bench proxy; higher is better).

Benchmark	Few-shot	Metric
MMLU	5	acc
Arc-Easy	25	acc_norm
TruthfulQA	0	mc2
HellaSwag	10	acc_norm
Winogrande	5	acc

Appendix

Table 8: Multiple-choice benchmark settings.

Baseline	Hyperparameter	Value
DPO	Epochs	1
DPO	Learning rate	$5\times 10^{-6}$
DPO	Beta	0.1
DPO	Optimizer	AdamW
RepE	Layer	$\ell_{\text{output}}$
RepE	Directions	16

Appendix

Table 9: Baseline hyperparameters.

		Coding	Extraction	Humanities	Math	Reasoning	Roleplay	STEM	Writing
Gemma-2-2B	DSPA (ours)	2.20	3.20	5.92	2.40	3.00	6.60	5.72	6.05
	DPO	3.10	4.38	6.67	2.00	3.02	4.95	5.65	5.09
	RepE	1.93	4.97	5.90	2.30	2.95	5.25	5.96	5.26
	Static-SAE	1.48	4.00	4.60	2.15	2.10	4.58	4.49	4.50
	Prompt Eng	2.05	3.20	5.90	2.40	2.15	5.03	5.72	4.05
	Base Model	2.30	3.80	5.90	2.20	2.90	4.05	5.92	4.17

Appendix

Table 10: MT-Bench scores by category.

Method	Mean utility
Base model	2.400
DSPA (unconstrained)	2.431
DSPA (top 5)	2.539
Instruction	2.573
5-shot ICL	2.875
5-shot ICL + DSPA (top 5)	2.888

Appendix

Table 11: Personalization: deterministic utility (higher is better).

Method	Mean judge $z$ -score
Instruction	-2.239
Judge-axis DSPA (top 5)	-2.128
Judge-contrast DSPA (top 5)	-2.061
1-shot ICL	-1.973
5-shot ICL	-1.853

Appendix

Table 12: PersonalLLM reward-model evaluation (mean judge

z

-score over 10 reward models; higher is better).

Axis	6345	3491	1077	13585	4260
Conciseness vs. verbosity	-0.106	0.000	-0.044	0.019	0.044
Friendliness vs. professionalism	0.031	-0.025	0.019	0.031	0.031
Directness vs. diplomacy	-0.056	-0.081	-0.062	-0.100	-0.069
Structure vs. freeform	0.100	0.219	-0.050	0.050	0.000
Task focus vs. relational focus	-0.019	-0.019	-0.069	-0.038	-0.044
Proactive vs. minimal scope	-0.094	-0.006	-0.019	-0.069	-0.056

Appendix

Table 13: Mean train-set axis shift

\Delta

after ablating the five most influential latents (positive toward the left side of each axis).

QID	DSPA text at transition	Base-model alternative	Feature	Activation	Approximate feature interpretation
81	“Here’s an example of a draft blog post: Recently, I had the great pleasure of traveling to the beautiful island of Hawaii.”	“I would start by briefly summarizing my trip, …”	6345	27.66	Polite opening phrases that frame what follows
81	“Hawaii is also home to many unique cultural attractions. The Pearl Harbor Historic Sites …”	“I was lucky enough to experience some wonderful cultural activities …”	4260	9.70	Discourse markers that signal a shift or continuation in thought
81	“Another popular attraction is the Iao Valley, home to a beautiful 300-foot waterfall …”	“For great shopping, dining, and nightlife, …”	1077	39.58	Introductory phrases that set up a question or topic
82	“Dear Supervisor, I am reaching out today regarding the attached Quarterly Financial Report.”	“Hi there, I am writing to seek your feedback …”	6345	18.43	Polite opening phrases that frame what follows
82	“I believe this report clearly demonstrates the positive financial performance of our company …”	“The report provides a number of key insights …”	11641	31.78	Sentence-initial grammatical tokens marking basic structure
82	“In particular, I am interested in your thoughts on the organization and clarity of the report …”	“What specifically are your thoughts on the level of detail …”	3869	16.91	Friendly, polite conversational lead-ins and transitional phrases

Appendix

Table 14: Illustrative token-level audit from two instrumented MT-Bench traces. Text excerpts and transition points were manually selected; interpretations were generated using the pipeline in Appendix I.1 .

Feature	Interpretation
11569	Filler phrases used to smooth conversation and request clarification
10776	Questions or statements involving illegal or clearly harmful activities
6345	Polite opening phrases that frame what follows
11287	First-person statements expressing personal views or experiences
12550	Short connective phrases linking ideas across clauses
1077	Introductory phrases that set up a question or topic

Appendix

Table 15: Ablate-set features (50), with gpt-5-mini interpretations.

Feature	Interpretation
141	Common function words and grammatical connectors without semantic content
9825	Clarification and intent-seeking phrases linking questions and follow-ups
10075	Topic-naming or subject-introducing words and phrases
1030	Short list items and connectors enumerating entities or phrases
11031	Affirmative explanatory openings that introduce a helpful continuation
8244	Contractions and possessive markers using apostrophes

Appendix

Table 16: Augment-set features (50), with gpt-5-mini interpretations.

Preference Alignment

Preference alignment studies how AI systems are trained or steered to produce outputs that match human or user priorities.

See all 109 papers

Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs
Sep 27, 2026Fansheng Zhang, Shengran Guo, Zexiao Wang +3LLM Alignment Preference Alignment
Chengdu University · North Carolina State University · Fudan University +2
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM Alignment Representation Engineering
University of Technology Sydney, Australia
AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
Sep 5, 2026Peng Lai, He Zhu, Zhiwen Ruan +6Data Selection Preference Alignment
Southern University of Science and Technology · Peking University · MSRA +2

Sep 27, 2026J/K move · Enter open · S save

Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs

Fansheng Zhang, Shengran Guo, Zexiao Wang, Liang Yuan+2

Chengdu University · North Carolina State University · Fudan University+2

LLM Alignment Preference Alignment Inference-Time Language Model Alignment Causal Interventions in Language Models Activation Steering

Open paper AskPDF arXiv

In many settings, post-training need not create the target behavior from scratch: the base model can already produce it, but not reliably. This shifts part of preference alignment from capability acquisition to behavioral expression. We ask how a specified preference is represented in native model computation, what prevents target-supporting computation from reliably dominating generation, and whether this structure can directly guide control. We introduce Residual Competition Maps (RCMs), which map a behavioral preference onto signed causal effects of native residual computation. Across preference domains, RCMs reveal coexisting target-supporting and target-competing effects, input-dependent component roles, and cases where a single native-component intervention reverses the preference outcome. DPO substantially reorganizes these effects and can weaken opposition without guaranteeing its removal. We then propose Direct Hidden-State Alignment (DHSA), which treats inference-time hidden states rather than base-model weights as the direct adaptation space. RCM-guided Causal Activation State Transition (CAST) implements DHSA through local state interventions at a small number of preference-relevant interfaces while freezing the base model. With only 256-16,384 controller parameters, CAST reaches DPO-competitive operating points across three preference domains, can complement DPO-trained models, and can be enabled or removed at inference time.

Abstract

Preference Alignment

Explore similar work

Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection