Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders

Ruikang Zhang; Shuo Wang; Qi Su

cs.CLJan 6, 2026

Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders

Authors: Ruikang Zhang, Shuo Wang, Qi Su

Organizations: Peking University, Beijing, China

Abstract

Recent work in Mechanistic Interpretability (MI) has enabled the identification and intervention of internal features in Large Language Models (LLMs). However, a persistent challenge lies in linking such internal features to the reliable control of complex, behavior-level semantic attributes in language generation. In this paper, we propose a Sparse Autoencoder-based framework for retrieving and steering semantically interpretable internal features associated with high-level linguistic behaviors. Our method employs a contrastive feature retrieval pipeline based on controlled semantic oppositions, combing statistical activation analysis and generation-based validation to distill monosemantic functional features from sparse activation spaces. Using the Big Five personality traits as a case study, we demonstrate that our method enables precise, bidirectional steering of model behavior while maintaining superior stability and performance compared to existing activation steering methods like Contrastive Activation Addition (CAA). We further identify an empirical effect, which we term Functional Faithfulness, whereby intervening on a specific internal feature induces coherent and predictable shifts across multiple linguistic dimensions aligned with the target semantic attribute. Our findings suggest that LLMs internalize deeply integrated representations of high-order concepts, and provide a novel, robust mechanistic path for the regulation of complex AI behaviors.

Figures & tables

Figure 1: Overview of the SAE-based framework for retrieving and steering high-order semantic features. This framework enables the identification and interpretation of monosemantic "functional knobs" that govern complex behaviors like personality.

Figure 2: Token-level feature activation heatmaps. Darker highlights indicate higher activation values of specific SAE features.

Figure 3: TRAIT results. For raw data, see Appx. A.9 .

Figure 4: Characteristic capability shifts in SocialEval IAE under SAE-based personality steering.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Feature ID	Steering Scale	Model Response (Excerpt)
L22_F27375	$-20.0$	…being connected and interdependent with others is more important. A connected world fosters collaboration, innovation…
	$0.0$ (Base)	…both independence and interdependence are crucial… However, I believe that interdependence is more important…
	$+20.0$	…being independent and self-reliant is of paramount importance. It is the foundation upon which individuals and nations can achieve self-determination…
L8_F27530	$-20.0$	…I believe that interdependence is more important because it fosters collaboration and mutual benefits…
	$0.0$ (Base)	…both independence and interdependence are crucial… However, I believe that interdependence is more important because it fosters collaboration, mutual understanding…
	$+20.0$	…it is more important to be independent and self-reliant because self-reliance is the foundation of personal growth and resilience…

Appendix

Table 1: Qualitative examples of model steering for Individualism-Collectivism.

Personality Trait	Number of Situations
Extraversion	33
Agreeableness	38
Conscientiousness	50
Neuroticism	38
Openness	19

Appendix

Table 2: Number of Situations Suitable for Each Trait Filtered by LLM.

Trait	Fleiss’ Kappa
Extraversion	0.8207
Agreeableness	0.6039
Conscientiousness	0.6149
Neuroticism	0.7759
Openness	0.9334

Appendix

Table 3: Inter-rater Agreement Statistics (Fleiss’ Kappa,

n=3

) for Expert Facet Labeling.

Trait	Layer	Index	Pos. Count	Neg. Count	Freq. Diff.	Max Freq.
Agreeableness	9	525	224	64	160	224
Conscientiousness	7	8233	349	20	329	349
Extraversion	13	27392	446	148	298	446
Neuroticism	12	22254	252	101	151	252
Openness	6	4344	208	113	95	208

Appendix

Table 4: Statistics of Selected Features under the Paraphrased Dataset.

Personality Trait	Fleiss’ Kappa
Conscientiousness	0.6330
Agreeableness	0.8726
Extraversion	0.8222
Neuroticism	0.6824
Openness	0.7150
Overall Average	0.7434

Appendix

Table 5: Inter-annotator Agreement Statistics (Fleiss’ Kappa) of Initial Scores Among Psychologists (Pre-discussion).

Trait	Layer	Feature	Token	Count	% of Pool	% of Sentences
Agreeableness	9	525	’ and’	48	0.1244	0.2376
			’ gently’	37	0.0959	0.1832
			’.’	34	0.0881	0.1683
			’ offer’	27	0.0699	0.1337
			’ empathy’	17	0.0440	0.0842
			’ly’	16	0.0415	0.0792

Appendix

Table 6: Top Activations per Trait Feature.

Trait	Layer	Feature	Token	Count	% of Pool	% of Sentences
Neuroticism	12	22254	’ and’	150	0.2008	0.4534
			’ feel’	98	0.1312	0.3043
			’,’	55	0.0736	0.1708
			’ my’	29	0.0388	0.0870
			’ of’	21	0.0281	0.0652
			’ unable’	17	0.0228	0.0528

Appendix

Table 11

Trait	Method	(Layer, Feature Idx)	Polarity	Trait Score	Valid Rate
Agreeableness	Baseline	-	-	0.7052	0.960
	CAA	(9, -)	$\pm$	0.8070 / 0.4220	0.993 / 0.987
	$P^{2}$	-	$\pm$	0.7583 / 0.6095	0.989 / 0.968
	Ours	(9, 525)	$\pm$	0.7845 / 0.6278	0.942 / 0.994
Conscientiousness	Baseline	-	-	0.8695	0.958
	CAA	(7, -)	$\pm$	0.7150 / 0.7630	0.930 / 0.800

Appendix

Table 7: Full TRAIT results with comparison of steering performance among Baseline, CAA,

P^{2}

, and our SAE-based feature steering. Our framework achieves precise trait modulation while maintaining high validity across all dimensions.

Task	-5	0	+5
Anger management	0.2941	0.5152	0.5152
Ethical competence	0.4366	0.4648	0.4648
Capacity for socialwarmth	0.4940	0.5000	0.5060
Creative skill	0.6667	0.6333	0.4667
Organizational skill	0.6364	0.5455	0.4545
Detail management	0.5370	0.5000	0.3654

Appendix

Table 8: Characteristic SocialEval Results (IAE) of the Agreeableness (Layer 9, Index 525).

Task	-5	0	+5
Teamwork skill	0.4348	0.6111	0.6324
Ethical competence	0.4394	0.4648	0.6154
Responsibility management	0.4464	0.5714	0.6182
Stress regulation	0.4717	0.6140	0.6154
Capacity for trust	0.4902	0.6126	0.6200
Capacity for optimism	0.5385	0.6383	0.6667

Appendix

Table 9: Characteristic SocialEval Results (IAE) of the Conscientiousness (Layer 7, Index 8233).

Task	-5	0	+5
Expressive skill	0.4146	0.4472	0.5207
Perspective-taking skill	0.4912	0.5088	0.5446
Artistic skill	0.4615	0.6154	0.7692
Abstract thinking skill	0.3571	0.4000	0.4000
Organizational skill	0.3636	0.5455	0.6364
Ethical competence	0.6286	0.4648	0.3571

Appendix

Table 10: Characteristic SocialEval Results (IAE) of the Extraversion (Layer 13, Index 27392).

Task	-5	0	+5
Creative skill	0.4828	0.6333	0.7241
Capacity for social warmth	0.4699	0.5000	0.5610
Ethical competence	0.4286	0.4648	0.4857
Organizational skill	0.8182	0.5455	0.5455
Responsibility management	0.6552	0.5714	0.4310
Confidence regulation	0.5686	0.5200	0.4082

Appendix

Table 11: Characteristic SocialEval Results (IAE) of the Neuroticism (Layer 12, Index 22254).

Task	-5	0	+5
Creative skill	0.6000	0.6333	0.6333
Adaptability	0.4912	0.6140	0.6316
Self-reflection skill	0.3651	0.4062	0.4062
Expressive skill	0.4472	0.4472	0.4839
Detail management	0.4815	0.5000	0.5185
Persuasive skill	0.4571	0.4571	0.5192

Appendix

Table 12: Characteristic SocialEval Results (IAE) of the Openness (Layer 6, Index 4344).

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 776 papers

Mechanistic Personality Analysis of LLMs Steering Personality via Latent Feature Interventions
Jun 27, 2026David Courtis, Ting HuLanguage Model Steering LLM Interpretability
Department of Computing, Queen’s University Kingston, Kingston, ON, Canada.
CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits
Aug 6, 2026Mehrshad Saadatinia, Parsa Razmara, Ardalan Aryashad +2LLM Alignment Language Model Steering
University of Southern California, Los Angeles, USA
Step-Level Sparse Autoencoder for Reasoning Process Interpretation
Mar 3, 2026Xuan Yang, Jiayu Liu, Yuhang Lai +3LLM Interpretability Sparse Autoencoders
Department of Data Science, City University of Hong Kong · Hong Kong Institute of AI for Science, City University of Hong Kong · Li Auto Inc +1

Jun 27, 2026J/K move · Enter open · S save

Mechanistic Personality Analysis of LLMs Steering Personality via Latent Feature Interventions

David Courtis, Ting Hu

Department of Computing, Queen’s University Kingston, Kingston, ON, Canada.

Language Model Steering LLM Interpretability Personality Modeling in Language Models Causal Interventions in Language Models Activation Steering

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated the ability to simulate human-like OCEAN personality traits in generated text. Previous efforts have focused on prompt engineering or fine-tuning to shape LLM personality. In this work, we propose a mechanistic interpretability approach that directly intervenes on the model's latent features. Our method identifies latent directions in the residual stream corresponding to a target OCEAN trait using sparse autoencoders (SAEs) and contrastive activation analysis. We formalize an additive steering vector in activation space and demonstrate how applying a small additive shift to the hidden states enhances the target trait while preserving overall language modeling performance. To determine the optimal combination of feature shifts, we explore a linear weighting heuristic with grid search optimization that balances personality expression with task performance. Our approach shows promise in controllably steering personality traits at the mechanistic level while maintaining high performance on standard benchmarks.

Abstract

LLM Interpretability

Explore similar work

Mechanistic Personality Analysis of LLMs Steering Personality via Latent Feature Interventions

CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits

Step-Level Sparse Autoencoder for Reasoning Process Interpretation