cs.CLNov 1, 2026

Information-Theoretic Design of Sparse Autoencoder-Based Steering

Authors: Hirofumi Morise

Organizations: Ricoh Co., Ltd.

Abstract

Sparse autoencoder (SAE)-based activation steering is a lightweight approach for modifying the behavior of large language models (LLMs) by adding small vectors to hidden states during inference. However, designing effective steering interventions remains challenging. In this work, we develop an information-theoretic framework for analyzing SAE activations through empirical feature-firing distributions. The central object of the framework is the Jensen--Shannon divergence (JSD) between such firing distributions, which quantifies how far a generated or intervened state deviates from a dataset-induced target distribution. Based on an analytic divergence response for virtually shifted states, we construct steering vectors toward the target firing distribution. The same divergence can also be used to monitor deviation from another empirical distribution, such as a general valid-language distribution. We demonstrate the effectiveness of the steering in a question-mode generation task study. We further use teacher-forced pulse experiments to show that local interventions induce structured downstream JSD responses in later layers, and apply the framework to a safety-related target--contrast response setting for feature analysis.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines

    May 29, 2026Mikkel Godsk Jørgensen, Lars Kai HansenLanguage Model SteeringLLM Interpretability

  2. Steered Generation via Gradient-Based Optimization on Sparse Query Features

    May 21, 2026Sumanta Bhattacharyya, Pedram RooshenasConstrained Motion PlanningLanguage Model Steering