Activation Steering

Follow this topic Ask about this topic

Activation steering studies how edits to a model’s hidden activations can control its behavior at inference time, enabling changes without retraining the full model. Common approaches add or remove contrastive steering vectors, target sparse autoencoder features, neurons or attention heads, and use conditional or feedback-based controllers to adapt interventions to the input or generation state. Researchers apply these methods to language tasks such as reasoning, refusal, factuality and hallucination mitigation, as well as image and video generation, speech, and embodied agents; evaluations include benchmarks such as GSM8K, TruthfulQA, POPE and LIBERO. A central challenge is that readable or decodable features are not necessarily controllable, and effective edits can vary by context and architecture or cause collateral changes to fluency, utility and safety.

Latest papers

376 papers77 papers in the last four weeks, up 93% on the four weeks before. 0.5% of all new papers.

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering
May 6, 2026Qiming Li, Zekai Ye, Xiaocheng Feng +9Hallucination in Language Models Large Vision-Language Models
1Harbin Institute of Technology · 2Peng Cheng Laboratory · 3Harbin Institute of Technology, Shenzhen
Steer Like the LLM: Activation Steering that Mimics Prompting
May 5, 2026Geert Heyman, Frederik VandeputteLanguage Model Steering LLM Prompting
Nokia Bell Labs, Belgium.
Pairwise matrices for sparse autoencoders: single-feature inspection mislabels causal axes
May 4, 2026Michael A. Riegler, Birk Sebastian Frostelid Torpmann-HagenSparse Autoencoders Mechanistic Interpretability
SimulaMet, Norway · Simula, Norway
SteeringDiffusion: A Bottlenecked Activation Control Interface for Diffusion Models
May 3, 2026Fangzheng Wu, Brian SummaDiffusion Model Fine-Tuning Activation Steering
Department of Computer Science Tulane University
Minimizing Collateral Damage in Activation Steering
May 1, 2026Tam Nguyen, Tu Anh Nguyen, Sina Alemohammad +1Language Model Steering Activation Steering
Department of Electrical & Computer Engineering, Rice University, Houston, USA · Department of Computational and Applied Mathematics, Rice University, Houston, USA · Department of Electrical & Computer Engineering, The University of Texas at Austin, Austin, USA
Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
Apr 28, 2026Chengsheng Zhang, Chenghao Sun, Xinyan Jiang +2Vision-Language Models Hallucination in Language Models
University of Science and Technology of China · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China · University of Chinese Academy of Sciences, Beijing, China
Contextual Linear Activation Steering of Language Models
Apr 27, 2026Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan +1Language Model Steering LLM Fine-Tuning
UC San Diego · MIT · Broad Institute of MIT and Harvard
Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMs
Apr 22, 2026Krishiv Agarwal, Ramneet Kaur, Colin Samplawski +6Language Model Safety Evaluation LLM Auditing
University of Florida · SRI
Discovering a Shared Logical Subspace: Steering LLM Logical Reasoning via Alignment of Natural-Language and Symbolic Views
Apr 21, 2026Feihao Fang, My T. Thai, Yuanyuan LeiSymbolic Reasoning Neuro-Symbolic Reasoning
University of Illinois Urbana-Champaign, Champaign, IL · Computer & Information Science and Engineering, University of Florida, Gainesville, FL
How Do Answer Tokens Read Reasoning Traces? Self-Reading Patterns in Thinking LLMs for Quantitative Reasoning
Apr 21, 2026Haoyang Chen, Yi Liu, Jianzhi Shao +3LLM Interpretability Mathematical Reasoning
State Key Laboratory for Novel Software Technology, Nanjing University, China · Alibaba Group, China · Ant Group, China
Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Control
Apr 21, 2026Julian Skifstad, Xinyue Annie Yang, Glen ChouLLM Alignment Language Model-Based Control
1Georgia Institute of Technology, Atlanta, GA, USA. Schools of Electrical and Computer Engineering · 2Computer Science · 3Cybersecurity and Privacy, Aerospace Engineering
From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models
Apr 20, 2026Qidong Wang, Junjie Hu, Ming JiangMulti-Task Learning Causal Attribution
Tongji University · University of Wisconsin-Madison
Surgical Repair of Insecure Code Generation in LLMs
Apr 17, 2026Gustavo Sandoval, Brendan Dolan-Gavitt, Siddharth GargMechanistic Interpretability Secure Code Generation
Tandon School of Engineering New York University · XBOW
RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning
Apr 17, 2026Jiayi Tian, Yupeng Su, Ryan Solgi +2LLM Routing Efficient Language Model Inference
University of California, Santa Barbara · Intel Labs
MidSteer: Optimal Affine Framework for Steering Generative Models
Apr 17, 2026Tatiana Gaintseva, Andrew Stepanov, Ziquan Liu +4Language Model Steering Generative Modeling
Queen Mary University of London, London, UK · Huawei Noah’s Ark, London, UK · QuantumLight Capital, London, UK +2
Predicting Where Steering Vectors Succeed
Apr 16, 2026Jayadev BillaLanguage Model Steering Representation Steering
FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models
Apr 16, 2026Zixuan Weng, Jinghuai Zhang, Kunlin Cai +3Language Model Steering LLM Safety
University of California, Los Angeles
From Weights to Activations: Is Steering the Next Frontier of Adaptation?
Apr 15, 2026Simon Ostermann, Daniil Gurgurov, Tanja Baeumel +4Online Language Model Adaptation Language Model Steering
Saarland University · German Research Center for Artificial Intelligence (DFKI) · Centre for European Research in Trusted AI (CERTAIN) +6
RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repair
Apr 14, 2026Jagadeesh Rachapudi, Pranav Singh, Ritali Vatsi +2Machine Unlearning LLM Unlearning
Indian Institute of Technology Mandi, Mandi, India
Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors
Apr 14, 2026Rui Yin, Tianxu Han, Naen Xu +8LLM Backdoor Attacks Language Model Editing
Zhejiang University · Palo Alto Networks · OPPO Research Institute +2
Beyond Prompt: Fine-grained Simulation of Cognitively Impaired Standardized Patients via Stochastic Steering
Apr 14, 2026Weikang Zhang, Zimo Zhu, Zhichuan Yang +3Large Language Model-Based Role-Play Simulation Language Model Steering
School of Software, Xi’an Jiaotong University · Institute of Data Science, National University of Singapore · College of Computer Science, Sichuan University +1
From Attribution to Action: A Human-Centered Application of Activation Steering
Apr 13, 2026Tobias Labarta, Maximilian Dreyer, Katharina Weitz +2Explainable Artificial Intelligence Activation Steering
Fraunhofer Heinrich-Hertz-Institut · Technische Universität Berlin · BIFOLD – Berlin Institute for the Foundations of Learning and Data
Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs
Apr 13, 2026Wenkai Li, Fan Yang, Shaunak A. Mehta +1Language Model Safety Evaluation LLM Safety
Carnegie Mellon University · Fujitsu Research of America Inc.
A Systematic Analysis of the Impact of Persona Steering on LLM Capabilities
Apr 13, 2026Jiaqi Chen, Ming Wang, Tingna Xie +2Language Model Steering Personality Modeling in Language Models
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · School of Computing and Information Systems, Singapore Management University, Singapore 178902, Singapore · School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China
Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion
Apr 11, 2026Vishal Pramanik, Maisha Maliha, Susmit Jha +1Adversarial Attacks on LLMs LLM Jailbreak Attacks
Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL 32611, USA · School of Computer Science, University of Oklahoma, Norman, OK 73019, USA · Computer Science Laboratory, SRI International, Menlo Park, CA 94025, USA
Steered LLM Activations are Non-Surjective
Apr 10, 2026Aayush Mishra, Daniel Khashabi, Anqi LiuActivation Steering LLM Interpretability
Johns Hopkins University
SHIFT: Steering Hidden Intermediates in Flow Transformers
Apr 10, 2026Nina Konovalova, Andrey Kuznetsov, Aibek AlanovControllable Image Generation Concept Erasure
FusionBrain Lab · HSE University · AXXX
Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs
Apr 10, 2026Jinqi Luo, Jinyu Yang, Tal Neiman +5MLLM Safety Multimodal Large Language Models
University of Pennsylvania · Amazon · University of Central Florida
Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence
Apr 9, 2026Niklas Herbster, Martin Zborowski, Alberto Tosato +2LLM Safety Alignment Language Model Robustness
Tara Research · Technical University of Munich · Mila Quebec AI Institute
Selective Neuron Amplification in Transformer Language Models
Apr 8, 2026Ryyan Akhtar, Payal Pahwa, Monika AroraCausal Interventions in Language Models Activation Steering
Department of Artificial Intelligence and Data Science, Guru Gobind Singh Indraprastha University, BPIT, New Delhi, India

May 6, 2026J/K move · Enter open · S save

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering

Qiming Li, Zekai Ye, Xiaocheng Feng, Weihong Zhong+8

1Harbin Institute of Technology · 2Peng Cheng Laboratory · 3Harbin Institute of Technology, Shenzhen

Hallucination in Language Models Large Vision-Language Models Object Hallucination in VLMs Activation Steering VLM Hallucination Mitigation

Open paper AskPDF arXiv

Although Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on downstream tasks, they frequently produce contents that deviate from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or decoding strategies which significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly enhanced when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-guided Visual Attention Steering (CAST), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern corresponding to caption queries to enhance LVLMs' visual perception capability. Specifically, we use probing techniques to identify attention heads that are highly sensitive to caption queries and estimate optimized steering directions for their outputs. This steering strengthens LVLM's fine-grained visual perception capabilities, thereby effectively mitigating object hallucination. CAST reduced object hallucination by an average of 6.03% across five widely used LVLMs and five benchmarks including both discriminative and generative tasks, demonstrating state-of-the-art performance while adding little inference cost and preserving other foundational capabilities.