Selective Neuron Amplification in Transformer Language Models

Ryyan Akhtar; Payal Pahwa; Monika Arora

cs.LGApr 8, 2026

Selective Neuron Amplification in Transformer Language Models

Authors: Ryyan Akhtar, Payal Pahwa, Monika Arora

Organizations: Department of Artificial Intelligence and Data Science, Guru Gobind Singh Indraprastha University, BPIT, New Delhi, India

Abstract

Large language models often fail on tasks they seem to already understand. In our experiments, this appears to be less about missing knowledge and more about certain internal circuits not being strongly activated during inference. We explore Selective Neuron Amplification, which increases the influence of task relevant neurons without changing the model's parameters. The method works at inference time and does not permanently alter the model. SNA helps mainly when the model is uncertain, while having low effect when the model is already confident. This suggests that some model failures are due to weak activation rather than lack of capability.

Figures & tables

Figure 1: Inverse relationship between baseline confidence and SNA improvement across tasks.

Figure 2: SST-2 sentiment classification results at Layer 21

Causal Interventions in Language Models

Causal interventions in language models test and change how internal representations and computational components cause outputs.

See all 484 papers

Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts
May 24, 2026Xiangtian Ji, Yuxin Chen, Zhengzhou Cai +3Fine-Tuning LLM Interpretability
National University of Singapore · Beijing University of Posts and Telecommunications · University of Science and Technology of China
A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models
May 8, 2026Hamid Kazemi, Atoosa Chegini, Maria SafiLLM Interpretability Adversarial Attacks on LLMs
Apple · Work done during an internship at Apple. Present affiliation: University of Maryland, College Park.
Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models
Apr 29, 2026M. K. Khalidi Siam, Md. Tausif-Ul-Islam, Md. Reshad Romim Khan +5LLM Pruning Language Modeling
1BRAC University

May 24, 2026J/K move · Enter open · S save

Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts

Xiangtian Ji, Yuxin Chen, Zhengzhou Cai, Xiang Wang+2

National University of Singapore · Beijing University of Posts and Telecommunications · University of Science and Technology of China

Fine-Tuning LLM Interpretability LLM Fine-Tuning Neuron-Level Interpretability Mechanistic Interpretability Selective Fine-Tuning+1

Open paper AskPDF arXiv

Large language models (LLMs) display strong comprehensive abilities, yet the internal mechanisms that support these behaviors remain insufficiently understood. In this work, we show that across a wide range of open-weight Transformers, a subset of neurons remains consistently highly activated during inference across tasks of multiple capability dimensions. By probing along the cross-task activation strength, an extremely sparse subset is isolated, whose removal causes a collapse in model behavior, which we term keystone neurons. Our analysis reveals that keystone neurons are a stable and intrinsic neuron subset of the model that is largely established during pretraining. The parameters associated with these neurons are tightly calibrated during the training process, and their precise values are critical for the capabilities of the model. Building on these insights, we propose a supervised fine-tuning approach that updates only keystone neurons, achieving task gains comparable to or even better than full-parameter fine-tuning while better preserving performance in other capability dimensions, despite modifying a much smaller number of parameters.

Abstract

Causal Interventions in Language Models

Explore similar work

Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts

A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models

Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models