cs.CLFeb 1, 2026

Sparse Reward Subsystem in Large Language Models

Authors: Guowei Xu, Mert Yuksekgonul, James Zou

Organizations: Tsinghua University · Stanford University

Abstract

Recent studies show that LLM hidden states encode reward-related information, such as answer correctness and model confidence. However, existing approaches typically fit black-box probes on the full hidden states, offering little insight into how this information is structured across neurons. In this paper, we show that reward-related information is concentrated in a sparse subset of neurons. Using simple probing, we identify two types of neurons: value neurons, whose activations predict state value, and dopamine neurons, whose activations encode step-level temporal difference (TD) errors. Together, these neurons form a sparse reward subsystem within LLM hidden states. These names are drawn by analogy with neuroscience, where value neurons and dopamine neurons in the biological reward subsystem also encode value and reward prediction errors, respectively. We demonstrate that value neurons are robust and transferable across diverse datasets and models, and provide causal evidence that they encode reward-related information. Finally, we show applications of the reward subsystem: value neurons serve as effective predictors of model confidence, and dopamine neurons can function as a process reward model (PRM) to guide inference-time search. Our code is available at https://github.com/zou-group/reward_subsystem.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts

    May 24, 2026Xiangtian Ji, Yuxin Chen, Zhengzhou Cai +3Fine-TuningLLM Interpretability

  2. Automatically Finding Reward Model Biases

    Feb 16, 2026Atticus Wang, Iván Arcuschin, Arthur ConmyReward ModelingLLM Auditing

  3. Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability

    Oct 1, 2026Yu Mao, Lei Yu, Zining Zhu +6RL for Language ModelsReinforcement Learning