Recent studies show that LLM hidden states encode reward-related information, such as answer correctness and model confidence. However, existing approaches typically fit black-box probes on the full hidden states, offering little insight into how this information is structured across neurons. In this paper, we show that reward-related information is concentrated in a sparse subset of neurons. Using simple probing, we identify two types of neurons: value neurons, whose activations predict state value, and dopamine neurons, whose activations encode step-level temporal difference (TD) errors. Together, these neurons form a sparse reward subsystem within LLM hidden states. These names are drawn by analogy with neuroscience, where value neurons and dopamine neurons in the biological reward subsystem also encode value and reward prediction errors, respectively. We demonstrate that value neurons are robust and transferable across diverse datasets and models, and provide causal evidence that they encode reward-related information. Finally, we show applications of the reward subsystem: value neurons serve as effective predictors of model confidence, and dopamine neurons can function as a process reward model (PRM) to guide inference-time search. Our code is available at https://github.com/zou-group/reward_subsystem.
Figures & tables
Figure 1: Overview of the sparse reward subsystem in LLMs. The left panel illustrates a computational analogy with biological value and dopamine neurons. Value neurons encode the value of the current state, while dopamine neurons encode step-level TD errors. The signal illustrations show value estimates tracking the chance of success (§ 3 ) and dopamine-neuron activations exhibiting peaks and troughs at unexpected progress and reasoning errors (§ 4 ). We apply value neurons to confidence prediction (§ 5.1 ) and dopamine neurons to process-reward-guided search (§ 5.2 ).
Figure 2: AUC curves for layers 2–4 of the Qwen-2.5-14B-SimpleRL-Zoo model on the GSM8K and MATH500 datasets. The curves indicate that the value probe can accurately predict the value by relying on only a very small number of value neurons. Shaded regions indicate uncertainty.
Layer
Value Neurons
Random
NTP Neurons
Weight Magnitude
Wanda
2
37.0 (-38.2)
77.0 (+1.8)
76.4 (+1.2)
74.6 (-0.6)
74.8 (-0.4)
3
13.6 (-61.6)
73.4 (-1.8)
76.8 (+1.6)
61.6 (-13.6)
48.2 (-27.0)
4
29.4 (-45.8)
73.8 (-1.4)
77.4 (+2.2)
75.4 (+0.2)
67.0 (-8.2)
5
1.2 (-74.0)
74.4 (-0.8)
75.2 (+0.0)
75.0 (-0.2)
73.4 (-1.8)
Avg
20.3 (-54.9)
74.6 (-0.6)
76.4 (+1.2)
71.6 (-3.6)
65.8 (-9.4)
Table 1: Intervention results for the Qwen-2.5-7B-SimpleRL-Zoo model on the MATH500 dataset. Performance is measured by accuracy (%) after zeroing out a 1% subset of neurons in a single layer. The original accuracy is 75.2%.
MATH500
GSM8K
ARC
PopQA
EntityQuestions
Task type
Reasoning
Reasoning
Reasoning
Factual recall
Factual recall
Baseline
75.2
93.0
82.2
42.6
27.2
Value Neurons
37.0
52.8
66.2
46.2
24.2
Δ
−38.2
−40.2
−16.0
+3.6
−3.0
Table 2: Comparison of reasoning and factual-recall performance after zeroing out the top 1% of value neurons in layer 2 of Qwen-2.5-7B-SimpleRL-Zoo, using the probe trained on MATH. Performance is measured by accuracy (%). Δ denotes the change in percentage points.
Figure 3: IoU as a function of the pruning ratio. The IoU values for value neurons across different datasets and models are significantly higher than the random baseline.
Figure 4: Dopamine neurons encode information regarding the model’s prediction error for the current state. (a) Positive Surprise: The model initially lacks confidence in answering the problem but ultimately provides the correct solution. This neuron exhibits two significant peaks when the model identifies a critical logical step and subsequently derives the final key result. (b) Negative Surprise: Conversely, the model begins with high confidence but fails to solve the problem correctly. The neuron displays a distinct trough at the exact moment a logical flaw occurs.
Figure 5: Spearman correlation curves for Qwen-2.5-7B/14B-SimpleRL-Zoo on MATH500.
Model
Value Neurons
LCD [ 9 ]
Verbal Conf.
Token Conf.
Question Len.
MATH500
Llama-3.1-8B
0.69
0.64
0.69
0.54
0.63
Phi-3.5-mini
0.73
0.63
0.54
0.41
0.62
Gemma-3-4B
0.72
0.65
0.62
0.41
0.71
Qwen3.5-0.8B
0.71
0.70
0.41
0.56
0.77
ARC
Table 3: Comparison of AUC for different confidence prediction methods across models and benchmarks. Higher is better. Bold denotes the best performing method per model.
Method
Qwen-2.5-7B SimpleRL-Zoo
Qwen-2.5-14B SimpleRL-Zoo
Llama-3.1-8B Instruct
Average
Greedy
69.6
67.0
47.3
61.3
Random
77.3
74.8
46.3
66.1
Majority of N
64.8
76.0
54.2
65.0
Implicit PRM
69.8
74.8
45.8
63.5
Mean logprob
72.2
78.7
41.7
64.2
Min logprob
74.1
76.0
48.6
66.2
Table 4: Comparison of accuracy (%) on MATH500 for different methods.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Hyperparameter = Value
Generation
max_tokens_per_call \ =\ 16000; temperature \ =\ 1.0;
Table 7: Hyperparameters used for reward subsystem evaluation with neuron pruning.
Figure 6: AUC curves for layers 2–4 of the Qwen-2.5-7B-SimpleRL-Zoo model on the MBPP+ and IFEval datasets. Shaded regions indicate uncertainty.
Figure 7: AUC curves for layers 2–4 of the Qwen-3.5-0.8B, Phi-3.5-mini-instruct, Llama-3.1-8B-Instruct, and Qwen-2.5-14B-SimpleRL-Zoo models on the MATH500 and ARC dataset. Shaded regions indicate uncertainty.
Figure 8: AUC curves for different layers of the Qwen-2.5-7B/14B-SimpleRL-Zoo model on the GSM8K dataset.
Figure 9: IoU as a function of the pruning ratio. The IoU values for value neurons across different datasets are significantly higher than the random baseline, indicating that for the same LLM, the positions of identified value neurons are closely correlated across tasks.
Figure 10: Dopamine neurons encode information regarding the model’s prediction error for the current state. (a) Positive Surprise: The model initially lacks confidence in answering the problem but ultimately provides the correct solution. This neuron exhibits a significant peak when the model identifies a critical logical step and subsequently derives the final key result. (b) Negative Surprise: Conversely, the model begins with high confidence but fails to solve the problem correctly. The neuron displays a distinct trough at the exact moment a wrong step occurs.
Method
MATH500
GSM8K
ARC
Baseline
75.2
93.0
82.2
Massive Activations
76.0 (+0.8)
91.6 (-1.4)
83.0 (+0.8)
Super Weights
74.6 (-0.6)
91.2 (-1.8)
80.4 (-1.8)
Rogue Dimensions
65.8 (-9.4)
81.6 (-11.4)
79.8 (-2.4)
Attention Sink
73.2 (-2.0)
93.0 (+0.0)
81.6 (-0.6)
Value Neurons
37.0 (-38.2)
52.8 (-40.2)
66.2 (-16.0)
Appendix
Table 8: Additional intervention controls for layer 2 of Qwen-2.5-7B-SimpleRL-Zoo on MATH500, GSM8K, and ARC. Baseline denotes the unmodified model, and parentheses indicate changes in percentage points.
Figure 11: The activation curves of a dopamine neuron across states under different ablation conditions. The yellow line represents the original trajectory, the blue line shows the result of zeroing 20% random neurons, and the red line depicts the result of zeroing the top 20% value neurons. While random ablation has a minimal impact on the overall trend, value neuron ablation significantly alters the trajectory of the curve. This indicates a close correlation between value and dopamine neurons.
Figure 12: AUC curves at the final response position for four models on the Minerva Math dataset. The AUC remains mostly stable above 0.8 across a wide range of pruning ratios. Shaded regions indicate uncertainty.
Method
Accuracy (%)
Greedy
90.7
Random
90.7
Implicit PRM
92.8
Dopamine Neurons (MATH500-trained)
94.4
Appendix
Table 9: Cross-task evaluation of process-reward-guided search.
Method
AUC
Token Confidence
0.570
Verbal Confidence
0.640
Question Length
0.661
LCD
0.641
Value Neurons (MATH500-trained)
0.692
Appendix
Table 10: Cross-task evaluation of model confidence prediction.
Threshold τ
Mean ρ
0.1
0.152
0.2
0.267
0.3
0.281
0.4
0.193
Appendix
Table 11: Mean Spearman correlation for different TD-error thresholds on Qwen-2.5-7B-SimpleRL-Zoo with MATH500, averaged across all layers.
p
Random Baseline
Base vs SimpleRL-Zoo
Base vs PPO-Zero
SimpleRL-Zoo vs PPO-Zero
0.50
0.333
0.723
0.705
0.676
0.80
0.111
0.513
0.525
0.454
0.90
0.053
0.429
0.408
0.362
0.96
0.020
0.326
0.316
0.266
0.98
0.010
0.290
0.310
0.235
0.99
0.005
0.283
0.290
0.236
Appendix
Table 12: IoU between value-neuron sets identified in Qwen-2.5-7B and its two RL-tuned variants. Value probes are trained on MATH500, and results are averaged over layers 2–4. p denotes the pruning ratio.