Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, yet the internal mechanisms driving these multi-step processes remain opaque. Existing mechanistic interpretability approaches often rely on superficial text-pattern co-occurrences or capture only short-term effects, struggling to trace the long-horizon, outcome-driven influence of internal components such as neurons or sparse features in multi-step reasoning.In this paper, we introduce Integrated Policy Gradient (IPG), a novel, training-free framework that brings policy-based attribution into mechanistic interpretability for LLM reasoning. Grounded in outcome-oriented and sequential-influence-aware principles, it backpropagates outcome-level, non-differentiable behavioral signals (e.g., reasoning outcome correctness) through entire inference trajectories, utilizing path integration to isolate outcome-relevant mediators at component level for reliable attribution. This provides a linear approximation of Causal Mediation Analysis (CMA) in reasoning. Empirical evaluations demonstrate that \method achieves precise localization and reveals mechanistic insights by identifying sparse, concentrated subsets of internal components associated with specific reasoning behaviors. Our method also enables modulation of reasoning capabilities, offering a powerful and reliable framework for understanding LLM reasoning. The project page is at https://foundation-model-research.github.io/IPG .
Figures & tables
Figure 1: Interpretability paradigms for reasoning in LLMs, regarding (i) our proposed Integrated Policy Gradient method that attributes outcome-level behavioral signals. t is the reasoning time step; (ii) text-pattern methods that associate high activations with special tokens (e.g., “Wait”); and (iii) control-vector methods that rely on human-craft contrastive input pairs.
Figure 2: SAE feature activation frequency across reasoning and non-reasoning domains with IPG -selected features (red) and all features (gray).
Task
Original
Random
IPG Feature
Reasoning
63.00
59.80 (-3.20)
28.80 (-34.2)
Non-Reasoning
12.11
9.81 (-2.30)
10.81 (-1.3)
Table 1: Ablation of IPG-identified components.
Method
Qwen2.5-Math-1.5B
Llama3.1-8B
GSM8K
MATH
AIME
GPQA
GSM8K
MATH
AIME
GPQA
Original
82.41 ± 0.65
63.00 ± 0.95
10.00 ± 3.17
24.75 ± 1.92
85.89 ± 0.72
41.80 ± 0.29
6.67 ± 3.17
23.23 ± 0.96
Random
82.86 ± 0.65
63.20 ± 0.88
16.67 ± 4.75
26.77 ± 1.68
86.20 ± 0.65
41.40 ± 0.29
6.67 ± 4.75
27.78 ± 0.96
Activation
82.56 ± 0.65
63.40 ± 1.33
20.00 ± 4.75
25.75 ± 1.92
85.75 ± 0.61
43.00 ± 0.19
13.33 ± 4.75
25.25 ± 0.72
R.N. ( Rai and Yao, 2024 )
82.26 ± 0.83
62.20 ± 1.14
13.33 ± 4.62
24.24 ± 1.92
86.05 ± 0.65
40.80 ± 1.23
6.67 ± 4.75
21.21 ± 2.16
C.V. ( Højer et al., 2025 )
83.85 ± 0.07
62.60 ± 1.04
16.67 ± 3.17
28.78 ± 1.44
85.97 ± 0.86
40.00 ± 0.57
6.67 ± 4.75
28.78 ± 0.48
Table 2: Accuracy (%) after enhancing components selected by IPG and baseline methods across reasoning benchmarks under greedy decoding. Results are reported for Qwen2.5-Math-1.5B-Instruct and Llama3.1-8B-Instruct. Values following ± denote the half-width of the 95% confidence interval, in percentage points. The best and second-best point estimates in each column are shown in bold and underlined , respectively; ties receive the same formatting.
Method
Qwen
DeepSeek
Random
0.064±0.023
0.064±0.023
IG ( Sundararajan et al., 2017 )
0.091±0.025
0.066±0.022
Neuron SHAP ( Ghorbani and Zou, 2020 )
0.072±0.020
0.065±0.005
AttnLRP ( Achtibat et al., 2024 )
0.082±0.018
0.062±0.013
IPG (Ours)
0.305±0.113
0.246±0.030
Table 3: Alignment with a group-based zero-ablation mediation proxy on a subset of MATH-500. Values are mean Overlap@k± standard deviation with k corresponding to approximately 5% of neurons. Qwen and DeepSeek denote Qwen2.5-Math-1.5B-Instruct and DeepSeek-R1-Distill-Qwen-1.5B, respectively.
Figure 3: Responses generated by Qwen2.5-Math-1.5B-Instruct ( Yang et al., 2024 ) on one example in GSM8K ( Cobbe et al., 2021 ) , including both original and intervened outputs, with the model’s reasoning ability elicited to arrive at the correct answer. More examples are in Appendix D.8.2 .
Figure 4: Effects of targeted interventions on reasoning performance. Left: impact of the intervention scaling factor on Qwen2.5-Math-1.5B-Instruct ( Yang et al., 2024 ) . Right: Reasoning strength control in GSM8K ( Cobbe et al., 2021 ) for DeepSeek-R1-Distilled-Qwen-1.5B ( Guo et al., 2025 ) .
Cross-Dataset
Cross-Domain
MATH-500 → GSM8K
MATH-500 → HumanEval
HumanEval → CSQA
Method
Acc. (%) ↑
Avg. Tok.
Acc. (%) ↑
Avg. Tok.
Acc. (%) ↑
Avg. Tok.
Original
82.41
319.68
11.58
212.12
21.14
317.05
Activation
82.94
317.17
12.80
161.93
20.00
276.58
R.N. ( Rai and Yao, 2024 )
82.87
312.63
15.24
163.98
20.26
258.78
C.V. ( Højer et al., 2025 )
83.39
320.03
14.02
195.33
20.96
275.47
Table 4: Cross-dataset (mathematics) and cross-domain (code and commonsense) transfer under greedy decoding. Best and second best results are shown in bold and underlined format. We provide the half-width 95% CI for our method.
Figure 5: Layer-wise localization and concentration of reasoning components. (a) The heatmap for reasoning task (MATH-500) performance enhancement across different network layers and intervention widths (number of targeted neurons). (b) The cumulative IPG scores for selected layers.
Figure 10
Model
GSM8K
MATH500
AIME2024
AIME2025
DeepSeek-R1-Distill-Qwen-1.5B
75.21
60.60
16.67
13.33
+ IPG Neuron Transferring
78.01
63.80
30.00
30.00
Table 6: Enhancing neurons from prompt-elicited to reason-distilled models without re-running IPG .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: K-SAE Training Curve
Attribution Configuration
Path integral steps
10
GRPO group size
3,5
Decoding Strategy
Top- p
0.8
Temperature
0.6
Max Generation Tokens
512,2048,8192
Appendix
Table 7: IPG-GRPO Hyperparameter Settings.
k
GSM8K
Acc. (Enhance) ↑
Original
82.41
8
84.38
32
84.76
64
83.70
Appendix
Table 8: Investigation of the impact of varying TopK parameters in k-SAE [ Belitsky et al., 2025 ] on controlling reasoning behavior in Qwen2.5-Math-1.5B-Instruct [ Yang et al., 2024 ] for GSM8K [ Cobbe et al., 2021 ] . TopK refers to preserving the top k features while deactivating others in the feature space, as described in Section 3.2 .
p
GSM8K
Acc. (Enhance) ↑
Original
82.41
10
82.94
20
83.85
30
83.78
40
83.85
Appendix
Table 9: Evaluation on the impact of varying the top- p number of neurons in 3.2 in Qwen2.5-Math-1.5B-Instruct [ Yang et al., 2024 ] for GSM8K [ Cobbe et al., 2021 ] .This evaluation is based on our k-SAE implementation [ Gao et al., 2025 ] with a fixed k=32 .
# Samples
GSM8K
Acc. (Enhance) ↑
1
82.79
10
83.17
20
83.17
50
83.54
100
83.39
Appendix
Table 10: Effect of batch size (# samples used to compute IPG directions) on steering performance. Model: Qwen2.5-Math-1.5B-Instruct on GSM8K [ Cobbe et al., 2021 ] . Original accuracy: 82.41%.
Model
Reward Method
GSM8K
MATH-500
AIME-2024
GPQA Diamond
Qwen2.5-Math-1.5B
Rule Function
84.15
64.20
20.00
26.77
Reward Model
84.60
65.80
20.00
35.35
Appendix
Table 11: Performance of IPG in enhancing accuracy across reasoning datasets for Qwen2.5-Math-1.5B-Instruct [ Yang et al., 2024 ] under different reward signals.
h′
GSM8K
Acc. (Enhance) ↑
Zero vector
84.60
Mean hidden state
84.45
Random noise
84.00
Appendix
Table 12: Sensitivity analysis on the choice of baseline representation h′ . The model is Qwen-Math-2.5-1.5B-Instruct [ Qwen Team, 2024 ] evaluated on GSM8K [ Cobbe et al., 2021 ] . Original accuracy before any intervention: 82.41%.
Method
GSM8K
MATH-500
AIME-2024
Original
93.83
76.38
25.07
IPG (Ours)
94.77
77.77
32.61
Appendix
Table 13: Comparison of IPG and baseline methods on pass@8 accuracy across reasoning datasets for Qwen2.5-Math-1.5B-Instruct [ Yang et al., 2024 ]
Method
MATH500 ↑
AIME2024 ↑
Original
63.00
10.00
PG
64.60
16.67
IPG
65.80
20.00
PG-SAE
64.40
16.67
IPG -SAE
66.00
20.00
Appendix
Table 14: Ablation study comparing raw policy gradient (PG) and integrated policy gradient (IPG) attribution on Qwen2.5-Math-1.5B-Instruct . Accuracy (%) is reported.
Method
GSM8K
MATH
AIME
GPQA
Original
82.41
63.00
10.00
24.75
IG
82.94
64.40
13.33
29.29
Neuron Shapley
83.47
64.80
13.33
30.30
AttnLRP
83.62
64.00
16.67
28.28
IPG
84.61
65.80
20.00
33.33
Appendix
Table 15: Accuracy (%) under neuron-space enhancement on Qwen2.5-Math-1.5B-Instruct. The highest accuracy in each column is shown in bold .
Figure 8: Dice similarity between neuron indices across datasets. We compare neuron indices discovered in 4 different datasets, highlighting shared reasoning structure across tasks, with diagonal showing random baseline.
Figure 9: SAE features 486 and 513 from layer 20 of Qwen2.5-Math-1.5B-Instruct, identified by IPG. Tokens receiving higher activation are shown in darker green.
Figure 10: SAE features 12581 and 13446 from layer 24 of Qwen2.5-Math-1.5B-Instruct, identified by IPG. Tokens receiving higher activation are shown in darker green.
Figure 11: SAE features 13499 and 24361 from layer 26 of Qwen2.5-Math-1.5B-Instruct, identified by IPG. Tokens receiving higher activation are shown in darker green.
Aspect
Correlation-Based (e.g., SAE-R)
IPG
Signal source
Text pattern co-occurrence
Reasoning outcome (accuracy/length)
Attribution target
Activates with reasoning tokens
Influences final outcome
Temporal scope
Local / point-wise
Long-horizon (full trajectory)
Appendix
Table 16: What IPG Provides Compared to Correlation-Based Methods
Models
Original
IPG (Enhanced)
ReFT
Qwen2.5-Math1.5B-Instruct
82.41
84.76
62.47
Llama3.1-8B-Instruct
85.89
87.41
77.94
Appendix
Table 17: Performance comparison across models under different methods.
Method
Accuracy (%) ↑
Reflection Tokens (Avg.) ↑
Original
16.67
113.9
Intervened (Ours)
20.00
156.5
Appendix
Table 18: Impact of intervening on reflection-based signals on AIME 2024. We report the task accuracy and the average number of reflection tokens per sample (calculated over 30 samples).
Method
Original (%)
Enhance ↑ (%)
Suppress ↓ (%)
Activation
89.52
89.64
89.05
IPG (Ours)
89.52
90.27
85.81
Appendix
Table 19: Comparison of IPG and activation-based methods for controlling harmfulness on Qwen2.5-1.5B-Instruct using the WildGuard benchmark [ Han et al., 2024 ] .
Model Setup
GPQA
AIME
MATH500
Qwen3-14B (Original)
27.27
33.33
73.60
Qwen3-14B ( IPG )
34.34
40.00
76.00
Δ Improvement
+7.07
+6.67
+2.40
Appendix
Table 20: Performance comparison of the baseline Qwen3-14B and the IPG -enhanced model (intervening on neurons) using greedy decoding. Metrics are reported as absolute accuracy (%).
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.