Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, yet the internal mechanisms driving these multi-step processes remain opaque. Existing mechanistic interpretability approaches often rely on superficial text-pattern co-occurrences or capture only short-term effects, struggling to trace the long-horizon, outcome-driven influence of internal components such as neurons or sparse features in multi-step reasoning.In this paper, we introduce Integrated Policy Gradient (IPG), a novel, training-free framework that brings policy-based attribution into mechanistic interpretability for LLM reasoning. Grounded in outcome-oriented and sequential-influence-aware principles, it backpropagates outcome-level, non-differentiable behavioral signals (e.g., reasoning outcome correctness) through entire inference trajectories, utilizing path integration to isolate outcome-relevant mediators at component level for reliable attribution. This provides a linear approximation of Causal Mediation Analysis (CMA) in reasoning. Empirical evaluations demonstrate that \method achieves precise localization and reveals mechanistic insights by identifying sparse, concentrated subsets of internal components associated with specific reasoning behaviors. Our method also enables modulation of reasoning capabilities, offering a powerful and reliable framework for understanding LLM reasoning. The project page is at https://foundation-model-research.github.io/IPG .
Figures & tables
Figure 1: Interpretability paradigms for reasoning in LLMs, regarding (i) our proposed Integrated Policy Gradient method that attributes outcome-level behavioral signals. t is the reasoning time step; (ii) text-pattern methods that associate high activations with special tokens (e.g., “Wait”); and (iii) control-vector methods that rely on human-craft contrastive input pairs.
Figure 2: SAE feature activation frequency across reasoning and non-reasoning domains with IPG -selected features (red) and all features (gray).
Task
Original
Random
IPG Feature
Reasoning
63.00
59.80 (-3.20)
28.80 (-34.2)
Non-Reasoning
12.11
9.81 (-2.30)
10.81 (-1.3)
Table 1: Ablation of IPG-identified components.
Method
Qwen2.5-Math-1.5B
Llama3.1-8B
GSM8K
MATH
AIME
GPQA
GSM8K
MATH
AIME
GPQA
Original
82.41 ± 0.65
63.00 ± 0.95
10.00 ± 3.17
24.75 ± 1.92
85.89 ± 0.72
41.80 ± 0.29
6.67 ± 3.17
23.23 ± 0.96
Random
82.86 ± 0.65
63.20 ± 0.88
16.67 ± 4.75
26.77 ± 1.68
86.20 ± 0.65
41.40 ± 0.29
6.67 ± 4.75
27.78 ± 0.96
Activation
82.56 ± 0.65
63.40 ± 1.33
20.00 ± 4.75
25.75 ± 1.92
85.75 ± 0.61
43.00 ± 0.19
13.33 ± 4.75
25.25 ± 0.72
R.N. ( Rai and Yao, 2024 )
82.26 ± 0.83
62.20 ± 1.14
13.33 ± 4.62
24.24 ± 1.92
86.05 ± 0.65
40.80 ± 1.23
6.67 ± 4.75
21.21 ± 2.16
C.V. ( Højer et al., 2025 )
83.85 ± 0.07
62.60 ± 1.04
16.67 ± 3.17
28.78 ± 1.44
85.97 ± 0.86
40.00 ± 0.57
6.67 ± 4.75
28.78 ± 0.48
Table 2: Accuracy (%) after enhancing components selected by IPG and baseline methods across reasoning benchmarks under greedy decoding. Results are reported for Qwen2.5-Math-1.5B-Instruct and Llama3.1-8B-Instruct. Values following ± denote the half-width of the 95% confidence interval, in percentage points. The best and second-best point estimates in each column are shown in bold and underlined , respectively; ties receive the same formatting.
Method
Qwen
DeepSeek
Random
0.064±0.023
0.064±0.023
IG ( Sundararajan et al., 2017 )
0.091±0.025
0.066±0.022
Neuron SHAP ( Ghorbani and Zou, 2020 )
0.072±0.020
0.065±0.005
AttnLRP ( Achtibat et al., 2024 )
0.082±0.018
0.062±0.013
IPG (Ours)
0.305±0.113
0.246±0.030
Table 3: Alignment with a group-based zero-ablation mediation proxy on a subset of MATH-500. Values are mean Overlap@k± standard deviation with k corresponding to approximately 5% of neurons. Qwen and DeepSeek denote Qwen2.5-Math-1.5B-Instruct and DeepSeek-R1-Distill-Qwen-1.5B, respectively.
Figure 3: Responses generated by Qwen2.5-Math-1.5B-Instruct ( Yang et al., 2024 ) on one example in GSM8K ( Cobbe et al., 2021 ) , including both original and intervened outputs, with the model’s reasoning ability elicited to arrive at the correct answer. More examples are in Appendix D.8.2 .
Figure 4: Effects of targeted interventions on reasoning performance. Left: impact of the intervention scaling factor on Qwen2.5-Math-1.5B-Instruct ( Yang et al., 2024 ) . Right: Reasoning strength control in GSM8K ( Cobbe et al., 2021 ) for DeepSeek-R1-Distilled-Qwen-1.5B ( Guo et al., 2025 ) .
Cross-Dataset
Cross-Domain
MATH-500 → GSM8K
MATH-500 → HumanEval
HumanEval → CSQA
Method
Acc. (%) ↑
Avg. Tok.
Acc. (%) ↑
Avg. Tok.
Acc. (%) ↑
Avg. Tok.
Original
82.41
319.68
11.58
212.12
21.14
317.05
Activation
82.94
317.17
12.80
161.93
20.00
276.58
R.N. ( Rai and Yao, 2024 )
82.87
312.63
15.24
163.98
20.26
258.78
C.V. ( Højer et al., 2025 )
83.39
320.03
14.02
195.33
20.96
275.47
Table 4: Cross-dataset (mathematics) and cross-domain (code and commonsense) transfer under greedy decoding. Best and second best results are shown in bold and underlined format. We provide the half-width 95% CI for our method.
Figure 5: Layer-wise localization and concentration of reasoning components. (a) The heatmap for reasoning task (MATH-500) performance enhancement across different network layers and intervention widths (number of targeted neurons). (b) The cumulative IPG scores for selected layers.
Figure 10
Model
GSM8K
MATH500
AIME2024
AIME2025
DeepSeek-R1-Distill-Qwen-1.5B
75.21
60.60
16.67
13.33
+ IPG Neuron Transferring
78.01
63.80
30.00
30.00
Table 6: Enhancing neurons from prompt-elicited to reason-distilled models without re-running IPG .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: K-SAE Training Curve
Attribution Configuration
Path integral steps
10
GRPO group size
3,5
Decoding Strategy
Top- p
0.8
Temperature
0.6
Max Generation Tokens
512,2048,8192
Appendix
Table 7: IPG-GRPO Hyperparameter Settings.
k
GSM8K
Acc. (Enhance) ↑
Original
82.41
8
84.38
32
84.76
64
83.70
Appendix
Table 8: Investigation of the impact of varying TopK parameters in k-SAE [ Belitsky et al., 2025 ] on controlling reasoning behavior in Qwen2.5-Math-1.5B-Instruct [ Yang et al., 2024 ] for GSM8K [ Cobbe et al., 2021 ] . TopK refers to preserving the top k features while deactivating others in the feature space, as described in Section 3.2 .
p
GSM8K
Acc. (Enhance) ↑
Original
82.41
10
82.94
20
83.85
30
83.78
40
83.85
Appendix
Table 9: Evaluation on the impact of varying the top- p number of neurons in 3.2 in Qwen2.5-Math-1.5B-Instruct [ Yang et al., 2024 ] for GSM8K [ Cobbe et al., 2021 ] .This evaluation is based on our k-SAE implementation [ Gao et al., 2025 ] with a fixed k=32 .
# Samples
GSM8K
Acc. (Enhance) ↑
1
82.79
10
83.17
20
83.17
50
83.54
100
83.39
Appendix
Table 10: Effect of batch size (# samples used to compute IPG directions) on steering performance. Model: Qwen2.5-Math-1.5B-Instruct on GSM8K [ Cobbe et al., 2021 ] . Original accuracy: 82.41%.
Model
Reward Method
GSM8K
MATH-500
AIME-2024
GPQA Diamond
Qwen2.5-Math-1.5B
Rule Function
84.15
64.20
20.00
26.77
Reward Model
84.60
65.80
20.00
35.35
Appendix
Table 11: Performance of IPG in enhancing accuracy across reasoning datasets for Qwen2.5-Math-1.5B-Instruct [ Yang et al., 2024 ] under different reward signals.
h′
GSM8K
Acc. (Enhance) ↑
Zero vector
84.60
Mean hidden state
84.45
Random noise
84.00
Appendix
Table 12: Sensitivity analysis on the choice of baseline representation h′ . The model is Qwen-Math-2.5-1.5B-Instruct [ Qwen Team, 2024 ] evaluated on GSM8K [ Cobbe et al., 2021 ] . Original accuracy before any intervention: 82.41%.
Method
GSM8K
MATH-500
AIME-2024
Original
93.83
76.38
25.07
IPG (Ours)
94.77
77.77
32.61
Appendix
Table 13: Comparison of IPG and baseline methods on pass@8 accuracy across reasoning datasets for Qwen2.5-Math-1.5B-Instruct [ Yang et al., 2024 ]
Method
MATH500 ↑
AIME2024 ↑
Original
63.00
10.00
PG
64.60
16.67
IPG
65.80
20.00
PG-SAE
64.40
16.67
IPG -SAE
66.00
20.00
Appendix
Table 14: Ablation study comparing raw policy gradient (PG) and integrated policy gradient (IPG) attribution on Qwen2.5-Math-1.5B-Instruct . Accuracy (%) is reported.
Method
GSM8K
MATH
AIME
GPQA
Original
82.41
63.00
10.00
24.75
IG
82.94
64.40
13.33
29.29
Neuron Shapley
83.47
64.80
13.33
30.30
AttnLRP
83.62
64.00
16.67
28.28
IPG
84.61
65.80
20.00
33.33
Appendix
Table 15: Accuracy (%) under neuron-space enhancement on Qwen2.5-Math-1.5B-Instruct. The highest accuracy in each column is shown in bold .
Figure 8: Dice similarity between neuron indices across datasets. We compare neuron indices discovered in 4 different datasets, highlighting shared reasoning structure across tasks, with diagonal showing random baseline.
Figure 9: SAE features 486 and 513 from layer 20 of Qwen2.5-Math-1.5B-Instruct, identified by IPG. Tokens receiving higher activation are shown in darker green.
Figure 10: SAE features 12581 and 13446 from layer 24 of Qwen2.5-Math-1.5B-Instruct, identified by IPG. Tokens receiving higher activation are shown in darker green.
Figure 11: SAE features 13499 and 24361 from layer 26 of Qwen2.5-Math-1.5B-Instruct, identified by IPG. Tokens receiving higher activation are shown in darker green.
Aspect
Correlation-Based (e.g., SAE-R)
IPG
Signal source
Text pattern co-occurrence
Reasoning outcome (accuracy/length)
Attribution target
Activates with reasoning tokens
Influences final outcome
Temporal scope
Local / point-wise
Long-horizon (full trajectory)
Appendix
Table 16: What IPG Provides Compared to Correlation-Based Methods
Models
Original
IPG (Enhanced)
ReFT
Qwen2.5-Math1.5B-Instruct
82.41
84.76
62.47
Llama3.1-8B-Instruct
85.89
87.41
77.94
Appendix
Table 17: Performance comparison across models under different methods.
Method
Accuracy (%) ↑
Reflection Tokens (Avg.) ↑
Original
16.67
113.9
Intervened (Ours)
20.00
156.5
Appendix
Table 18: Impact of intervening on reflection-based signals on AIME 2024. We report the task accuracy and the average number of reflection tokens per sample (calculated over 30 samples).
Method
Original (%)
Enhance ↑ (%)
Suppress ↓ (%)
Activation
89.52
89.64
89.05
IPG (Ours)
89.52
90.27
85.81
Appendix
Table 19: Comparison of IPG and activation-based methods for controlling harmfulness on Qwen2.5-1.5B-Instruct using the WildGuard benchmark [ Han et al., 2024 ] .
Model Setup
GPQA
AIME
MATH500
Qwen3-14B (Original)
27.27
33.33
73.60
Qwen3-14B ( IPG )
34.34
40.00
76.00
Δ Improvement
+7.07
+6.67
+2.40
Appendix
Table 20: Performance comparison of the baseline Qwen3-14B and the IPG -enhanced model (intervening on neurons) using greedy decoding. Metrics are reported as absolute accuracy (%).
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
Latent reasoning enables Large Language Models (LLMs) to perform multi-step inference within continuous hidden states, offering efficiency gains over explicit Chain-of-Thought (CoT). However, the opacity of these continuous thought vectors hinders their reliability and controllability. This paper bridges the gap between mechanistic interpretability and actionable control. We first present a systematic analysis using structural, causal, and geometric probes, revealing that latent vectors encode compressed, faithful representations of reasoning steps, with early vectors acting as critical causal hubs. Building on this, we operationalize these interpretability insights into a suite of training-free, decode-time interventions that refine the latent reasoning process by imposing the identified geometric and semantic priors. Extensive experiments across multiple model scales and diverse task domains demonstrate that our approaches consistently improve reasoning accuracy. Our interpretability-guided interventions consistently unlock latent capabilities and improve reasoning accuracy without any parameter updates.
Shuochen Chang, Tong Bai, Xiaofeng Zhang +5
1Shanghai Jiao Tong University · 2Fudan University
Understanding how LLMs reason is hindered by a practical asymmetry: while their generated outputs are observable, the underlying reasoning patterns remain opaque. Relying on single probes, such as Mutual Information Peak (MIP) or Deep-Thinking Ratio (DTR), risks underestimating the genuine inferential structure. To response this deficiency, we present an Integrated, cross-Architecture Reasoning (IAR) framework, designed to provide a unified approach to LLM reasoning interpretability. Specifically, we first propose to use bandwidth-calibrated MIP coupled with Tukey IQR peak-detection to isolate reasoning-crucial tokens at the output layer. Second, we performed an overlap analysis between MIP-picked tokens and DTR-deep tokens to trace the cross-layer trajectories of those tokens. This also discloses whether reasoning-crucial tokens are computation-intensive as well, further facilitating to understand how reasoning patterns evolve across model layers. Finally, we apply a Jaccard stability metric over multi-domain problems to verify if the MIP-identified tokens are reasoning quality-guaranteed. Extensive experiments on six models across four domains (mathematics, code, logic, and common sense) demonstrate IAR's generalizable interpretation capabilities across architectures. Our code is available at https://github.com/LeonardoMatthew/IAR.
Leonardo Matthew Yauw, Wei-Bin Kou, Yujiu Yang
Tsinghua Shenzhen International Graduate School · Harbin Institute of Technology, Shenzhen
Large language models (LLMs) increasingly rely on chain-of-thought (CoT) reasoning to solve complex tasks, yet a reasoning trace may merely accompany the final answer rather than causally contribute to it or faithfully reflect how it was reached. We introduce differentiable attention manipulation to train salient reasoning with reinforcement learning -- method which we call AtManRL. By optimizing an additive attention mask that identifies the CoT tokens needed to recover the model's answer, we derive a saliency reward and combine it with an outcome reward in GRPO with decoupled advantages. Because both saliency and faithfulness depend strongly on trace length, we evaluate against a sweep of length-penalty baselines that traces the trade-off a pure length penalty can reach, rather than against a single baseline. Across three models (Qwen2.5-3B, Granite-3.3-2B, and Phi-4-mini) and four benchmarks (GSM8K, MMLU, ProofWriter, and QuestBench), AtManRL sits above this trade-off curve by 3.9-4.0pp in saliency, measured as answer changes under mistake injection, and by 5.7-7.7pp in faithfulness, measured as hint verbalization under hint injection, while matching the accuracy of baselines of similar length. These results suggest that training explicitly for saliency is a promising route towards chain-of-thought reasoning that is more faithful to the model's actual decision process.
Max Henning Höth, Kristian Kersting, Björn Deiseroth +1
Aleph Alpha Research Lab1141 · TU Darmstadt Hessian.AI Lab1141