Large language models (LLMs) have demonstrated strong capabilities in complex reasoning tasks, yet their decision-making processes remain difficult to interpret. Existing explanation methods often lack trustworthy structural insight and are limited to single-instance analysis, failing to reveal reasoning stability and systematic failure mechanisms. To address these limitations, we propose the Trustworthy Unified Explanation Framework (TRUE), which integrates executable reasoning verification, feasible-region directed acyclic graph (DAG) modeling, and causal failure mode analysis. At the instance level, we redefine reasoning traces as executable process specifications and introduce blind execution verification to assess operational validity. At the local structural level, we construct feasible-region DAGs via structure-consistent perturbations, enabling explicit characterization of reasoning stability and the executable region in the local input space. At the class level, we introduce a causal failure mode analysis method that identifies recurring structural failure patterns and quantifies their causal influence using Shapley values. Extensive experiments across multiple reasoning benchmarks demonstrate that the proposed framework provides multi-level, verifiable explanations, including executable reasoning structures for individual instances, feasible-region representations for neighboring inputs, and interpretable failure modes with quantified importance at the class level. These results establish a unified and principled paradigm for improving the interpretability and reliability of LLM reasoning systems.
Figures & tables
Figure 1: Generation and verification of executable explanations. Given an input instance, the LLM interpreter produces a structured executable explanation consisting of explicit reasoning steps. The explanation is executed by a blind verifier to obtain a predicted answer, while white-box tools independently verify correctness of each reasoning step.
Figure 2: Construction of the local feasible-region graph via structure-preserving perturbations. Verified perturbed samples are aggregated into a directed acyclic graph, where nodes represent reasoning steps and edges encode their dependencies.
Figure 3: Overview of the proposed failure mode discovery and attribution framework. Given a sample problem class, semantically similar instances are grouped via LLM-assisted clustering, followed by systematic failure mode discovery. Controlled perturbations and mixed-mode testing are then performed to evaluate reasoning behavior under failure combinations. Finally, Shapley value attribution quantifies the causal impact of each failure mode on prediction outcomes.
MATH
MMLU
GSM8K
Method
EA
OA
EC
ERR
EA
OA
EC
ERR
EA
OA
EC
ERR
CoT
73.0
76.0
96.1
0.0
69.5
72.5
95.9
0.0
92.0
93.0
97.8
14.3
Self-Refine
72.5
75.0
96.7
0.0
63.0
66.0
91.7
7.4
90.5
92.0
98.4
0.0
Zero-Shot-CoT
74.5
78.5
93.6
4.7
59.0
69.0
83.3
4.8
93.5
95.0
97.9
10.0
Plan-and-Solve
70.0
73.0
95.2
1.9
56.0
72.5
72.4
12.7
92.5
93.0
98.4
14.3
Avg.
72.5
75.6
95.4
1.7
61.9
70.0
85.8
6.2
92.1
93.2
98.1
9.6
Table 1: Executable Explanation Evaluation across six benchmarks (%). Base model: GPT-4o-mini, N=200 per dataset.
MATH
MMLU
Provider
Model
EA
OA
EC
ERR
EA
OA
EC
ERR
OpenAI
GPT-4o-mini
71.5
74.0
95.9
1.9
70.0
72.0
95.8
3.6
GPT-4.1-nano
65.5
70.5
90.1
6.8
66.0
69.0
91.3
9.7
GPT-5-nano
93.0
97.0
95.9
0.0
73.5
75.0
98.0
0.0
Google
Gemini-2.0-Flash
46.0
48.0
95.8
0.0
65.0
74.5
81.2
17.6
Alibaba
Qwen3.5-35B
98.0
98.5
99.0
33.3
80.5
81.5
98.8
0.0
Table 2: Executable Explanation Evaluation across seven model APIs on four benchmarks (CoT, N=200 , verifier: GPT-4o-mini). Models grouped by provider; horizontal rules separate provider families. Upper block: MATH and MMLU; lower block: BBH and CRUXEval.
Category
Pert. SR
Test SR
DAG CE ↓
Baseline CE ↓
Δ CE
MATH
Prealgebra
50.0%
87.5%
1.05
11.51
+10.46
Number Theory
83.3%
75.0%
0.75
1.39
+0.64
Intermediate Alg.
33.3%
50.0%
0.30
0.75
+0.45
Counting & Prob.
50.0%
75.0%
1.72
8.69
+6.97
Precalculus
16.7%
37.5%
0.43
0.79
+0.36
Table 3: Success Rate Prediction Results on MATH and MMLU-CF. Lower CE indicates better calibration. Δ CE = Baseline CE − DAG CE.
MATH Dataset
MMLU-CF Dataset
Perturbation
Nodes
Edges
Pret.(%)
GT(%)
Nodes
Edges
Pret.(%)
GT(%)
Mild
9
35
51.9
34.8
18
29
50.8
48.3
Moderate
10
35
74.2
62.9
23
30
58.8
53.1
Aggressive
13
31
96.4
96.4
26
32
66.6
61.3
Table 4: DAG trajectory coverage results on MATH and MMLU-CF datasets. Nodes and Edges denote the size of the feasible-region DAG. Pret.(%) represents the average trajectory coverage rate of perturbed instances, while GT(%) denotes the coverage rate of ground-truth reasoning trajectories.
Figure 4: Cluster-level failure mode profiles on MATH, BBH, and CRUXEval. Top-3 failure modes per cluster ranked by Shapley value ϕ . Reasoning failures concentrate in domain-specific structural patterns rather than distributing uniformly.
Table 6: Ablation over prompting strategies: Original Accuracy (OA%) and Hidden Failure Rate (HFR% =1− EC). Lower HFR indicates more faithful reasoning. Best HFR per dataset in bold .
Figure 5: Error breakdown (% of n=200 samples) for CoT. FP ≈0 across all tasks; FN (hidden failures) dominates on HotpotQA.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
Existing explainability methods for Large Language Models (LLMs) typically treat hidden states as static points in activation space, assuming that correct and incorrect inferences can be separated using representations from an individual layer. However, these activations are saturated with polysemantic features, leading to linear probes learning surface-level lexical patterns rather than underlying reasoning structures. We introduce Truth as a Trajectory (TaT), which models the transformer inference as an unfolded trajectory of iterative refinements, shifting analysis from static activations to layer-wise geometric displacement. By analyzing displacement of representations across layers, TaT uncovers geometric invariants that distinguish valid reasoning from spurious behavior. We evaluate TaT across dense and Mixture-of-Experts (MoE) architectures on benchmarks spanning commonsense reasoning, question answering, and toxicity detection. Without access to the activations themselves and using only changes in activations across layers, we show that TaT effectively mitigates reliance on static lexical confounds, outperforming conventional probing, and establishes trajectory analysis as a complementary perspective on LLM explainability.
Hamed Damirchi, Ignacio Meza De la Jara, Ehsan Abbasnejad +3
Australian Institute for Machine Learning, Adelaide University · Monash University · Concordia University
Large Language Models (LLMs) achieve strong performance through extended inference-time deliberation, yet how their reasoning failures arise remains poorly understood. By analyzing model-generated reasoning trajectories, we find that errors are not uniformly distributed but often originate from a small number of early transition points, after which reasoning remains locally coherent but globally incorrect. These transitions coincide with localized spikes in token-level entropy, and alternative continuations from the same intermediate state can still lead to correct solutions. Based on these observations, we introduce GUARD, a targeted inference-time framework that probes and redirects critical transitions using uncertainty signals. Empirical evaluations across multiple benchmarks confirm that interventions guided by these failure dynamics lead to more reliable reasoning outcomes. Our findings highlight the importance of understanding when and how reasoning first deviates, complementing existing approaches that focus on scaling inference-time computation.
Wei Zhu, Jian Zhang, Lixing Yu +2
School of Information Science and Engineering, Yunnan University, Kunming, China · Yunnan Key Laboratory of Intelligent Systems and Computing, Kunming, China
Large language models (LLMs) solve complex problems by generating multi-step reasoning traces. Yet these traces are typically analyzed from only one of two perspectives: the sequence of tokens across different reasoning steps in the generated text, or the hidden-state vectors across model layers within one step. We introduce PRISM (Probabilistic Reasoning Inspection through Semantic and Implicit Modeling), a framework and diagnostic tool for jointly analyzing both levels, providing a unified view of how reasoning evolves across steps and layers. Across multiple reasoning models and benchmarks, PRISM uncovers systematic patterns in the reasoning process, showing that failed trajectories are more likely to become trapped in unproductive verification loops and further diverge into distinct modes such as overthinking and premature commitment, which behave differently once a candidate answer is reached. It further reveals how prompting reshapes reasoning behavior beyond aggregate accuracy by altering both semantic transitions and internal computational patterns. By modeling reasoning trajectories as structured processes, PRISM makes these behaviors observable and analyzable rather than relying solely on final-task accuracy. Taken together, these insights position PRISM as a practical tool for analyzing and diagnosing reasoning processes in LLMs.