cs.CLOct 15, 2026
SaveARBOR: Locating Reasoning Failures in Long Chains of Thought through Reasoning State Tracking
Organizations: University of Southern California · The Chinese University of Hong Kong · New York University
Abstract
Large reasoning models solve complex problems through long chains of thought, in which prior work has documented failure modes such as overthinking, underthinking, and unsupported deductions. Existing methods detect these failures from local signals: textual similarity to nearby steps, switch tokens, or an LLM's judgment of a step's correctness. However, what a step contributes to the solution depends on the state that earlier steps have established, so identifying failures requires a global view that tracks that state across the trajectory. Therefore, we propose ARBOR, a reasoning-failure attribution framework that represents each trajectory as a tree of reasoning states, built as each step creates, verifies, prunes, or revisits a state, and attributes each failure to a relation between a step and the state preceding it. Using ARBOR, we refine the three failure modes into distinct subtypes that harm reasoning through different mechanisms and at different costs. We find that the tree state ARBOR maintains before a step separates failure steps from the at-risk steps at which the failure did not occur. We discuss how ARBOR integrates with existing mitigations at inference time and in training data. Code, data and interactive examples are available at https://inklab.usc.edu/ARBOR.
Figures & tables
Figure 1: How a step relates to the states already established defines a reasoning failure. (a) An excerpt of a Qwen3-8B trajectory on a ZebraLogic puzzle; numbers are step indices, and colored steps are labeled by function. (b) Its tree of reasoning states: nodes are intermediate states, tagged with the steps that created or revisited them, edges link each state to the states derived from it, and the case split at step 66 opens two branches. Steps 87–88 are a repetition : they re-derive the state of step 68 in new words. Step 86 is underthinking : it abandons Case 2 while that branch is still viable. Step 90 is an unsupported deduction : it contradicts the Case 1 assumption. Step 113 is a justified switch : it follows the pruning of Case 1 at step 112.
Figure 2: Local signals miss or misattribute the failures that the reasoning state identifies. Columns are steps of the trajectory in Figure 1 . For each failure mode, local signals (gray, dotted) are compared with Arbor (colored, solid), whose arrow points to the earlier step whose state determines the label. Repetition: step 88 shares no -gram with step 68, and embedding similarity links it to the clue recap at step 53 rather than to the state it re-derives. Underthinking: a switch token fires at both step 86 and step 113, but only step 86 leaves a viable branch. Unsupported deduction: the process reward model misses step 90, which violates the Case 1 assumption of step 66, and flags the correct retraction at step 112 instead.
Figure 3: Arbor ’s three-stage framework. An LLM parses each step into a canonical operation, then a deterministic engine replays it against the accumulated reasoning state, and deterministic rules map the resulting state relations to functional labels.
| Row | Definition: the model … |
| Repetition | |
| R1 Re-Derivation | derives again a result it has already derived, after deriving or verifying others. |
| R2 Derive-after-Prune | derives from a hypothesis it has already rejected. |
| R3 Redundant Re-Verification | verifies a result again although nothing it depends on has changed. |
| R4 Subgoal Churn | posts a subgoal again after working on others, without progress on it. |
| Underthinking | |
Table 1: The failure rows. Each row is one relation between a step and the results the model has established before it; § 2.3 gives the predicate on the reasoning-state tree. A row is undefined, rather than approximated, on a task whose structure has no state for it (Appendix H ).
| Operation | Payload | Effect on the tree |
| Derive | a result , or alternatives | adds a child holding under the node depends on; one child per |
| Assume | a hypothesis | adds a child holding under the current leaf |
| Verify | a result , verdict | records and on the node holding |
| Prune | a result or hypothesis | closes the node holding and its subtree |
| Confirm | an answer | commits the node holding ; records |
| Restate | a result | records the restatement on the node holding |
Table 2: The seven operations: the parser assigns each step one operation and its payload, and the engine applies the effect. A result is a fact, value, action or subgoal.
Figure 4: Three tasks, three shapes of reasoning, three failure patterns. Schematic Arbor trees of a typical failing trajectory for each task (statistics in Appendix H.1 ).
| Category | Method | Prec. | Rec. |
| Repetition | Arbor | 64% | 37% |
| n-gram | 49% | 31% | |
| similarity | 62% | 45% | |
| Underthinking | Arbor | 32% | 71% |
| switch tokens | 27% | 62% | |
| Post-Answer Attempt | Arbor | 77% | 87% |
Table 3: Expert review confirms Arbor and rejects local signals. Precision: share of a method’s flagged steps the annotator confirms as the stated failure. Recall: share of the confirmed steps of the failure mode (all methods’ steps pooled) that the method also flags. For underthinking we also showed the annotators the lineage Arbor tracked, since the text rarely settles whether the abandoned branch was still viable (Appendix G.2 ).
Figure 5: Arbor ’s subtype counts predict the outcome better than all local signals combined. Gain in AUROC for separating incorrect from correct trajectories over a baseline of trajectory length and task (solid line, 0.86), pooled over the three tasks. Error bars: 95% confidence intervals.
Figure 6: The evidence behind § 5 , per task (Qwen3-8B; exact values in Tables 21 to 25 ). In (d), the label above each bar is the share of failing trajectories that held a correct solution; in (h), PlanBench has no branch lifecycle.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Model | Correct | Truncated | Invalid |
| ZebraLogic | Qwen3-1.7B | 49.0 | 39.6 | 0.0 |
| Qwen3-8B | 70.2 | 27.0 | 0.0 | |
| Qwen3-32B | 75.4 | 22.9 | 0.0 | |
| 24 Game | Qwen3-1.7B | 80.8 | 6.4 | 12.1 |
| Qwen3-8B | 78.0 | 7.0 | 10.8 | |
| Qwen3-32B | 78.8 | 5.7 | 14.8 |
Table 6: Outcome shares (%) per task and model size over the first three samples of every question (ZebraLogic 3,000, 24 Game 4,086, PlanBench 4,992 trajectories per size). PlanBench “correct” is plan validity; “invalid” is no plan emitted.
Figure 7: Subtype rates by model size. For each task and subtype, the number of events per 100 steps (log scale) at 1.7B, 8B and 32B, left to right within each group, with 95% intervals over problems, on the same 86, 88 and 80 questions per task at every size. A group whose three dots descend is a subtype that becomes rarer with size; most groups are flat. Derive-after-Prune (R2, shaded) is the only subtype that falls at both size increases. Subtypes a task does not define are absent from its panel.
Table 8: Prompt Template for 24 Game
Table 9: Prompt Template for Zebra Logic
Table 10: Prompt Template for PlanBench (the benchmark’s one-shot query for a Blocksworld instance, followed by our instruction line)
Figure 8: The annotation interface. One card as the annotators saw it: the flagged step in its surrounding steps, the instruction stating the failure to judge, and the response options including “undecided”.
Figure 9: What the local measures flag.
| Median tokens opened / share of the trajectory (%) | Pooled share by outcome (%) | ||||||
| Subtype | ZebraLogic | 24 Game | PlanBench | Pooled | correct | incorrect | truncated |
| Repetition | |||||||
| R1 Re-Derivation | 17 / 0.3 | 15 / 0.4 | 16 / 0.2 | 16 / 0.3 | 0.4 | 0.3 | 0.2 |
| R2 Derive-after-Prune | 25 / 0.4 | 17 / 0.4 | 220 / 2.7 | 20 / 0.4 | 0.5 | 0.5 | 0.3 |
| R3 Redundant Re-Verification | 11 / 0.2 | 24 / 0.8 | 12 / 0.2 | 16 / 0.3 | 0.6 | 0.2 | 0.2 |
| R4 Subgoal Churn | – | 14 / 0.3 | 17 / 0.2 | 15 / 0.3 | 0.4 | 0.2 | 0.2 |
Table 19: Extended cost: the tokens an event opens (Qwen3-8B). For each event, the region it opens is, for Branch and Subgoal Abandonment, the steps from the event until the model returns to the left branch, else to the end; for Post-Answer Attempt, until the commit; for Premature Finalization, Infeasible Derivation and False Verdict, every later step resolved to the event’s node or a descendant; for Derive-after-Prune, the later steps in the pruned subtree; for Re-Derivation, Redundant Re-Verification and Subgoal Churn, the steps of the burst the event starts; and for False Rejection, the steps until the rejected subtree is next entered, else to the end. –: undefined on the task.
| Subtype | State | Labels | Pos. | Prev. op |
| 24 Game | ||||
| R1 Re-Derivation | 0.73 | 0.70 | 0.50 | 0.69 |
| R2 Derive-after-Prune † | 0.59 | 0.51 | 0.50 | 0.57 |
| R3 Redundant Re-Verification | 0.83 | 0.81 | 0.58 | 0.77 |
| R4 Subgoal Churn | 0.77 | 0.64 | 0.51 | 0.60 |
| R5 Branch Abandonment | 0.91 | 0.81 | 0.56 | 0.63 |
Table 27: The state before a step separates a failure from the steps where it could have happened. Held-out AUROC of a classifier on the tree state over the preceding steps (State) against each subtype’s matched opportunity steps, and the same classifier restricted to operation labels (Labels), to position in the trajectory (Pos.), and to the previous operation (Prev. op). Bold: state above the previous-operation baseline by more than 0.04; † : within 0.04, inconclusive. –: no label-only feature survived screening.
| Subtype | State | Labels | Pos. | Prev. op |
| Pooled | ||||
| R1 Re-Derivation † | 0.57 | 0.50 | 0.50 | 0.60 |
| R2 Derive-after-Prune | 0.71 | 0.51 | 0.50 | 0.56 |
| R3 Redundant Re-Verification † | 0.74 | 0.73 | 0.58 | 0.70 |
| R9 Infeasible Derivation | 0.73 | 0.53 | 0.53 | 0.54 |
Table 28: Pooled cells (the three tasks together, with task indicators as features), as in Table 27 .
| Pair | Task | Jaccard | State | Pos. | Prev. op |
| R1 vs R2 | ZebraLogic | 0.00 | 0.63 | 0.54 | 0.57 |
| R1 vs R2 | 24 Game | 0.00 | 0.65 | 0.54 | 0.62 |
| R1 vs R2 | Pooled | 0.14 | 0.83 | 0.60 | 0.60 |
| R1 vs R3 | ZebraLogic | 0.00 | 0.84 | 0.74 | 0.69 |
| R1 vs R3 | 24 Game | 0.00 | 0.88 | 0.78 | 0.77 |
| R1 vs R3 | Pooled | 0.13 | 0.81 | 0.81 | 0.72 |
Table 29: Sibling subtypes arise after different states. Jaccard: overlap of the two subtypes’ stable precursor sets. State: AUROC of a classifier on the union of the two sets at telling one sibling’s occurrences from the other’s, with the two baselines.
| Feature | Condition the stand-in reads | Mitigation it gates, and what the gate changes |
| Branch content | the constraints fixed on the branch within the last ten steps | a thinking intervention ( Wu et al., 2025 ) , a chain-of-verification ( Dhuliawala et al., 2023 ) or a process reward ( Lightman et al., 2024 ; Wang et al., 2024 ) : the inserted text or the verification question names those constraints, so the check lands on the constraint an Infeasible Derivation violates |
| Frontier status | whether the current frontier leaf has a verdict | the thought-switch penalty ( Wang et al., 2025 ) : applied only while the frontier leaf has no verdict, so it stops Branch Abandonment and spares the switches that leave a ruled-out branch, which an unconditional penalty suppresses as well |
| Frontier status | whether a checked complete solution stands | budget forcing ( Muennighoff et al., 2025 ) and confidence-gated early exit ( Yang et al., 2025 ; Zhang et al., 2025a ) : generation stops when a checked complete solution stands rather than at a length or confidence threshold, which ends the Post-Answer Attempt without cutting off unfinished search |
| Recent operations | the branch’s last operation and its pruned status | steering vectors for verification and backtracking ( Venhoff et al., 2025 ; Ward et al., 2025 ) : the verification vector is damped right after a check, which removes Redundant Re-Verification, and the backtracking vector is damped on a pruned branch, which removes Derive-after-Prune |
| Offline | the return rows and the dead-end subtrees opened at a departure | spans to remove when compressing trajectories for fine-tuning ( Xia et al., 2025 ; Zhao et al., 2025 ; Munkhbat et al., 2025 ) , replacing token-importance or length criteria |
| Offline | the per-step failure rows | a structure-aware penalty for length-harmonizing reinforcement learning ( Luo et al., 2025a ; Chen et al., 2025b ) that charges a repeated operation rather than every token past the answer |
Table 30: The two routes, feature by feature. Route 1 (first four rows): a stand-in trained on Arbor ’s labels reads the condition that precedes the failure and gates an existing inference-time mitigation on it. Route 2 (last three rows): Arbor ’s labels, computed offline, replace length and token criteria in training-side methods.
More in this topic
Reasoning Trace Analysis
Reasoning Trace Analysis studies the structure, reliability, and causal role of language models’ reasoning traces.
See all 125 papers