Can Language Models Learn to Reject Their Own Bad Reasoning Steps?

Siheng Xiong; Xiaoze Liu; Yiqiao Jin; Xiaoqian Wang; Jing Gao

cs.CLOct 5, 2026

Can Language Models Learn to Reject Their Own Bad Reasoning Steps?

Authors: Siheng Xiong, Xiaoze Liu, Yiqiao Jin, Xiaoqian Wang, Jing Gao

Organizations: Georgia Institute of Technology · Purdue University

Abstract

Verifier-guided decoding can prevent harmful reasoning steps from contaminating subsequent generation, but typically relies on an external learned verifier. We ask whether a language model can instead reject its own bad reasoning steps. We define a prefix's recoverability as the probability that the frozen generator can complete it correctly. Diagnostics show that adjacent recoverability changes are often difficult to resolve with practical Monte Carlo budgets, while same-prefix candidates exhibit a sparse low-recoverability tail. We introduce Self-Step Rejection (SSR), which trains a lightweight LoRA acceptance gate on the generator backbone while keeping the base model frozen. SSR uses confidence-qualified first-passage supervision: steps before the first resolved crossing of a root-relative recoverability barrier are accepted, the crossing step is rejected, and unresolved steps and suffixes are excluded. Training combines pointwise classification, same-prefix pairwise learning, and group-relative policy refinement using final-answer correctness. At inference, SSR accepts candidates or resamples from the unchanged prefix under rejection budgets, without an external learned verifier. Across three reasoning models and five mathematical reasoning benchmarks, SSR improves macro-average accuracy over single-pass decoding by 5.4--10.1 points using 1.21--1.40x as many generated tokens, and achieves the highest macro-average accuracy among evaluated step-level methods. Full-solution scaling methods require 4.47--8.27x the single-pass token cost for comparable performance.

Figures & tables

Figure 1: Macro-average accuracy across five benchmarks versus generated-token cost relative to single-pass decoding. Full-solution scaling includes Majority@8, ESC, ORM reranking, PRM reranking, and USC; other step-level methods include fixed-early resampling, low-confidence resampling, budget forcing, random rejection, and LATTS. SSR achieves the highest accuracy among step-level methods for all three generators and remains competitive with full-solution scaling at substantially lower token cost. See Table 1 for detailed results.

Figure 2: Why same-prefix candidate rejection is a distinct problem. (a) Recoverability along one trajectory evaluated at paragraph boundaries; the open circle marks the first crossing of a root-relative

\tau=0.30

barrier. (b) Candidate steps sampled from one shared prefix exhibit a small low-value tail. (c) Mean root-relative degradation increases with depth across the recoverability audit populations.

Figure 3: Cumulative degradation is statistically visible when local degradation is not. (a) Signed likelihood-ratio statistics for adjacent and root-relative changes along the measured trajectory. Positive values indicate a loss of recoverability; the dotted line at

z=2

provides an approximate significance reference. (b) Approximate rollouts per endpoint required to distinguish two Bernoulli success probabilities with 80% power.

Figure 4: Top: SSR uses the frozen model to generate candidates and enables LoRA to score them. Acceptance extends the prefix; rejection triggers resampling from the unchanged prefix. Bottom: an actual decoding example with at most

L_{\mathrm{step}}

new tokens per step, shown as reasoning summaries. Attempts at each position proceed from top to bottom. Resampling is bounded by a per-position attempt cap

M

and an episode-level rejection budget

B

Model	Method	A25	Olym-E	Olym-H	HMMT25	HMMT26	Avg.	Tok. $\times$
Qwen3-14B	Single pass	70.0	83.0	21.0	50.0	48.5	54.5	1.00
	Majority@8	76.7	94.0	33.0	56.7	51.5	62.4	8.11
	ESC ( $w=2$ )	77.8	94.7	34.3	56.7	52.5	63.2	4.47
	ORM reranking	76.7	91.0	28.0	53.3	57.6	61.3	8.11
	PRM reranking	70.0	85.0	23.0	46.7	48.5	54.6	8.11
	USC	80.0	93.0	35.0	60.0	57.6	65.1	8.24

Table 1: End-to-end accuracy (%) and generation cost, averaged over three evaluation runs. “Avg.” is the macro-average and “Tok.

\times

” is relative to single-pass generation. Gray rows are step-level methods; underlining and bold mark the best full-solution-scaling and step-level accuracy, respectively.

Model	$\mathbb{E}[\|\Delta_{t}\|]$	Adj. resolved@16 (%)	Cand. SD	Tail (%)
Qwen3-14B	0.215 [0.175, 0.253]	24.0 [19.2, 28.2]	0.113 [0.027, 0.176]	6.0 [2.0, 12.0]
R1-Distill-Llama-8B	0.051 [0.014, 0.097]	2.3 [0.0, 9.2]	0.120 [0.058, 0.168]	11.0 [3.0, 21.0]
gpt-oss-20b	0.140 [0.071, 0.227]	18.7 [2.9, 47.2]	0.184 [0.067, 0.265]	11.5 [3.1, 21.9]

Table 2: Recoverability diagnostics across five benchmarks.

\mathbb{E}[|\Delta_{t}|]

: mean absolute adjacent change; resolved@16: fraction resolved with 16 rollouts per endpoint; Cand. SD: noise-corrected within-prefix candidate standard deviation; Tail: fraction significantly below the best candidate. Brackets indicate 95% problem-bootstrap confidence intervals.

Figure 5: Left: verifier discrimination measured by global AUROC over pooled candidates and within-prefix AUROC over candidates sampled from the same prefix; error bars are 95% confidence intervals. Center: reject precision at a fixed rejection rate of 17.8%. Right: accepted-candidate value lift. Qwen3-14B is the generator throughout. Skywork verifier denotes Skywork-Reward-V2-Qwen3-8B ( Liu et al., 2026 ) .

Method	A25	Olym-E	Olym-H	HMMT25	HMMT26	Avg.	Tok. $\times$
Single pass	70.0	83.0	21.0	50.0	48.5	54.5	1.00
SSR (full)	77.8	95.7	35.0	62.2	52.5	64.6	1.21
Label all below-barrier steps	72.2	86.0	24.3	53.3	49.5	57.1	1.62
No same-prefix pairwise loss	73.3	88.0	26.3	54.4	50.5	58.5	1.14
No policy refinement	74.4	89.7	28.7	56.7	50.5	60.0	1.12

Table 3: Training-component ablations on Qwen3-14B, averaged over three evaluation runs. Indented rows remove or replace one component of the full SSR recipe. “Avg.” is the five-benchmark macro-average and “Tok.

\times

” is relative to single-pass generation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Held-out development accuracy (left) and realized on-policy rejection rate (right) over optimizer steps during policy refinement with Qwen3-14B.

Table 10

Table 11

Step	Decision	Abridged candidate reasoning
$t=1$	Accept	Identifies the row and $3\times 3$ -block constraints and begins reducing the problem to a combinatorial count.
$t=2$ , cand. 1	Reject	Incorrectly infers that entries must also be distinct within each column of a block and repeatedly recasts the block as a Latin-square-like object.
$t=2$ , cand. 2	Accept	Retains only the stated row and block constraints and represents each block by an ordered partition of $\{1,\ldots,9\}$ into three row subsets.
$t=3$ – $5$	Accept	$\cdots$ Reduces the remaining task to counting the assignments for the second and third blocks after fixing the first block.
$t=6$ , cand. 1	Reject	Formulates the assignment as a constrained bipartite matching problem but does not obtain the required count.
$t=6$ , cand. 2	Accept	Derives the row-assignment count $\sum_{c=0}^{3}\binom{3}{c}^{3}=56$ and obtains $9!\cdot 56\cdot(3!)^{6}$ .

Appendix

Table 8: Abridged SSR trajectory for AIME 2025 Problem 10 with Qwen3-14B. Two candidates are rejected; omitted steps are accepted on their first attempt.

Step	Decision	Abridged candidate reasoning
$t=1$	Accept	Reads $\|A_{i}\|=i$ from the case $i=j$ and restates the objective as minimizing the eight maxima $\max A_{d}$ over the divisors of $250$ .
$t=2$	Accept	Observes that the elements are arbitrary positive integers chosen by the construction, so only which labels are assigned to which sets matters.
$t=3$ , cand. 1	Reject	Assigns one element $e_{t}$ to every $A_{k}$ with $t\mid k$ , computes that this yields $\|A_{i}\cap A_{j}\|\geq d(\gcd(i,j))$ rather than $\gcd(i,j)$ , and on finding the count too small begins a prime-by-prime case analysis instead of revising the construction.
$t=3$ , cand. 2	Accept	Derives the structural lemma: for $d\mid n$ , $\|A_{n}\cap A_{d}\|=d=\|A_{d}\|$ forces $A_{d}\subseteq A_{n}$ , so the family is nested along divisibility. Assigns each element its minimal index $m$ and concludes $e\in A_{n}$ iff $m\mid n$ .
$t=4$ – $7$	Accept	$\cdots$ Writes $\|A_{n}\|=\sum_{m\mid n}f(m)=n$ for the count $f(m)$ of elements of minimal index $m$ , giving $f=1,1,4,4,20,20,100,100$ on the divisors of $250$ , and reduces the objective to choosing each group’s largest label $g(m)$ , since $\max A_{d}=\max_{m\mid d}g(m)$ .
$t=8$ – $12$	Accept	$\cdots$ Evaluates orderings of the eight groups. Labeling them in ascending order of size gives $g=1,2,6,10,30,50,150,250$ and the sum $499$ , while moving index $5$ earlier raises the total to $502$ .

Appendix

Table 9: Abridged SSR trajectory for HMMT February 2026 Problem 19 with gpt-oss-20b. One candidate is rejected; omitted steps are accepted on their first attempt.

Explore similar work

Dec 16, 2025cs.CL

Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring

The field of Language Reasoning Models (LRMs) has been very active over the past few years with advances in training and inference techniques enabling LRMs to reason longer, and more accurately. However, a growing body of studies show that LRMs are still inefficient, over-generating verification and reflection steps. To address this challenge, we introduce the Step-Tagging framework, a lightweight sentence-classifier enabling real-time annotation of the type of reasoning steps that an LRM is generating. To monitor reasoning behaviors, we introduced ReasonType: a novel taxonomy of reasoning steps. Building on this framework, we demonstrated that online monitoring of the count of specific steps can produce effective interpretable early stopping criteria of LRM inferences. We evaluate the Step-tagging framework on three open-source reasoning models across standard benchmark datasets: MATH500, GSM8K, AIME and non-mathematical tasks (GPQA and MMLU-Pro). We achieve 20 to 50% token reduction while maintaining comparable accuracy to standard generation, with largest gains observed on more computation-heavy tasks. This work offers a novel way to increase control over the generation of LRMs, and a new tool to study behaviors of LRMs.

Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo +1

1IBM Research Europe, Dublin, Ireland · 2Trinity College Dublin, Ireland · 3ADAPT Research Centre, Dublin, Ireland

Large Reasoning Models Progressive Reasoning Tagging Natural Language Inference Stance

Ask PDF

Apr 16, 2026cs.CL

From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning

Speculative decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose outputs that a stronger target model verifies. However, its token-centric nature allows erroneous steps to propagate. Prior approaches mitigate this using external reward models, but incur additional latency, computational overhead, and limit generalizability. We propose SpecGuard, a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals. At each step, SpecGuard samples multiple draft candidates and selects the most consistent step, which is then validated using an ensemble of two lightweight model-internal signals: (i) an attention-based grounding score that measures attribution to the input and previously accepted steps, and (ii) a log-probability-based score that captures token-level confidence. These signals jointly determine whether a step is accepted or recomputed using the target, allocating compute selectively. Experiments across a range of reasoning benchmarks show that SpecGuard improves accuracy by 3.6% while reducing latency by ~11%, outperforming both SD and reward-guided SD.

Kiran Purohit, Ramasuri Narayanam, Soumyabrata Pal

IIT Kharagpur · Adobe Research

Speculative Decoding LLM Inference Optimization Reasoning Benchmark Token-Level Uncertainty

Ask PDF

May 19, 2026cs.CL

Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution

Large Language Models have achieved strong performance on reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step reasoning trace might fail remains difficult. Confidence estimation offers a diagnostic signal, yet existing methods are restricted to final answers or require internal model access. In this paper, we introduce Stepwise Confidence Attribution (SCA), a framework for closed-source LLMs that assigns step-level confidence based only on generated reasoning traces. SCA applies the Information Bottleneck principle: steps aligning with consensus structures across correct solutions receive high confidence, while deviations are flagged as potentially erroneous. We propose two complementary methods: (1) NIBS, a non-parametric IB approach measuring consistency without graph structures, and (2) GIBS, a graph-based IB model that learns subgraphs through a differentiable mask to capture logical variability. Extensive experiments on mathematical reasoning and multi-hop question answering show that SCA reliably identifies low-confidence steps strongly correlated with reasoning errors. Moreover, using step-level confidence to guide self-correction improves the correction success rate by up to 13.5% over answer-level feedback.

Xiaoou Liu, Tiejin Chen, Dengjia Zhang +3

1Arizona State University · 2Johns Hopkins University · 3Purdue +1

LLM Reasoning Strategies Reasoning Traces Information Bottleneck

Ask PDF