Can Language Models Learn to Reject Their Own Bad Reasoning Steps?

Siheng Xiong; Xiaoze Liu; Yiqiao Jin; Xiaoqian Wang; Jing Gao

cs.CLOct 5, 2026

Can Language Models Learn to Reject Their Own Bad Reasoning Steps?

Authors: Siheng Xiong, Xiaoze Liu, Yiqiao Jin, Xiaoqian Wang, Jing Gao

Organizations: Georgia Institute of Technology · Purdue University

Abstract

Verifier-guided decoding can prevent harmful reasoning steps from contaminating subsequent generation, but typically relies on an external learned verifier. We ask whether a language model can instead reject its own bad reasoning steps. We define a prefix's recoverability as the probability that the frozen generator can complete it correctly. Diagnostics show that adjacent recoverability changes are often difficult to resolve with practical Monte Carlo budgets, while same-prefix candidates exhibit a sparse low-recoverability tail. We introduce Self-Step Rejection (SSR), which trains a lightweight LoRA acceptance gate on the generator backbone while keeping the base model frozen. SSR uses confidence-qualified first-passage supervision: steps before the first resolved crossing of a root-relative recoverability barrier are accepted, the crossing step is rejected, and unresolved steps and suffixes are excluded. Training combines pointwise classification, same-prefix pairwise learning, and group-relative policy refinement using final-answer correctness. At inference, SSR accepts candidates or resamples from the unchanged prefix under rejection budgets, without an external learned verifier. Across three reasoning models and five mathematical reasoning benchmarks, SSR improves macro-average accuracy over single-pass decoding by 5.4--10.1 points using 1.21--1.40x as many generated tokens, and achieves the highest macro-average accuracy among evaluated step-level methods. Full-solution scaling methods require 4.47--8.27x the single-pass token cost for comparable performance.

Figures & tables

Figure 1: Macro-average accuracy across five benchmarks versus generated-token cost relative to single-pass decoding. Full-solution scaling includes Majority@8, ESC, ORM reranking, PRM reranking, and USC; other step-level methods include fixed-early resampling, low-confidence resampling, budget forcing, random rejection, and LATTS. SSR achieves the highest accuracy among step-level methods for all three generators and remains competitive with full-solution scaling at substantially lower token cost. See Table 1 for detailed results.

Figure 2: Why same-prefix candidate rejection is a distinct problem. (a) Recoverability along one trajectory evaluated at paragraph boundaries; the open circle marks the first crossing of a root-relative

\tau=0.30

barrier. (b) Candidate steps sampled from one shared prefix exhibit a small low-value tail. (c) Mean root-relative degradation increases with depth across the recoverability audit populations.

Figure 3: Cumulative degradation is statistically visible when local degradation is not. (a) Signed likelihood-ratio statistics for adjacent and root-relative changes along the measured trajectory. Positive values indicate a loss of recoverability; the dotted line at

z=2

provides an approximate significance reference. (b) Approximate rollouts per endpoint required to distinguish two Bernoulli success probabilities with 80% power.

Figure 4: Top: SSR uses the frozen model to generate candidates and enables LoRA to score them. Acceptance extends the prefix; rejection triggers resampling from the unchanged prefix. Bottom: an actual decoding example with at most

L_{\mathrm{step}}

new tokens per step, shown as reasoning summaries. Attempts at each position proceed from top to bottom. Resampling is bounded by a per-position attempt cap

M

and an episode-level rejection budget

B

Model	Method	A25	Olym-E	Olym-H	HMMT25	HMMT26	Avg.	Tok. $\times$
Qwen3-14B	Single pass	70.0	83.0	21.0	50.0	48.5	54.5	1.00
	Majority@8	76.7	94.0	33.0	56.7	51.5	62.4	8.11
	ESC ( $w=2$ )	77.8	94.7	34.3	56.7	52.5	63.2	4.47
	ORM reranking	76.7	91.0	28.0	53.3	57.6	61.3	8.11
	PRM reranking	70.0	85.0	23.0	46.7	48.5	54.6	8.11
	USC	80.0	93.0	35.0	60.0	57.6	65.1	8.24

Table 1: End-to-end accuracy (%) and generation cost, averaged over three evaluation runs. “Avg.” is the macro-average and “Tok.

\times

” is relative to single-pass generation. Gray rows are step-level methods; underlining and bold mark the best full-solution-scaling and step-level accuracy, respectively.

Model	$\mathbb{E}[\|\Delta_{t}\|]$	Adj. resolved@16 (%)	Cand. SD	Tail (%)
Qwen3-14B	0.215 [0.175, 0.253]	24.0 [19.2, 28.2]	0.113 [0.027, 0.176]	6.0 [2.0, 12.0]
R1-Distill-Llama-8B	0.051 [0.014, 0.097]	2.3 [0.0, 9.2]	0.120 [0.058, 0.168]	11.0 [3.0, 21.0]
gpt-oss-20b	0.140 [0.071, 0.227]	18.7 [2.9, 47.2]	0.184 [0.067, 0.265]	11.5 [3.1, 21.9]

Table 2: Recoverability diagnostics across five benchmarks.

\mathbb{E}[|\Delta_{t}|]

: mean absolute adjacent change; resolved@16: fraction resolved with 16 rollouts per endpoint; Cand. SD: noise-corrected within-prefix candidate standard deviation; Tail: fraction significantly below the best candidate. Brackets indicate 95% problem-bootstrap confidence intervals.

Figure 5: Left: verifier discrimination measured by global AUROC over pooled candidates and within-prefix AUROC over candidates sampled from the same prefix; error bars are 95% confidence intervals. Center: reject precision at a fixed rejection rate of 17.8%. Right: accepted-candidate value lift. Qwen3-14B is the generator throughout. Skywork verifier denotes Skywork-Reward-V2-Qwen3-8B ( Liu et al., 2026 ) .

Method	A25	Olym-E	Olym-H	HMMT25	HMMT26	Avg.	Tok. $\times$
Single pass	70.0	83.0	21.0	50.0	48.5	54.5	1.00
SSR (full)	77.8	95.7	35.0	62.2	52.5	64.6	1.21
Label all below-barrier steps	72.2	86.0	24.3	53.3	49.5	57.1	1.62
No same-prefix pairwise loss	73.3	88.0	26.3	54.4	50.5	58.5	1.14
No policy refinement	74.4	89.7	28.7	56.7	50.5	60.0	1.12

Table 3: Training-component ablations on Qwen3-14B, averaged over three evaluation runs. Indented rows remove or replace one component of the full SSR recipe. “Avg.” is the five-benchmark macro-average and “Tok.

\times

” is relative to single-pass generation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Held-out development accuracy (left) and realized on-policy rejection rate (right) over optimizer steps during policy refinement with Qwen3-14B.

Table 10

Table 11

Step	Decision	Abridged candidate reasoning
$t=1$	Accept	Identifies the row and $3\times 3$ -block constraints and begins reducing the problem to a combinatorial count.
$t=2$ , cand. 1	Reject	Incorrectly infers that entries must also be distinct within each column of a block and repeatedly recasts the block as a Latin-square-like object.
$t=2$ , cand. 2	Accept	Retains only the stated row and block constraints and represents each block by an ordered partition of $\{1,\ldots,9\}$ into three row subsets.
$t=3$ – $5$	Accept	$\cdots$ Reduces the remaining task to counting the assignments for the second and third blocks after fixing the first block.
$t=6$ , cand. 1	Reject	Formulates the assignment as a constrained bipartite matching problem but does not obtain the required count.
$t=6$ , cand. 2	Accept	Derives the row-assignment count $\sum_{c=0}^{3}\binom{3}{c}^{3}=56$ and obtains $9!\cdot 56\cdot(3!)^{6}$ .

Appendix

Table 8: Abridged SSR trajectory for AIME 2025 Problem 10 with Qwen3-14B. Two candidates are rejected; omitted steps are accepted on their first attempt.

Step	Decision	Abridged candidate reasoning
$t=1$	Accept	Reads $\|A_{i}\|=i$ from the case $i=j$ and restates the objective as minimizing the eight maxima $\max A_{d}$ over the divisors of $250$ .
$t=2$	Accept	Observes that the elements are arbitrary positive integers chosen by the construction, so only which labels are assigned to which sets matters.
$t=3$ , cand. 1	Reject	Assigns one element $e_{t}$ to every $A_{k}$ with $t\mid k$ , computes that this yields $\|A_{i}\cap A_{j}\|\geq d(\gcd(i,j))$ rather than $\gcd(i,j)$ , and on finding the count too small begins a prime-by-prime case analysis instead of revising the construction.
$t=3$ , cand. 2	Accept	Derives the structural lemma: for $d\mid n$ , $\|A_{n}\cap A_{d}\|=d=\|A_{d}\|$ forces $A_{d}\subseteq A_{n}$ , so the family is nested along divisibility. Assigns each element its minimal index $m$ and concludes $e\in A_{n}$ iff $m\mid n$ .
$t=4$ – $7$	Accept	$\cdots$ Writes $\|A_{n}\|=\sum_{m\mid n}f(m)=n$ for the count $f(m)$ of elements of minimal index $m$ , giving $f=1,1,4,4,20,20,100,100$ on the divisors of $250$ , and reduces the objective to choosing each group’s largest label $g(m)$ , since $\max A_{d}=\max_{m\mid d}g(m)$ .
$t=8$ – $12$	Accept	$\cdots$ Evaluates orderings of the eight groups. Labeling them in ascending order of size gives $g=1,2,6,10,30,50,150,250$ and the sum $499$ , while moving index $5$ earlier raises the total to $502$ .

Appendix

Table 9: Abridged SSR trajectory for HMMT February 2026 Problem 19 with gpt-oss-20b. One candidate is rejected; omitted steps are accepted on their first attempt.

Explore similar work

Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring
Dec 16, 2025Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo +1Large Reasoning Models Progressive Reasoning
1IBM Research Europe, Dublin, Ireland · 2Trinity College Dublin, Ireland · 3ADAPT Research Centre, Dublin, Ireland
From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning
Apr 16, 2026Kiran Purohit, Ramasuri Narayanam, Soumyabrata PalSpeculative Decoding LLM Inference Optimization
IIT Kharagpur · Adobe Research
Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
May 19, 2026Xiaoou Liu, Tiejin Chen, Dengjia Zhang +3LLM Reasoning Strategies Reasoning Traces
1Arizona State University · 2Johns Hopkins University · 3Purdue +1

Dec 16, 2025, cs.CLJ/K move · Enter open · S save

Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring

Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo, John D. Kelleher

1IBM Research Europe, Dublin, Ireland · 2Trinity College Dublin, Ireland · 3ADAPT Research Centre, Dublin, Ireland

Large Reasoning Models Progressive Reasoning Tagging Natural Language Inference Stance

Open paper AskPDF arXiv

The field of Language Reasoning Models (LRMs) has been very active over the past few years with advances in training and inference techniques enabling LRMs to reason longer, and more accurately. However, a growing body of studies show that LRMs are still inefficient, over-generating verification and reflection steps. To address this challenge, we introduce the Step-Tagging framework, a lightweight sentence-classifier enabling real-time annotation of the type of reasoning steps that an LRM is generating. To monitor reasoning behaviors, we introduced ReasonType: a novel taxonomy of reasoning steps. Building on this framework, we demonstrated that online monitoring of the count of specific steps can produce effective interpretable early stopping criteria of LRM inferences. We evaluate the Step-tagging framework on three open-source reasoning models across standard benchmark datasets: MATH500, GSM8K, AIME and non-mathematical tasks (GPQA and MMLU-Pro). We achieve 20 to 50% token reduction while maintaining comparable accuracy to standard generation, with largest gains observed on more computation-heavy tasks. This work offers a novel way to increase control over the generation of LRMs, and a new tool to study behaviors of LRMs.

Abstract

Explore similar work

Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring

From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning

Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution