StepBack: Step-Level Error-Localized Resampling for Efficient Test-Time Reasoning

Yang Ouyang; Jung-Eun Kim

cs.AINov 5, 2026

StepBack: Step-Level Error-Localized Resampling for Efficient Test-Time Reasoning

Authors: Yang Ouyang, Jung-Eun Kim

Organizations: Department of Computer Science, North Carolina State University

Abstract

Self-consistency (SC) is a widely used decoding strategy for improving chain-of-thought reasoning, but it requires sampling many complete reasoning trajectories with a preset budget. Its efficient variants reduce the cost mainly by stopping full-trajectory sampling early or distributing the number of samples across questions. In this paper, we propose StepBack, a step-level error-localized resampling method for efficient test-time reasoning. Given a small set of draft trajectories, StepBack localizes likely first error positions using lightweight signals and then resamples the detected error step and its suffix, followed by majority answer voting. Empirically, StepBack matches the accuracy of SC on average across seven mathematical reasoning and multi-hop QA benchmarks, while reducing token usage by up to 41.26%. In addition, by nature, StepBack can be combined with adaptive inference methods such as Adaptive Consistency, further improving their inference-time efficiency.

Figures & tables

Figure 1: StepBack achieves higher accuracy with significantly fewer generated tokens than self-consistency on GSM8K.

Figure 2: Instead of regenerating many full trajectories, StepBack identifies a likely first error step, reuses valid early steps, and resamples only the erroneous step and its suffix before majority voting with the same voting budget (32 in this example).

Dataset	Method	Accuracy	Tokens / Q	Savings
GSM8K	SC@32	91.21%	9506	–
	Oracle error-localized resampling $n_{d}=8,n_{s}=4$	91.89%	3427	64.0%
	Oracle error-localized resampling $n_{d}=16,n_{s}=2$	92.27%	5449	42.7%
MATH500	SC@32	72.80%	18335	–
	Oracle error-localized resampling $n_{d}=8,n_{s}=4$	72.40%	9815	46.5%
	Oracle error-localized resampling $n_{d}=16,n_{s}=2$	73.00%	12615	31.2%

Table 1: Oracle first-error localization: suffix resampling starts from annotated first-error positions, with

n_{d}

drafts and

n_{s}

answers per draft (

n_{d}n_{s}=32

). Savings are computed relative to SC@32.

	Greedy	SC@32		NLL-signal				PRM-signal
Dataset	Acc	Acc	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\rho_{16}$	$\Delta$ Acc	$\eta$	G/1kT	$\rho_{16}$
Llama-3B
GSM8K	77.41	88.02	1.488	+0.00	34.47	2.270	9.14	+0.99	34.77	2.494	10.47
MATH500	42.20	53.40	0.691	+0.60	24.22	0.961	8.23	+1.20	20.81	0.966	9.18
AMC2023	15.66	33.73	0.705	-3.61	14.92	0.663	7.27	+2.41	10.83	0.896	8.46
AIME24	7.78	15.56	0.273	-2.22	12.41	0.222	7.08	-2.22	9.72	0.216	9.29

Table 2: Main comparison between self-consistency and StepBack on Llama-3B , Qwen-3B , and Qwen-7B under the same candidate-answer budget

B=32

. StepBack uses

(n_{d},n_{s})=(16,2)

with thresholds

\delta_{\mathrm{NLL}}=0.1

and

\delta_{\mathrm{PRM}}=0.05

for all three models. We report SC@32 accuracy once per dataset, followed by the accuracy change

\Delta\mathrm{Acc}

(percentage points), token saving

\eta

(%) and localization overhead

\rho_{16}

(%) of StepBack .

\rho_{16}

is the share of wall-clock GPU time spent scoring the 16 drafts, relative to drafting, scoring, and suffix generation combined. G/1kT is the accuracy gain over greedy CoT per 1k generated tokens; the highest G/1kT in each row is in bold. Positive, zero, and negative accuracy changes are shown in green, gray, and dark red, respectively. Avg. is the mean over the seven datasets weighted by their number of questions (4,166 in total). Five-seed results are reported in Appendix K .

Method	MATH500				GSM8K				HotpotQA
Method	Acc.	PExp	Tok.	G/1kT	Acc.	PExp	Tok.	G/1kT	Acc.	PExp	Tok.	G/1kT
CoT	61.80	1.00	577	0.000	85.14	1.00	295	0.000	23.20	1.00	240	0.000
SC	72.80	32.00	18,335	0.600	91.21	32.00	9,506	0.638	33.00	32.00	7,643	1.282
StepBack (ours)	73.20	32.00	13,620	0.837	91.59	32.00	5,860	1.100	34.20	32.00	7,056	1.559
AC	72.80	16.57	11,386	0.966	91.13	10.84	3,509	1.707	33.00	28.09	6,792	1.443
StepBack + AC	72.60	15.46	8,749	1.234	92.34	9.81	2,166	3.326	34.80	28.11	6,286	1.845

Table 3: Combining StepBack with adaptive reasoning methods using Qwen2.5-3B-Instruct and maximum budget

B=32

. CoT denotes greedy chain-of-thought decoding. For StepBack , the candidate pool is built from

n_{d}=16

draft anchors and

n_{s}=2

candidate answers per draft (the draft’s own answer and one localized suffix sample), using PRM-signal . PExp ( Jindal et al., 2026 ) is the average number of consumed candidate expansions per question. For SC/AC/ESC/DSC, each expansion is a full trajectory. For StepBack , each expansion is a draft trajectory or a localized suffix continuation. Tok. is the number of generated tokens per question, and G/1kT is the accuracy gain over CoT per 1k generated tokens. Within each pair of a baseline and its StepBack counterpart, bold numbers indicate the better value, with higher being better for Acc. and G/1kT and lower being better for Tok. and PExp.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Content
Question	James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?
Gold answer	$540$
Predicted answer	$180$
Model response	To determine the total number of meters James runs in a week, we can break down the problem into smaller, manageable steps. 1. Calculate the total meters run per sprint: James runs 60 meters per sprint. 2. Calculate the total meters run per week for one sprint: James runs 3 sprints each week. Therefore, the total meters run in one week for one sprint is: $60\,\text{meters/sprint}\times 3\,\text{sprints/week}=180\,\text{meters/week}$ . 3. Calculate the total meters run per week for all sprints: Since we have already calculated that James runs 180 meters per week for one sprint, we can state the final total meters run per week as: $180\,\text{meters/week}$ . Putting the final answer in the required format: $\boxed{180}$
Annotated first error	Parsed Step 3, corresponding to the model’s numbered Step 2.
Reason	The model incorrectly interprets “3 sprints 3 times a week” as only 3 sprints per week instead of $3\times 3=9$ sprints per week. Therefore, it uses the wrong base for the weekly-distance calculation.

Appendix

Table 4: Example of a first-error annotation. The model predicts

180

, while the correct answer is

540

. The annotated first error is Step 3 under our parsed-step indexing, corresponding to the model’s numbered Step 2.

Method	Exact	Prev-1	Not-Later	Mean Pre-Error Dist.
PRM-signal $(\delta=0.05)$	22.3%	32.8%	52.3%	1.07
PRM-signal $(\delta=0.10)$	21.4%	31.2%	48.6%	0.98
PRM-signal $(\delta=0.20)$	20.5%	29.5%	45.4%	0.93
NLL-signal $(\delta=0.10)$	15.5%	26.4%	43.5%	1.23
NLL-signal $(\delta=0.20)$	11.5%	19.4%	29.2%	0.83
NLL-signal $(\delta=0.40)$	8.7%	13.8%	19.5%	0.20

Appendix

Table 5: Full first-error localization diagnostics. We compare lightweight predicted error positions

\hat{e}

against annotated first-error positions

e

. Prev-1 measures whether the prediction is exactly the first-error step or one step before it, indicating accurate localization without excessive early resampling.

Dataset	Method	Accuracy	Tokens / Q	Savings
GSM8K	SC@32	91.21%	9506	–
	Oracle error-localized resampling $n_{d}=1,n_{s}=32$	88.48%	1580	83.4%
	Oracle error-localized resampling $n_{d}=2,n_{s}=16$	89.46%	1863	80.4%
	Oracle error-localized resampling $n_{d}=4,n_{s}=8$	91.43%	2381	74.9%
	Oracle error-localized resampling $n_{d}=8,n_{s}=4$	91.89%	3427	64.0%
	Oracle error-localized resampling $n_{d}=16,n_{s}=2$	92.27%	5449	42.7%

Appendix

Table 6: Full oracle error-localized resampling results. Resampling starts from annotated first-error positions. Savings are computed relative to SC@32 on the same dataset.

NLL-signal
$\delta$	MATH	HotpotQA	Macro	$\eta$ (%)
SC@32	80.00	44.00	62.00	–
0.05	80.00	45.00	62.50	19.67
0.1 †	80.00	45.50	62.75	25.25
0.15	79.00	45.50	62.25	29.90
0.2	79.00	46.00	62.50	33.86

Appendix

Table 7: Threshold calibration on held-out training questions. Results are averaged over two seeds on 100 MATH and 100 HotpotQA training questions, using Qwen2.5-3B-Instruct in the

16\times 2

setting. Macro is the mean of MATH accuracy and HotpotQA, and

\eta

is the token saving relative to SC@32. The selected threshold for each signal is shown in bold, and † marks the default used in the main experiments.

Signal-free
Rollback rule	Acc	$\Delta$ Acc	$\eta$
SC@32 (no rollback)	58.71	–	–
Random	58.47	-0.24	20.40
Fixed -25%	58.62	-0.09	5.29
Fixed -50%	58.59	-0.12	16.41
Fixed -75%	58.76	+0.05	33.66

Appendix

Table 8: Rollback rules within StepBack over all seven datasets ( Qwen-3B ,

(n_{d},n_{s})=(16,2)

B=32

, single run). Acc is the accuracy over all 4,166 questions and

\Delta

Acc is its change from SC@32 in points.

\eta

is the token saving relative to SC@32, averaged over the seven datasets with equal weight; it therefore differs from the question-weighted average in Table 2 . All PRM-based rules use Qwen2.5-Math-PRM-7B . The highest accuracy and savings are in bold.

MATH500
	4 $\times$ 8		8 $\times$ 4		16 $\times$ 2
Rollback rule	Acc	$\eta$	Acc	$\eta$	Acc	$\eta$
Random	$70.16\pm 0.82$	34.8	$71.88\pm 0.87$	29.9	$72.60\pm 0.51$	19.9
Fixed -25%	$71.48\pm 0.61$	11.0	$72.32\pm 0.30$	9.6	$72.60\pm 0.25$	6.4
Fixed -50%	$71.08\pm 0.36$	28.9	$71.60\pm 0.55$	24.7	$72.16\pm 0.72$	16.6
Fixed -75%	$69.16\pm 1.32$	56.1	$71.52\pm 0.67$	48.1	$72.20\pm 0.45$	32.2

Appendix

Table 9: Five-seed comparison of rollback rules on MATH500 and HotpotQA ( Qwen-3B ,

B=32

). Acc is the mean over five seeds

\pm

the sample standard deviation across seeds, and

\eta

is the token saving relative to SC@32. PRM-signal uses

\delta_{\mathrm{PRM}}=0.05

with either Skywork-o1-Open-PRM-Qwen-2.5-1.5B (1.5B) or Qwen2.5-Math-PRM-7B (7B). For reference, SC@32 reaches

72.84\pm 0.17

on MATH500 over the same seeds and 33.00 on HotpotQA (Table 2 ). The highest mean accuracy in each column is in bold.

	4 $\times$ 8		8 $\times$ 4		16 $\times$ 2
Rollback rule	Acc	$\eta$	Acc	$\eta$	Acc	$\eta$
Final-step	56.53	76.87	58.23	66.10	58.76	44.07
NLL-signal ( $\delta_{\mathrm{NLL}}=0.1$ )	57.23	43.20	58.47	36.73	58.83	24.56
PRM-signal ( $\delta_{\mathrm{PRM}}=0.05$ )	58.59	29.75	58.79	25.61	58.88	16.96
NLL-signal ( $\delta_{\mathrm{NLL}}=0.5$ )	56.53	72.06	57.85	61.84	58.33	41.33
PRM-signal ( $\delta_{\mathrm{PRM}}=0.8$ )	58.21	64.53	59.29	55.03	58.95	36.76

Appendix

Table 10: Final-step versus StepBack at each allocation over all seven datasets ( Qwen-3B , single run). Acc and

\eta

are defined as in Table 8 ; SC@32 reaches 58.71%. The last two rows use larger thresholds, which send more drafts to the final-step fallback. The highest accuracy in each column is in bold.

Allocation	Draft votes	Rewrite votes	Conflicts	$V_{\mathrm{full}}=V_{R}$	$V_{\mathrm{full}}=V_{D}$	Neither
4 $\times$ 8	4	28	1,098	981 (89.34%)	76 (6.92%)	41 (3.73%)
8 $\times$ 4	8	24	853	634 (74.33%)	167 (19.58%)	52 (6.10%)
16 $\times$ 2	16	16	756	363 (48.02%)	328 (43.39%)	65 (8.60%)

Appendix

Table 11: Draft/rewrite conflicts of StepBack with PRM-signal over the 4,166 questions ( Qwen-3B ,

\delta_{\mathrm{PRM}}=0.05

, single run).

V_{D}

V_{R}

, and

V_{\mathrm{full}}

are the most frequent answers among the draft answers, the rewrite answers, and all 32 answers. A conflict is a question with

V_{D}\neq V_{R}

; the last three columns split the conflicts by whether

V_{\mathrm{full}}=V_{R}

V_{\mathrm{full}}=V_{D}

, or neither.

Dataset	Greedy Acc	SC@32		1 $\times$ 32			2 $\times$ 16			4 $\times$ 8			8 $\times$ 4			16 $\times$ 2
Dataset	Greedy Acc	Acc	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\Delta$ Acc	$\eta$	G/1kT
GSM8K	85.14	91.21	0.64	-0.99	74.60	2.10	-0.23	71.74	2.17	+0.53	67.24	2.12	+0.22	57.38	1.55	+0.38	38.37	1.10
MATH500	61.80	72.80	0.60	-2.00	48.74	0.96	-2.20	45.89	0.89	-0.40	42.70	1.01	+0.60	36.70	1.00	+0.40	25.60	0.84
AMC2023	33.73	43.37	0.37	+0.00	33.58	0.56	+2.41	31.88	0.68	+1.21	29.22	0.59	+1.21	24.12	0.55	+2.41	15.07	0.55
AIME24	4.44	6.67	0.08	+0.00	16.04	0.09	+1.11	15.30	0.14	+1.11	14.53	0.13	+1.11	13.11	0.13	+1.11	8.66	0.13
OlympiadBench	26.26	34.87	0.33	-1.64	28.39	0.37	+0.59	26.98	0.48	+0.14	26.53	0.45	+1.33	22.65	0.49	+0.00	14.83	0.39

Appendix

Table 12: Draft–suffix allocation study for StepBack with PRM-signal on Qwen-3B under a fixed candidate-answer budget

B=32

. Greedy Acc and SC@32 are shown as reference points. For each

n_{d}\times n_{s}

configuration,

\Delta\mathrm{Acc}

is relative to SC@32,

\eta

is token saving relative to SC@32, and G/1kT is accuracy gain over Greedy CoT per 1k generated tokens. The highest G/1kT in each row is bolded.

Dataset	Greedy Acc	SC@32		1 $\times$ 32			2 $\times$ 16			4 $\times$ 8			8 $\times$ 4			16 $\times$ 2
Dataset	Greedy Acc	Acc	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\Delta$ Acc	$\eta$	G/1kT	$\Delta$ Acc	$\eta$	G/1kT
GSM8K	85.14	91.21	0.64	-4.25	67.97	0.60	-4.40	65.44	0.51	-0.69	60.89	1.45	-0.46	51.87	1.23	-0.45	34.64	0.90
MATH500	61.80	72.80	0.60	-6.20	48.90	0.51	-8.20	48.02	0.29	-3.40	44.80	0.75	-0.60	37.90	0.91	-0.40	25.46	0.78
AMC2023	33.73	43.37	0.37	-4.82	52.70	0.39	-3.61	49.37	0.46	+1.21	42.03	0.72	+4.82	35.59	0.86	+2.41	24.37	0.61
AIME24	4.44	6.67	0.08	-1.11	40.92	0.07	-1.11	34.86	0.06	+0.00	32.57	0.11	+0.00	26.55	0.10	+0.00	18.47	0.09
OlympiadBench	26.26	34.87	0.33	-4.31	40.08	0.27	-3.86	38.06	0.29	-3.56	34.90	0.30	+0.14	30.13	0.48	-0.45	20.74	0.39

Appendix

Table 13: Draft–suffix allocation study for StepBack with NLL-signal on Qwen-3B under a fixed candidate-answer budget

B=32

. Greedy Acc and SC@32 are shown as reference points. For each

n_{d}\times n_{s}

configuration,

\Delta\mathrm{Acc}

is relative to SC@32,

\eta

is token saving relative to SC@32, and G/1kT is accuracy gain over Greedy CoT per 1k generated tokens. The highest G/1kT in each row is bolded.

	PRM or Baseline	ms/sample	Config	Acc.	$\Delta$ Acc	$\eta$
Baseline	SC@32	–	–	72.84	+0.00	0.0
w/ StepBack	Skywork-o1-Open-PRM-Qwen-2.5-1.5B	13.7	4 $\times$ 8	71.88	-0.96	36.0
		13.7	8 $\times$ 4	73.20	+0.36	23.8
		13.7	16 $\times$ 2	72.48	-0.36	24.6
w/ StepBack	Qwen2.5-Math-PRM-7B	19.7	4 $\times$ 8	72.40	-0.44	39.1
		19.7	8 $\times$ 4	73.20	+0.36	26.8

Appendix

Table 14: Effect of PRM size on MATH500 using the same Qwen2.5-3B-Instruct generator and the same candidate-answer budget

B=32

. We compare SC@32 with StepBack using PRM-signal localization (

\delta_{\mathrm{PRM}}=0.05

), where the likely error step is predicted by either Skywork-o1-Open-PRM-Qwen-2.5-1.5B or Qwen2.5-Math-PRM-7B . Acc. is the mean over five seeds;

\Delta\mathrm{Acc}

and

\eta

are relative to SC@32. ms/sample reports PRM-signal wall-clock overhead per draft sample.

Asset	Type	Source / Version	License or Terms
GSM8K	Dataset	OpenAI grade-school-math / HF openai/gsm8k	MIT
MATH / MATH500	Dataset	Hendrycks MATH / MATH500 split	MIT
OlympiadBench	Dataset	OpenBMB OlympiadBench	MIT
HotpotQA	Dataset	HotpotQA official release	CC BY-SA 4.0
2WikiMultihopQA	Dataset	Official 2WikiMultihopQA release	Apache 2.0
Qwen2.5 models	Model checkpoint	HuggingFace Qwen release	Apache 2.0 or Qwen license, depending on model size

Appendix

Table 15: Existing assets used in this work.

Method	Seed 42	Seed 123	Seed 456	Seed 789	Seed 1024	Mean Acc.	Mean $\Delta_{\mathrm{SC}}$	$>$ SC
SC@32	72.8	72.8	72.6	73.0	73.0	$72.84\pm 0.17$	$+0.00\pm 0.00$	--
PRM 4 $\times$ 8	72.2	72.0	72.6	72.6	72.6	$72.40\pm 0.28$	$-0.44\pm 0.30$	0/5
PRM 8 $\times$ 4	72.6	73.2	73.2	73.6	73.4	$73.20\pm 0.37$	$+0.36\pm 0.33$	4/5
PRM 16 $\times$ 2	73.2	73.0	74.6	74.2	72.8	$73.56\pm 0.79$	$+0.72\pm 0.86$	4/5

Appendix

Table 16: Per-seed stability on MATH500 across five random seeds using StepBack with PRM-signal (

\delta_{\mathrm{PRM}}=0.05

). We compare SC@32 with three configurations under the same answer budget

B=32

. Here,

n_{d}\times n_{s}

means

n_{d}

draft trajectories with

n_{s}

candidate answers per draft. Seed columns report accuracy in percent; means are reported with the sample standard deviation across seeds.

\Delta_{\mathrm{SC}}

is the paired accuracy difference from SC@32 on the same seed, in percentage points.

>

SC counts the number of seeds where StepBack outperforms SC@32.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 386 papers

ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling
Aug 11, 2026Vaibhav Singh, Soumya Suvra Ghosal, Sarvesh Gharat +3Numerical Reasoning in Language Models Retrieval-Augmented Reasoning
IIT Bombay · University of Maryland, College Park · Adobe Research
Optimal Self-Consistency for Efficient Reasoning with Large Language Models
Nov 15, 2025Austin Feng, Marius Alonso, Ambroise Odonnat +2Self-Consistency Decoding Inference-Time Scaling
1Yale University, New Haven, CT, USA · 2Noah’s Ark Lab, Paris, France · 3Inria, Rennes, France +1
Beyond the Frontier: Stochastic Backtracking for Efficient Test-Time Scaling
May 24, 2026Dao Tran, Duc Anh Le, Ngoc Luu +3Inference-Time Scaling Efficient Language Model Reasoning
Qualcomm AI Research†

Aug 11, 2026J/K move · Enter open · S save

ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

Vaibhav Singh, Soumya Suvra Ghosal, Sarvesh Gharat, Soumyabrata Pal+2

IIT Bombay · University of Maryland, College Park · Adobe Research

Numerical Reasoning in Language Models Retrieval-Augmented Reasoning Test-Time Scaling Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) improve performance by allocating additional inference-time compute to generate extended chain-of-thought reasoning. However, recent studies reveal that sequential test-time scaling often yields diminishing or even negative returns, as longer traces exhibit increased uncertainty, error compounding, and drift from the original problem. We propose ThinkRetrieve, a test-time scaling framework that augments the reasoning traces of LRMs with dynamically retrieved solved examples at each reasoning step. Given an external corpus of problems paired with step-by-step solutions, ThinkRetrieve retrieves relevant exemplars at each intermediate step and injects them directly into the thinking trace, providing the model with guidance on how to reason rather than merely what facts are relevant. Experiments across five reasoning models (1.5B--8B parameters) on GSM-8K, MATH-500, AIME 2025, and SciQ demonstrate that ThinkRetrieve consistently improves accuracy over standard test-time scaling, with relative gains of up to

60\%

on AIME 2025.

Abstract

Efficient Language Model Reasoning

Explore similar work

ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

Optimal Self-Consistency for Efficient Reasoning with Large Language Models

Beyond the Frontier: Stochastic Backtracking for Efficient Test-Time Scaling