Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models

Wei Yu; Suxing Liu; Minjie Yu; Jiahao Wang; Zhijian Zheng; Haocheng Deng; Bing Li

cs.AIAug 19, 2026

Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models

Authors: Wei Yu, Suxing Liu, Minjie Yu, Jiahao Wang, Zhijian Zheng, Haocheng Deng, Bing Li

Organizations: School of Digital Arts, Jiangxi Arts & Ceramics Technology Institute, Jingdezhen 33001, China · School of Computing, Universiti Sains Malaysia, 11800 USM Penang, Malaysia

Abstract

Reinforcement-learning training of reasoning LLMs (e.g., GRPO) is expensive and requires a controllable environment, committing every contribution to a full training pipeline. We present EvoResearcher, a training-free, inference-time protocol that adds cost-bounded self-reflection to a single frozen LLM backbone. The protocol iterates generate -> self-critique -> revise until a maximum depth D is reached or the critique returns the CONFIRMED sentinel, an implicit early stop that lets the backbone self-verify its answer under a strict compute budget. Four self-reflective meta-reward components (correctness, efficiency, reflection depth, tool-call diversity) act as design principles instantiated as prompt-level mechanisms, so their benefits accrue with zero gradient updates. We validate the protocol on Big-Bench Hard (100 questions) and establish cross-domain behavior on GSM8K (500) and MATH (500) on the same frozen backbone, with cross-model replication on Qwen2.5-72B. All experiments use pure-reasoning benchmarks; the tool-call diversity component is validated in prompt-level form, and the environment-level and multi-agent extensions are design blueprints left to future work. On clean BBH the protocol does not raise accuracy beyond the 95% Wilson interval; its value is cost-bounded self-verification, with the CONFIRMED early stop terminating 82-88% of items at equal accuracy (about 2.1 generations per question).

Figures & tables

Figure 1: Conceptual overview of the EvoResearcher framework. Four synergistic dimensions are combined: a self-reflective meta-reward (M3) over correctness, path efficiency, reflection depth, and tool-call diversity; an evolving virtual world (M1) that injects time-dependent and adversarial content (expert opinions, scientific retractions, delayed evidence, misleading material); discovery-oriented tasks (M2) that go beyond fact retrieval toward hypothesis generation and contradiction resolution; and a heterogeneous multi-agent swarm (M4) of Scout/Filter/Synthesizer agents with task specialization. The agent draws on retrieval and search-tool infrastructure and is trained via curriculum reinforcement learning from a web-scale corpus. In this work the meta-reward components are realized as a training-free, inference-time self-reflective protocol over a frozen backbone (Section 3.1 ); the RL training pipeline, evolving environment, and multi-agent swarm depicted here are design blueprints and are not evaluated (Section 5 ).

Component	Formulation	Purpose	Behavior
$R_{\text{correctness}}$	Binary LLM-judge	Answer accuracy	Correct answering
$R_{\text{efficiency}}$	Step-count penalty	Concise search	Avoid excess calls
$R_{\text{reflection}}$	Hybrid backtrack indicator	Self-correction	Error recovery
$R_{\text{diversity}}$	Query/domain ratio	Exploration	Avoid repetition

Table 1: Meta-reward component design.

Component	Inference-time mechanism	Prompt	Experiment
$R_{\text{correctness}}$	Self-verification in critique	“verify every step”	E3, E4
$R_{\text{efficiency}}$	Bounded depth $D$ / CONFIRMED early stop	budget + sentinel	E4
$R_{\text{reflection}}$	Structured critique	critique framings	E1, E3, E4
$R_{\text{diversity}}$	Multi-path / evidence weighing	“weigh evidence across interpretations”	E3

Table 2: From reward components to inference-time mechanisms. Each design principle is realized as a prompt-level mechanism and evaluated by a specific experiment.

Figure 2: The four meta-reward components and the policy-update loop they drive. The self-reflective meta-reward $R_{\text{meta}}$ combines the correctness reward $R_{c}$ (reference-based answer comparison), the path-efficiency reward $R_{e}$ (penalizing excessive steps), the reflection-depth reward $R_{r}$ (rewarding explicit error recognition and strategic backtracking), and the tool-diversity reward $R_{d}$ (rewarding query/tool scaling, penalizing repetition) via Eq. ( 1 ). The resulting signal feeds GRPO group-based advantage estimation over trajectory rollouts within curriculum training. In this work the same components are instantiated at inference time as the prompt-level mechanisms of Section 3.1 and are not computed numerically; the process-reward formulation and GRPO training depicted here are a design blueprint (Section 5 ).

Method	Stopping criterion	Avg. gen.	Avg. tok./q.	BBH	GSM8K	MATH
Single-shot ( $d{=}1$ )	None	1.00	$\sim$ 890	73.8%	93.6%	26.2%
Self-Consistency ( $k{=}2$ )	Fixed (2 paths)	2.00	$\sim$ 1,780	74.2%	96.0%	32.4%
Self-Consistency ( $k{=}3$ )	Fixed (3 paths)	3.00	$\sim$ 2,670	74.5%	96.8%	35.6%
Self-Refine ( $d{=}3$ )	Fixed depth	3.00	$\sim$ 2,650	71.2% ∗	97.0%	36.8%
EvoResearcher ( $d{=}3$ )	CONFIRMED sentinel	2.15	$\sim$ 1,050	74.0%	97.8%	40.4%

Table 3: Performance and compute-cost comparison under matched compute budgets (deepseek-v4-flash,

n{=}500

). ∗ Self-Refine’s BBH accuracy reflects the over-correction of fixed-depth refinement on clean logical tasks.

Condition	Mean Acc.	Runs	Role
C1: Vanilla ReAct	73.3%	74/73/73	Baseline
C2: + Correctness	73.3%	72/73/75	Meta-reward component
C3: + Efficiency	73.7%	73/74/74	Meta-reward component
C4: + Reflection	74.0%	73/74/75	Meta-reward component
C5: + Diversity	74.0%	73/74/75	Meta-reward component
C6: Full Meta-Reward	73.3%	74/74/72	Complete reward

Table 4: E1: Controlled comparison on the same backbone (deepseek-v4-flash), BBH 100 questions. “Runs” gives the accuracy at temperatures

\{0.1,0.3,0.5\}

Configuration	Accuracy
A: Full meta-reward	73.0%
B: $-{}$ Efficiency	75.0%
C: $-{}$ Reflection	71.0%
D: $-{}$ Diversity	71.0%
E: Outcome-only	73.0%

Table 5: E2: Reward component ablation on BBH (accuracy, temperature 0.3).

Critique framing	Accuracy	Avg. Steps	Loop Rate
Vanilla	69.0%	2.05	100%
Meta-reward	73.0%	2.11	100%

Table 6: E3: Reflective loop (depth 3) under vanilla vs. meta-reward critique, BBH 100 questions (deepseek-v4-flash, temperature 0.3).

Max depth $D$	Accuracy	95% CI	Avg. Steps	Loop Rate	Early-Stop Rate
1 (single-shot)	74.0%	[64.6, 81.6]	1.00	0%	0%
2	74.0%	[64.6, 81.6]	2.00	100%	85%
3	74.0%	[64.6, 81.6]	2.08	100%	82%
4	73.0%	[63.6, 80.7]	2.06	100%	88%

Table 7: E4: Reflection-depth sweep on BBH (deepseek-v4-flash, temperature 0.3, 100 questions).

	Correct	Incorrect
Emitted CONFIRMED (stop)	TP: 68%	FP: 16%
Proposed revision (loop)	FN: 6%	TN: 10%

Table 8: Confusion matrix of the CONFIRMED sentinel at the first reflection step (BBH,

n{=}500

). Rows are the critique’s decision; columns are the correctness of the Step 1 answer. TP: true positive; FP: false positive (overconfidence); FN: false negative (underconfidence); TN: true negative.

Prompt	Accuracy	95% CI
Clean	73.0%	[63.6, 80.7]
Misleading context	72.0%	[62.5, 79.9]

Table 9: E5: Adversarial robustness probe on BBH (deepseek-v4-flash, temperature 0.3, single-shot).

$\tau$	Acc.	95% CI	Avg. steps	Early-stop	Calls saved
0.0 (always loop)	76.0%	[66.8, 83.3]	2.06	83%	—
0.6	79.0%	[70.0, 85.8]	1.00	100%	51%
0.7	81.0%	[72.2, 87.5]	1.00	100%	51%
0.8	80.0%	[71.1, 86.7]	1.00	100%	51%
0.9	77.0%	[67.8, 84.2]	1.00	100%	51%
1.0 (single shot)	81.0%	[72.2, 87.5]	1.00	0%	51%

Table 10: E6: Confidence-threshold sweep on BBH (deepseek-v4-flash, temperature 0.3, depth 3). Calls saved are relative to the always-loop anchor.

Benchmark	Condition	Accuracy	95% CI (Wilson)	Avg. steps	Early-stop
GSM8K	single-shot (d1)	93.6%	[91.1, 95.5]	1.00	0%
GSM8K	reflective loop (d3)	97.8%	[96.0, 98.9]	2.15	88%
MATH	single-shot (d1)	26.2%	[22.5, 30.2]	1.00	0%
MATH	reflective loop (d3)	40.4%	[36.2, 44.8]	2.42	82%

Table 11: E7: Single-shot vs. reflective loop on GSM8K and MATH (deepseek-v4-flash, temperature 0.3,

n{=}500

Condition	Benchmark	Acc.	Avg. tokens	Avg. calls
Single-shot (d1)	BBH	81.0%	892	1.00
Loop d3, $\tau{=}0$	BBH	76.0%	1061	2.06
$\tau{=}0.7$ (collapse)	BBH	81.0%	453	1.00
Single-shot (d1)	GSM8K	93.6%	541	1.00
Loop d3	GSM8K	97.8%	993	2.15
Single-shot (d1)	MATH	26.2%	886	1.00

Table 12: E8: Compute cost vs. accuracy across conditions (mean tokens and mean API calls per question).

Figure 3: E8: Compute cost (mean tokens per question) vs. accuracy across conditions (deepseek-v4-flash; $n{=}100$ for the BBH anchors, $n{=}500$ for GSM8K/MATH). Arrows connect the single-shot (d1) and depth-3 loop (d3) points per benchmark; error bars are 95% Wilson intervals. The CONFIRMED sentinel bounds the loop at $\approx 2.1$ –2.4 steps, and the $\tau{=}0.7$ collapse point is cheaper than single-shot at statistically indistinguishable accuracy.

Benchmark	Condition	Accuracy	Avg. steps	Early-stop
BBH	single-shot (d1)	78.2%	1.00	0%
BBH	reflective loop (d3)	78.4%	2.18	81%
MATH	single-shot (d1)	31.6%	1.00	0%
MATH	reflective loop (d3)	43.2%	2.45	78%

Table 13: E9: Cross-model replication using Qwen2.5-72B-Instruct (

n{=}500

H	Dimension	Expected	Observed (BBH, deepseek-v4-flash)	Status
H1	Component framing (E1)	No component changes single-shot acc.	73.3–74.0%, all within CI	$\checkmark$
H2	Ablation (E2)	No component is essential	removal $\leq 4$ pp, all within CI	$\checkmark$
H3	Loop (E3)	Meta $\geq$ vanilla critique	73.0% vs. 69.0% ( $+4.0$ pp, n.s.)	$\circ$
H4	Depth (E4)	No drop; majority early-stop	flat 74/74/74/73; early-stop 82–88%	$\checkmark$
H5	Robustness (E5)	Drop $<5$ pp under weak probe	$-1.0$ pp, within CI	$\checkmark$
H6	Confidence threshold (E6)	Calibrated, selective $\tau^{*}$ : fewer steps, acc. within $\tau{=}0$ CI	No interior point—every $\tau\geq 0.6$ halts at step 1 (overconfident tags); the sentinel bounds cost	$\times$

Table 14: Hypothesis validation status. Status:

\checkmark

SUPPORTED,

\circ

PARTIAL,

\times

REFUTED (final values from Sections 4.8 – 4.10 ).

Language Model Self-Verification

See all 16 papers

Enhancing reasoning accuracy in large language models during inference time
Mar 22, 2026Vinay Sharma, Manish JainLLM Inference LLM Reasoning
A Training-Free Regeneration Paradigm: Contrastive Reflection Memory Guided Self-Verification and Self-Improvement
Mar 20, 2026Yuran Li, Di Wu, Benoit BouletLanguage Model Self-Verification Language Model Self-Improvement
Department of Electrical and Computer Engineering, McGill University · Department of Mechanical, Industrial and Aerospace Engineering, Concordia University
TTSR: Test-Time Self-Evolving via Reflection
Feb 6, 2026Haoyang He, Zihua Rong, Yunjia Zhao +3LLM Self-Refinement Test-Time Optimization
Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.

Mar 22, 2026J/K move · Enter open · S save

Enhancing reasoning accuracy in large language models during inference time

Vinay Sharma, Manish Jain

LLM Inference LLM Reasoning Inference-Time Guidance Self-Consistency Decoding

Open paper AskPDF arXiv

Large Language Models (LLMs) often exhibit strong linguistic abilities while remaining unreliable on multi-step reasoning tasks, particularly when deployed without additional training or fine-tuning. In this work, we study inference-time techniques to improve the reasoning accuracy of LLMs. We systematically evaluate three classes of inference-time strategies: (i) self-consistency via stochastic decoding, where the model is sampled multiple times using controlled temperature and nucleus sampling and the most frequent final answer is selected; (ii) dual-model reasoning agreement, where outputs from two independent models are compared and only consistent reasoning traces are trusted; and (iii) self-reflection, where the model critiques and revises its own reasoning. Across all evaluated methods, we employ Chain-of-Thought (CoT) [1] prompting to elicit explicit intermediate reasoning steps before generating final answers. In this work, we provide a controlled comparative evaluation across three inference-time strategies under identical prompting and verification settings. Our experiments on LLM [2] show that self-consistency with nucleus sampling and controlled temperature value yields the substantial gains, achieving a 9% to 15% absolute improvement in accuracy over greedy single-pass decoding, well-suited for low-risk domains, offering meaningful gains with minimal overhead. The dual-model approach provides additional confirmation for model reasoning steps thus more appropriate for moderate-risk domains, where higher reliability justifies additional compute. Self-reflection offers only marginal improvements, suggesting limited effectiveness for smaller non-reasoning models at inference time.

Abstract

Language Model Self-Verification

Explore similar work

Enhancing reasoning accuracy in large language models during inference time

A Training-Free Regeneration Paradigm: Contrastive Reflection Memory Guided Self-Verification and Self-Improvement

TTSR: Test-Time Self-Evolving via Reflection