Answer Delivery Under a Shared Token Budget: An Audited Study of Stopping and Finalization

Yupeng Chang; Linyuan Gao; Zhiyuan Li; Yuan Wu; Yi Chang

cs.AIOct 19, 2026

Answer Delivery Under a Shared Token Budget: An Audited Study of Stopping and Finalization

Authors: Yupeng Chang, Linyuan Gao, Zhiyuan Li, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Abstract

An inference budget is useful only when the system delivers an answer within it. We study a fixed-budget policy combining first-valid-box stopping with a charged finalization cue. On 100 fresh, independently audited GSM8K test questions per model, it improves delivered endpoint accuracy over first-box stopping by 8.0 percentage points for Qwen3-4B-Thinking-2507 and 7.0 for DeepSeek-R1-Distill-Llama-8B. A prelocked repeat on the same questions gives 7.0 and 10.0 points; neither seed shows a gross harm relative to first-box stopping. The point estimates of incremental physical-slot savings are below one percent. A finalization-only ablation separates cue recovery from stopping, and a fixed scalar reader tests extraction as an alternative explanation. The cue is prior art; the contribution is a component and readout analysis with charged injections and verified paths. Results cover two checkpoints, one task, one prompt family, and two fixed seed conditions.

Figures & tables

Model	Policy	Correct / 100	No value	Mean slots	Mean wall (s)
Qwen	Full	83	14	1179.80	47.64
Qwen	FirstBox	82	14	1158.97	46.81
Qwen	Solution	90	0	1151.69	46.44
R1	Full	80	12	1225.07	36.15
R1	FirstBox	80	12	1054.47	31.16
R1	Solution	87	0	1048.12	30.94

Table 1: All three physically generated policies on 100 fresh GSM8K test questions per model. Correctness uses the numeric value actually returned; No value means the endpoint returned null. Mean post-prompt slots include injected cue tokens. Mean wall time includes generation, monitoring, flush, and result output, excluding shared process setup and offline audit. Bold marks highest correctness among shown arms.

Model	Pair	Slots $\downarrow$ (%)	Wall $\downarrow$ (%)
Qwen	B/F	1.77	1.73
Qwen	S/F	2.38	2.51
Qwen	S/B	0.63	0.80
R1	B/F	13.93	13.79
R1	S/F	14.44	14.41
R1	S/B	0.60	0.71

Table 2: Reductions in total physical slots and measured request wall time. F= Full , B= FirstBox , S= Solution ;

X/Y

compares policy

X

with baseline

Y

. These are singleton instrumented costs on the same paired questions, not serving-throughput estimates.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Primary	Valid	EOS	Legacy
Qwen	83	83	82	82
R1	80	80	78	78

Appendix

Table 3: Descriptive endpoint sensitivities for Full . Primary is the delivered-value rule, Valid selects the last valid box, EOS requires native termination, and Legacy uses the nontruncated score. Entries count correct questions with the same 100-question denominator.

Model	Policy	Total slots	Sampled	Injected	Input tokens	Wall (s)
Qwen	Full	117980	117980	0	126946	4763.57
Qwen	FirstBox	115897	115897	0	124863	4681.28
Qwen	Solution	115169	115025	144	124135	4643.82
R1	Full	122507	122507	0	130721	3614.71
R1	FirstBox	105447	105447	0	113661	3116.08
R1	Solution	104812	104704	108	113026	3093.92

Appendix

Table 4: Physical P0 costs from completed path and process receipts. Each arm contains 100 executed questions per model. Request wall includes generation, flush, and result output; the separately stated whole-process totals contain all three arms and their shared overhead. Nested timing scopes are not summed.

Model	FirstBox $\to$ Solution	$n$	Active	Gain
Qwen	Null $\to$ correct	8	8	8
	Null $\to$ wrong value	6	6	0
	Correct $\to$ correct	82	2	0
	Wrong value $\to$ wrong value	4	0	0
R1	Null $\to$ correct	7	7	7
	Null $\to$ wrong value	5	5	0

Appendix

Table 5: Post-hoc descriptive FirstBox

\to

Solution endpoint transitions on the existing P0 cohort.

n

counts questions, Active counts cue activations, and Gain/Harm count correctness discordances. Models remain separate; subgroup membership does not increase independent sample size.

Model	Policy	Correct	Null	Mean slots	Cue	Wall (s)
Qwen	Full	83	14	1179.80	0	47.64
Qwen	FirstBox	82	14	1158.97	0	46.81
Qwen	Finalize	91	0	1177.18	16	48.20
Qwen	Solution	90	0	1151.69	16	46.44
R1	Full	80	12	1225.07	0	36.15
R1	FirstBox	80	12	1054.47	0	31.16

Appendix

Table 6: Complete same-seed component analysis. F/B/S are the inherited P0 paths; T is the 200-path later ablation. Mean slots include injected tokens, Cue counts activated questions, and wall is mean actual request time. Later T wall contrasts are confounded by stage; inherited P0 costs are not new costs.

Model	Arm	P	R	Null P
Qwen	Full	83	84	14
Qwen	FirstBox	82	83	14
Qwen	Finalize	91	91	0
Qwen	Solution	90	90	0
R1	Full	80	81	12
R1	FirstBox	80	81	12

Appendix

Table 7: Fixed post-hoc R-v1 sensitivity on the existing 100-question cohort per model. P denotes primary delivered correctness, R the alternative reader, and Null the number of absent numeric values. This is a complete offline sensitivity, not a separately deployed policy or additional test sample.

Model	Policy	Correct	Null	Mean slots	Cue	Wall (s)
Qwen	Full	83	14	1181.86	0	47.81
Qwen	FirstBox	83	14	1149.42	0	46.55
Qwen	Solution	90	0	1142.49	17	46.24
R1	Full	75	18	1290.80	0	38.45
R1	FirstBox	75	18	1125.49	0	33.48
R1	Solution	85	1	1116.12	18	33.18

Appendix

Table 8: Complete fixed-seed repeat on the same 100 questions per model. Slots include sampled, forced, and emitted EOS tokens. Wall is mean actual request wall; it does not include shared loading or the offline audit. This is a separate secondary seed condition, not another 100-question cohort or a replicated T arm.

Model	Arm	P	R	Null P
Qwen	Full	83	85	14
Qwen	FirstBox	83	85	14
Qwen	Solution	90	90	0
R1	Full	75	77	18
R1	FirstBox	75	77	18
R1	Solution	85	85	1

Appendix

Table 9: Unchanged R-v1 readout at the second fixed seed. P, R and Null have the same meaning as Table 7 . Every row has 100 questions; adding seeds or policies does not increase independent question count.

Efficient Language Model Inference

Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.

See all 318 papers

When Does Learning to Stop Help? A Cost-Aware Study of Early Exits in Reasoning Models
Jun 29, 2026Zhe Dong, Fang Qin, Manish ShahEfficient Language Model Reasoning Cost-Aware Inference
University of Maine at Presque Isle · Stanford University · Independent Researcher
Thinking Costs Tokens: When More Structure is Worth the Price
Aug 27, 2026Thomas Nolasque, John Grey, Calista Pham +1Efficient Language Model Reasoning Test-Time Scaling
Royal Bank of Canada
Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces
Aug 4, 2026Francesca Carlon, Vincent Ginis, Andres AlgabaLLM Evaluation Overthinking in Language Models
Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussels, Belgium · imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussels, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA

Jun 29, 2026J/K move · Enter open · S save

When Does Learning to Stop Help? A Cost-Aware Study of Early Exits in Reasoning Models

Zhe Dong, Fang Qin, Manish Shah

University of Maine at Presque Isle · Stanford University · Independent Researcher

Efficient Language Model Reasoning Cost-Aware Inference Optimal Stopping Early Stopping Large Reasoning Models Efficient Language Model Inference+1

Open paper AskPDF arXiv

Reasoning models spend test-time compute unevenly across instances, and a growing family of early-exit rules -- confidence thresholds, entropy monitors, answer-stability checks, and learned stoppers -- promises to reclaim the waste. These rules, however, are evaluated under heterogeneous protocols that leave the deployment question unanswered: at a fixed tolerance for losing correct answers, which policy saves more compute, and does the saving survive probe overhead? We answer this question with a controlled study across 18 task-model settings spanning GSM8K, MATH-500, MMLU-Pro, AIME-90, and GPQA on Qwen3 and DeepSeek-R1-distilled models, using LearnStop, a hidden-state-free logistic stopper over prefix-observable features, as the learned policy instrument. Under matched lost-correct risk at

α

= 0.15, with the scalar competitor selected on calibration data from confidence, entropy, confidence-leap, and run-stability exits, the answer forms three regimes. Learned stopping wins on all four primary Qwen3 free-form math settings (+3.2 to +21.2 pp additional total-token saving); calibrated scalar exits win on multiple-choice MMLU-Pro; and small hard benchmarks (AIME-90, GPQA) admit no certifiable aggressive policy at all. A trajectory decomposition predicts the regime: learning pays where answers oscillate and correctness evidence is spread across complementary signals, while a single confidence threshold suffices where most instances are already solved at the first checkpoint. Cost accounting sharpens the picture further -- the same policy that saves 32% of tokens under KV-cache forking costs 121% extra under black-box repeated prefilling. Together, these results replace the single-method race with a decision procedure for choosing a stopping rule from the trajectory structure and serving regime of the target workload.

Abstract

Efficient Language Model Inference

Explore similar work

When Does Learning to Stop Help? A Cost-Aware Study of Early Exits in Reasoning Models

Thinking Costs Tokens: When More Structure is Worth the Price

Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces