Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-makers

Zhongyin Wang; Zhaoqian Xue; Junting Mu; Haozheng Liu; Ye Yuan; Haolun Wu; Lingyao Li

cs.CLOct 19, 2026

Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-makers

Authors: Zhongyin Wang, Zhaoqian Xue, Junting Mu, Haozheng Liu, Ye Yuan, Haolun Wu, Lingyao Li

Organizations: University of Illinois Urbana-Champaign · University of Arizona · University of California, Irvine · New York University ·

Abstract

Large language models (LLMs) often generate more reasoning tokens than needed. Limiting the reasoning budget reduces cost, but forcing an answer from an incomplete trace can hurt accuracy. System-1 decision models select among candidate options without producing a reasoning response. We evaluate Jev as a decision reader of truncated reasoning across seven LLM reasoners and five multiple-choice benchmarks (35 pairs). Jev achieves higher mean accuracy than forced answering at tested budgets from 500 to 16,000 tokens, including the estimated costs of reasoning and answering. At a 1,500-token budget, it improves accuracy by 4.1 pp overall and 8.6 pp on truncated questions, with significant gains in 17 pairs under Holm-corrected bootstrap tests. Offline replay estimates 34.5% of the full-run cost, with a 1.13 pp accuracy loss. A policy selected on training folds saves 71.3% of the estimated full-run cost on held-out folds, with a 0.58 pp accuracy loss. We also find that Jev shows strong question-only accuracy, exceeding 500-token forced answering on average (68.0% vs. 65.5%), although this advantage varies by benchmark. These findings suggest that System-1 decision models can complement LLM reasoning and reduce the need for costly reasoning completion.

Figures & tables

Figure 1: Seven LLM reasoners cut at 1,500 tokens (mean over five benchmarks). Left: full-run accuracy and the change when the reasoner answers (blue) or Jev answers (red). Right: cost as % of the uncapped run.

Figure 2: Comparing four answerers by offline replay at a 1,500-token budget. Runs with a valid natural answer within the budget retain that answer for every policy. At a cut, the original reasoner receives the whole prefix, while readers receive the question, options, and the final approximately 1,000 reasoning tokens.

Experiment procedure
	Input: retained traces, answer records, cut settings, window $w$ .
1	for each trace and budget $B$ or fraction $f$ do
2	$c\leftarrow\min(B,L)$ or $\operatorname{round}(fL)$ .
3	if budget mode and a saved generation failure is reached before answering then
4	Record failure. Do not invent an answer call.
5	else if budget mode, $L\leq B$ , and a valid natural answer exists then

Table 1: Offline replay using precollected answers. Policy cost includes only calls used by that policy.

Figure 3: Accuracy of the four answerers, mean of the seven reasoners per benchmark. Top: fixed budget

B

, naturally completed runs keep the reasoner’s answer, so the curves approach one another at large budgets. Bottom: diagnostic cuts at a fraction of each saved trace’s length, known only after generation, and readers are evaluated even at 100%, rather than replaced by the natural answer. Readers see the last approximately 1,000 tokens, where forced answers generally use the whole prefix. Dashed line: full-run accuracy. Per-pair panels are in Appendix I .

	Accuracy (%)				Cost (% of full run)		Jev significantly better / worse (of 35 pairs)
$B$	Jev	Luna-dec.	Luna-gen.	Forced	Jev	Forced	vs. forced	vs. Luna-dec.	vs. Luna-gen.
0	68.0	54.2	55.2	53.4	1.7	4.7	28 / 1	21 / 2	21 / 7
0.5k	73.9	67.2	67.0	65.5	19.1	22.2	29 / 0	22 / 0	22 / 0
1k	75.2	71.1	70.3	70.1	28.2	31.4	25 / 0	20 / 0	21 / 0
1.5k	75.8	72.9	72.5	71.7	34.5	38.3	21 / 0	14 / 1	20 / 0
2k	76.3	73.8	73.3	73.1	39.5	43.4	18 / 0	16 / 0	18 / 0

Table 2: Fixed-budget policies, macro-average over 35 reasoner–benchmark pairs (full run 76.9%). Each policy’s estimated cost includes its retained generation and answering calls. Cost percentages are averaged within pairs, not ratios of pooled dollar totals. Significance counts refer to pairs whose uncorrected 95% paired-bootstrap interval for Jev-minus-comparator accuracy lies entirely above or below zero.

			$B=1.5$ k
Benchmark	Full	Jev $B{=}0$	Jev	L-d	L-g	Forced
GPQA-D	88.3	78.3	82.0	75.5	75.1	72.9
SuperGPQA	66.3	56.2	64.7	60.3	59.6	59.7
MedQA	95.1	88.6	95.0	94.9	94.9	94.1
MedXpertQA	48.9	32.2	49.9	49.3	48.9	47.5
MMLU-Pro	86.1	84.5	87.4	84.5	84.0	84.3

Table 3: Accuracy (%), mean over seven reasoners per benchmark. L-d/L-g: Luna decision/generative.

B=0

is question-only.

B=1.5

k includes the shared natural-answer branch. SuperGPQA is weighted by the full benchmark’s difficulty distribution.

Answerer	Reasoning window	Acc. (%)
Reasoner (forced)	Full prefix	56.41
Reasoner (forced)	Final 1k	57.01
Jev	Full prefix	64.84
Jev	Final 1k	64.97

Table 4:

B=1.5

k. Accuracy is macro-averaged over 35 pairs on the same 6,500 exact-cut outcomes. Naturally completed runs are excluded. Final 1k denotes the saved window of approximately 1,000 reasoning tokens. Acquired unparseable answers count as incorrect.

	vs. full run		vs. tuned cap
Policy	Saving	$\Delta$ Acc	$\Delta$ Saving (+/ $-$ )	$\Delta$ Acc ( $-$ )
Tuned cap	63.3	$-0.13$	–	–
conf+stable	68.8	$-0.42$	$+5.5$ (14/9)	$-0.28$ (3)
settled+useful	68.7	$-0.61$	$+5.4$ (19/1)	$-0.47$ (4)
either	70.8	$-0.62$	$+7.5$ (20/0)	$-0.49$ (5)
auto	71.3	$-0.58$	$+8.0$ (22/0)	$-0.45$ (5)

Table 5: Stop rules with Jev answering, out-of-fold selection, mean over 35 pairs. Saving in % of full run’s cost, and differences in points. In brackets: pairs significantly better/worse (saving) or significantly worse (accuracy). Checks every 500 tokens, where every Jev call is charged. auto selects the rule in the training folds.

Reader	Cut (%)	Fix (%)	Hurt (%)	Net (pp)
Jev	10	6.2	3.7	$+2.5$
	50	12.0	4.9	$+7.0$
	100	14.4	5.2	$+9.2$
Luna dec.	10	8.9	5.3	$+3.5$
	50	17.9	5.5	$+12.4$
	100	24.7	4.9	$+19.8$

Table 6: Trace contribution on the common parseable-answer group (

n=15{,}030

outcomes). Fix: question-only wrong, trace-conditioned right. Hurt: the reverse. Both divide by all included outcomes. Net gain equals fix minus hurt before rounding. A 100% cut supplies only the final 1k reasoning window. Conditional rates are presented in Appendix Q .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Reasoner	Benchmark	$n$	Rerun	No final	Median	Finish
			64k	answer	tokens	$\leq$ 1.5k (%)
DeepSeek V4 Pro	GPQA-D	198	9	1	852	59
	SuperGPQA	500	5	1	501	75
	MedQA	500	0	0	195	94
	MedXpertQA	500	1	0	725	67
	MMLU-Pro	500	3	0	172	86

Appendix

Table 7: Per pair: retained runs, 64k reruns, outcomes without a valid natural answer (including generation failure, scored incorrect for the full run), median reasoning tokens, and share with a valid natural answer within 1,500 tokens.

Criterion (all at $B=1{,}500$ unless stated)	DS V4 Pro	DS V4 Flash	MiniMax M3	Qwen3.8 Flash	Result
1. Reader $-$ forced on cut questions with an exact forced answer: positive in all four, CI excludes 0 in $\geq 2$	$+9.8^{*}$	$+6.8$	$+13.3^{*}$	$+3.5$	pass
	$n=61$	$n=59$	$n=135$	$n=113$
2. Fixed budget + reader not significantly below the full run in any cell	$+2.0^{*}$	$+2.0^{*}$	$+2.4^{*}$	$+1.4$	pass
3. Forced answers lose more than the reader (accuracy change vs full run, forced / reader)	$+0.8$ / $+2.0$	$0.0$ / $+2.0$	$-2.2$ / $+2.4$	$-0.2$ / $+1.4$	pass
4. Tuned cap (out of fold, $t=1$ ): reader saves more dollars than forced, CI excludes 0 in $\geq 3$ ; cap + reader not significantly below the full run	$+10.5^{*}$ / $+1.2$	$+7.7^{*}$ / $+1.4$	$+37.1^{}$ / $+2.6^{}$	$+7.2^{*}$ / $+0.6$	pass
5. Completed reasoning, limit-hit traces removed: reader within 1 point of the reasoner	$+0.4$	$+0.2$	$+1.2^{*}$	$+0.2$	fail

Appendix

Table 8: Pre-specified criteria on the four held-out MMLU-Pro cells. Differences in points; ∗ 95% paired-bootstrap interval excludes 0. Row 4 gives the dollar-saving difference / the accuracy change of cap + reader against the full run. Criterion 5 fails because the reader on the completed reasoning of MiniMax M3 is 1.2 points more accurate than the reasoner’s own answer [0.4, 2.4]; the stated tolerance was symmetric. † Trivially: on all four cells the out-of-fold selection chose a cap of 500 tokens in every fold; the stop rule checks every 500 tokens, so it has no chance to stop before the cap and the two policies coincide.

		Accuracy (%)				Cost (%)
Reasoner	Full	Jev	L-d	L-g	F	Jev	F
DeepSeek V4 Pro	79.0	77.7	76.8	76.7	76.5	35.3	38.7
Qwen3.7 Plus	79.1	77.8	74.5	74.0	74.3	35.2	40.6
MiniMax M3	75.8	74.1	70.1	70.6	69.3	27.2	32.2
GPT-OSS 120B	70.0	70.2	64.3	63.4	59.9	30.3	34.1
GLM-5.3 Flash	78.8	77.3	75.4	74.3	73.3	20.0	21.3

Appendix

Table 9: Accuracy is averaged over five benchmarks at

B=1.5

k on the matched cohort. Cost is the ratio of mean policy dollars to mean full-run dollars across those benchmarks, unlike the main table’s mean of within-pair ratios. Full-run cost per 1,000 questions, in row order: $8.86, 5.46, 6.06, 2.79, 3.22, 3.83, 0.84.

Figure 4: Accuracy against cost per reasoner, mean over the five benchmarks, in dollars per 1,000 questions at list prices (log scale). Each line traces the budgets

B=0.5

k to 64k; labels mark Jev’s points. Open circle: full run.

Reasoner + Jev, $B=1.5$ k
	GPQA-D ( $n=194$ )		SuperGPQA ( $n=297$ )		MedQA ( $n=500$ )		MedXpertQA ( $n=500$ )
Method	Acc.	Cost	Acc.	Cost	Acc.	Cost	Acc.	Cost
Luna none	64.9	0.087	53.4	0.071	91.4	0.059	42.4	0.066
Luna low	85.1	0.305	65.0	0.218	95.0	0.113	53.4	0.197
Luna medium	88.1	0.629	69.0	0.355	95.4	0.147	53.2	0.329
Luna high	88.1	1.028	67.1	0.562	95.6	0.181	53.0	0.494

Appendix

Table 10: Standalone Luna effort baselines and fixed-budget Jev policies on shared questions. Accuracy in %; estimated cost in USD per 1,000 questions. All rows use the same questions within each benchmark, with difficulty weighting on SuperGPQA. Point estimates are descriptive, without significance claims.

Effort	GPQA-D	SuperGPQA	MedQA	MedX.
none	123	89	66	69
low	559	384	176	331
medium	1207	658	243	593
high	2006	1071	311	924

Appendix

Table 11: Mean total output tokens for standalone Luna on the matched subsets of Table 10 . Reasoning tokens are included in output usage and are not added again.

Figure 5: Accuracy of the four answerers for every reasoner–benchmark pair against the reasoning budget

B

. Dashed line: full run.

Figure 6: As Figure 5 , with every question cut at the same fraction of its own reasoning (diagnostic view).

Procedure: adaptive Jev replay
	Input: retained trace, cap $B$ , window $w$ , interval $k$ , selected stopping rule.
1	for each $c\in\{k,2k,\ldots,16{,}000\}$ with $c<\min(B,L)$ do
2	Retrieve Jev’s scores from $(q,\mathcal{O},\operatorname{Tail}_{w}(z_{\leq c}))$ and charge the check.
3	if the stopping rule triggers then
4	Submit the current top option and terminate replay.
5	end if

Appendix

Table 12: Logical adaptive replay. Earlier checks remain charged when a run later finishes. No additional forced-answer call follows a rule-triggered stop.

	Tuned cap		auto
Benchmark	Save	Acc	Save	Acc	Acc
GPQA-D	38.6	87.5	59.6	85.7	88.3
SuperGPQA	64.4	65.4	77.0	65.3	66.3
MedQA	57.4	94.8	60.8	94.8	95.1
MedXpertQA	76.5	49.3	79.6	48.9	48.9
MMLU-Pro	79.6	87.1	79.6	87.1	86.1

Appendix

Table 13: Tuned cap and auto per benchmark, mean of seven reasoners, held-out folds. Saving in % of the full-run cost, accuracy in %.

Reasoner	Benchmark	Tuned cap	auto : family (thresholds), cap	$\Delta$ Saving
DeepSeek V4 Pro	GPQA-D	16k (20)	either (0.95, 0.9, 0.5), 16k (12)	$+13.3$
	SuperGPQA	1.5k (14)	s+u (0.2, 1.0), 1k (9)	$+8.5$
	MedQA	0.5k (25)	cap, 0.5k (25)	$+0.0$
	MedXpertQA	1k (23)	s+u (0.3, 0.7), 1k (13)	$+3.0$
	MMLU-Pro	0.5k (25)	cap, 0.5k (25)	$+0.0$
Qwen3.7 Plus	GPQA-D	8k (19)	either (0.9, 0.9, 0.5), 8k (3)	$+26.1$

Appendix

Table 14: Most frequent selection over the 25 training folds (count in brackets). c+s: conf+stable (

\tau

); s+u: settled+useful (

t_{s}

t_{u}

); either: (

\tau

t_{s}

t_{u}

\Delta

Saving: auto minus tuned cap, points of the full run’s cost.

	Easy	Middle	Hard
Full run	69.1	67.7	61.0
Forced, 1.5k	67.5 (52.5)	63.1 (36.6)	45.4 (19.0)
Jev, 1.5k	69.2 (48.5)	66.9 (32.7)	56.1 (16.3)
Luna-dec., 1.5k	68.4 (49.7)	64.4 (34.0)	44.8 (17.2)
Luna-gen., 1.5k	67.2 (49.5)	64.2 (33.7)	43.7 (17.0)

Appendix

Table 15: SuperGPQA by official difficulty, accuracy in % (cost as % of the full run). Questions are unweighted within each level, then seven reasoners are macro-averaged. The original sample contains 167 / 166 / 167 questions. One Qwen3.8 Flash outcome is excluded from the matched 1.5k comparison because its exact forced answer is unavailable. Tuned-policy results under the main selection protocol are reported in Table 13 .

Benchmark	Model	$n$	R Acc. (%)	Jev Acc. (%)	$\Delta$ (pp)	95% CI (pp)	Agree (%)	Jev+ / R+
GPQA-D	DeepSeek V4 Pro	197	93.4	93.4	+0.0	[0.0, 0.0]	100.0	0 / 0
	Qwen3.7 Plus	198	87.9	88.4	+0.5	[0.0, 1.5]	98.5	1 / 0
	MiniMax M3	194	93.3	93.3	+0.0	[-1.5, 1.5]	99.0	1 / 1
	GPT-OSS 120B	194	77.8	78.9	+1.0	[0.0, 2.6]	98.5	2 / 0
	GLM-5.3 Flash	190	94.2	94.2	+0.0	[0.0, 0.0]	100.0	0 / 0
	Qwen3.8 Flash	189	91.0	93.1	+2.1 *	[0.5, 4.2]	97.9	4 / 0

Appendix

Table 16: Completed reasoning: the reasoner’s own answer versus Jev reading the final approximately 1,000 reasoning tokens, on 15,309 runs with a valid natural answer. Agree is exact option-letter agreement. The last column counts outcomes where only Jev / only the reasoner is correct (77 / 14 in total). ∗ marks an uncorrected 95% paired-bootstrap interval excluding zero (2,000 resamples).

		Jev $-$ forced		Jev $-$ Luna dec.
Benchmark	Flagged	All	Removed	All	Removed
GPQA-D	4	9.2	9.4	6.5	7.3
SuperGPQA	49	5.0	5.7	4.3	4.9
MedQA	1	0.9	0.9	0.1	0.1
MedXpertQA	66	2.4	2.9	0.6	0.7
MMLU-Pro	20	3.1	3.3	2.9	3.0

Appendix

Table 17: Accuracy gaps (points) at

B=1.5

k with all questions and with flagged questions removed, mean of seven reasoners.

Tol.	Cap	Cap with $B{=}0$	Jev $\to$ full	Jev $\to$ cap
0.25	57.6 / $+0.05$	59.1 / $+0.07$	33.0 / $-0.29$	71.4 / $-0.70$
0.5	59.7 / $+0.03$	61.4 / $+0.04$	35.1 / $-0.40$	73.3 / $-0.83$
1	63.3 / $-0.13$	65.2 / $-0.17$	39.6 / $-0.79$	76.9 / $-1.21$
2	68.7 / $-0.83$	71.5 / $-0.96$	48.2 / $-1.50$	82.2 / $-2.17$

Appendix

Table 18: Cost saving (%) / accuracy change (points) against the full run on held-out folds, mean over 35 pairs. Tol.: accuracy loss allowed on the training folds. References: fixed 1.5k with Jev 65.5 /

-1.13

; Jev alone 98.3 /

-8.97

; stop rule auto (tol. 1) 71.3 /

-0.58

$B$	(%)	(tok.)	All cut	Exact only	All cut	Exact only
	Exact	Offset	Jev $-$ forced		Sig. better / worse
0.5k	93.7	500	$+11.5$	$+11.3$	29 / 0	30 / 0
1k	92.7	668	$+8.8$	$+8.0$	25 / 0	21 / 0
1.5k	100.0	–	$+8.6$	$+8.6$	21 / 0	21 / 0
2k	88.4	631	$+8.1$	$+7.6$	18 / 0	18 / 0
4k	82.1	615	$+5.3$	$+4.5$	10 / 0	8 / 0

Appendix

Table 19: Exact coverage is averaged over pairs. Offset is the mean fallback distance in nominal tokens. Accuracy gaps condition on cut outcomes using the main analysis’s question weights. The exact-only subset requires at least five cut outcomes per pair (32 eligible pairs at 8k; 25 at 16k). Significance uses 100,000 paired-bootstrap resamples, uncorrected. At 1.5k, all 6,500 matched cut outcomes have exact forced answers; no fallback offset applies.

Reader	Q-only	10%	50%	100%	Fixed	Hurt
		Net change (pts)			At 100% (%)
Jev	66.7	$+2.5$	$+7.0$	$+9.2$	43.1	7.7
Luna dec.	56.0	$+3.5$	$+12.4$	$+19.8$	56.1	8.8
Luna gen.	56.5	$+2.8$	$+11.3$	$+19.1$	55.6	9.1

Appendix

Table 20: Question-only accuracy (Q-only) and net change at the given trace fraction, pooled over 15,030 model–question outcomes from 2,182 unique questions (unweighted, including SuperGPQA). Fixed: share of incorrect question-only answers corrected; hurt: share of correct question-only answers made incorrect.

Subset	Cut at	Jev	Luna dec.	Luna gen.
All question-only answers	10%	13.4	12.8	14.5
incorrect ( $n=3{,}455$ )	50%	27.1	26.7	25.5
	100%	34.2	34.0	33.7
All question-only answers	10%	97.8	96.5	94.6
correct ( $n=6{,}597$ )	50%	96.9	95.7	95.3
	100%	96.3	96.2	96.3

Appendix

Table 21: Accuracy (%) after reading a prefix, on outcomes where all three readers are wrong (top) or all right (bottom) from the question alone; pooled over pairs, unweighted. Counts are model–question outcomes, not independent questions.

	Saving	Jev calls	Questions (%)			Costlier
Policy	(%)	per question	Rule	Cap	Finished	than cap
Tuned cap	63.3	0.48	–	48.2	51.8	–
conf+stable	68.8	1.73	20.0	40.9	39.1	9 (9)
settled+useful	68.7	1.92	28.7	39.0	32.3	4 (1)
either	70.8	1.69	30.1	38.0	31.9	1 (0)
auto	71.3	1.52	27.3	37.9	34.7	0 (0)

Appendix

Table 22: Stop rules on held-out folds, mean over 35 pairs. Rule / Cap / Finished: share of questions stopped by the rule, cut at the cap, or finished within the cap. Costlier than cap: pairs where the rule costs more than the tuned cap (significantly in brackets).

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 230 papers

Broken Chains: The Cost of Incomplete Reasoning in LLMs
Feb 16, 2026Ian Su, Gaurav Purushothaman, Jey Narayan +5LLM Reasoning CoT Reasoning
System1-Arena: Which Decisions Can Be Left to System 1?
Oct 15, 2026Tianyu Chen, Xinyang Liu, Ruyi An +2LLM Decision-Making LLM Evaluation
The University of Texas at Austin
Releasing the Inner Jev: A Simple Recipe for Open Decision Models
Oct 12, 2026Jonathan Geuter, Andrew Cheng, David Alvarez-MelisLLM Decision-Making LLM Fine-Tuning
Harvard University · Kempner Institute

Feb 16, 2026J/K move · Enter open · S save

Broken Chains: The Cost of Incomplete Reasoning in LLMs

Ian Su, Gaurav Purushothaman, Jey Narayan, Ruhika Goel+4

LLM Reasoning CoT Reasoning Efficient Language Model Reasoning Constraint-Guided Reasoning in Language Models

Open paper AskPDF arXiv

Reasoning-specialized models like OpenAI's 5.1 and DeepSeek-V3.2 allocate substantial inference compute to extended chain-of-thought (CoT) traces, yet reasoning tokens incur significant costs. How do different reasoning modalities of code, natural language, hybrid, or none do perform under token constraints? We introduce a framework that constrains models to reason exclusively through code, comments, both, or neither, then systematically ablates token budgets to 10%, 30%, 50%, and 70% of optimal. We evaluate four frontier models (GPT-5.1, Gemini 3 Flash, DeepSeek-V3.2, Grok 4.1) across mathematical benchmarks (AIME, GSM8K, HMMT). Our findings reveal: (1) truncated reasoning can hurt as DeepSeek-V3.2 achieves 53% with no reasoning but only 17% with truncated CoT at 50% budget; (2) code degrades gracefully as Gemini's comments collapse to 0% while code maintains 43-47%; (3) hybrid reasoning underperforms single modalities; (4) robustness is model-dependent as Grok maintains 80-90% at 30% budget where OpenAI and DeepSeek collapse to 7-27%. These results suggest incomplete reasoning chains actively mislead models, with implications for deploying reasoning-specialized systems under resource constraints.

Abstract

LLM Decision-Making

Explore similar work

Broken Chains: The Cost of Incomplete Reasoning in LLMs

System1-Arena: Which Decisions Can Be Left to System 1?

Releasing the Inner Jev: A Simple Recipe for Open Decision Models