$R^3$-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets

Peisong Wang; Zhiwei Ma; Bowen Liu; Feixue Liu; Aochuan Chen; Chenyi Zi; Hongchuan Zeng; Yuhan Li

cs.CLAug 17, 2026

$R^3$ -Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets

Authors: Peisong Wang, Zhiwei Ma, Bowen Liu, Feixue Liu, Aochuan Chen, Chenyi Zi, Hongchuan Zeng, Yuhan Li,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · Hunyuan Team, Tencent

Abstract

In cognitive science, resource rationality asks how an agent should allocate limited computation to maximize expected value. Most reasoning and agent benchmarks use independent per-task budgets; existing shared-budget studies do not calibrate suite performance against the same model's demonstrated single-problem competence. We introduce $R^3$ -Bench, which evaluates six-problem suites under shared budgets across mathematics, competitive programming, and abstract reasoning in tool-free and agentic settings. Matched single-problem response curves define an offline empirical oracle over observed successes. Across 72 main-table cells for six models, the oracle mean matches or exceeds the contest mean in all cells and is strictly higher in 71. Under moderate tool-free pressure, equal-allocation replay also exceeds contest performance for four of six models. Trajectory diagnostics reveal limited strategy updating and pressure-dependent failure patterns. In a three-model diagnostic under strong agentic pressure, at least one fixed scheduler exceeds the contest mean in six of nine cells, but no policy dominates across domains. These results expose a persistent gap between demonstrated competence and shared-budget realization.

Figures & tables

Figure 1: Problem coverage and resource-rational strategy-update rate for the six flagship models, aggregated across tool-free and agentic settings, both budget pressures, and all available domains. Coverage is the share of problem slots with visible, substantive problem-specific work; resource-rational updates are strategy changes that respond to new evidence and the remaining shared budget. Dashed lines mark descriptive 50% references on each metric. Appendix L provides the complete definitions.

Benchmark / Study	Evaluation Setting Domain Tool- Free Agentic Math Code Abstract Reasoning					Shared Budget
Wang et al. (2024)	✓	✗	✓	✗	✓	✗
TALE ( Han et al., 2025 )	✓	✗	✓	✗	✗	✗
SelfBudgeter ( Li et al., 2025 )	✓	✗	✓	✗	✗	✗
BATS ( Liu et al., 2025 )	✗	✓	✗	✗	✗	✗
General AgentBench ( Li et al., 2026 )	✗	✓	✗	✓	✓	✗
ZEBRA ( Hamri and Talgam-Cohen, 2026 )	✗	✓	✗	✓	✗	✗

Table 1: Comparison with budget-aware and resource-allocation-oriented LLM evaluations. A checkmark indicates explicit budget-aware evaluation in the corresponding setting. “Tool-free” denotes direct model reasoning without external tool interaction. “Shared budget” indicates that multiple tasks compete for one common budget, rather than receiving independent per-task budgets or allocating resources only within a single task.

Figure 2: Overview of $R^{3}$ -Bench . \scriptsize1⃝ We construct 50 six-problem contests per domain from frozen 300-problem pools stratified by the average output length of three reference models. Each contest contains three Easy, two Medium, and one Hard problem. \scriptsize2⃝ The same contests are evaluated in tool-free reasoning and agentic settings under a shared budget, measured in output tokens or counted actions. \scriptsize3⃝ Per-problem response curves define an offline multiple-choice knapsack oracle that grants each problem one budget level. The contest–oracle gap measures demonstrated competence lost under the shared budget. \scriptsize4⃝ Trajectory diagnostics characterize how agents observe feedback, revise strategies, and allocate the shared budget.

Model	Regime	Math			Code			AR
Model	Regime	Contest	Oracle	Gap Ratio	Contest	Oracle	Gap Ratio	Contest	Oracle	Gap Ratio
Tool-Free
	$\rho{=}0.2$	2.34	3.94	40.61%	1.38	3.12	55.77%	2.24	4.02	44.28%
DeepSeek-V4-Pro	$\rho{=}0.8$	3.94	5.08	22.44%	3.06	4.62	33.77%	2.90	4.64	37.50%
	$\rho{=}0.2$	3.54	3.70	4.32%	1.42	2.94	51.70%	2.76	4.06	32.02%
Qwen3.7-Max	$\rho{=}0.8$	4.32	5.06	14.62%	3.44	5.24	34.35%	3.82	4.70	18.72%

Table 2: Main results on

R^{3}

-Bench with domains as columns. For each domain, Contest and Oracle report the average number of correct answers per six-problem suite. Gap Ratio denotes

\Delta_{\mathrm{RR}}/\mathrm{Oracle}

, where

\Delta_{\mathrm{RR}}=\mathrm{Oracle}-\mathrm{Contest}

. Lower Gap Ratio is better. AR denotes abstract reasoning. The lowest Gap Ratios are shown in bold , and the second-lowest in underline .

Figure 3: Tool-free contest performance, equal-allocation replay, and response-curve oracle under strong (

\rho=0.2

) and moderate (

\rho=0.8

) budget pressure. Results are averaged equally over mathematics, competitive programming, and abstract reasoning. The unit is average correct answers per six-problem suite.

Figure 4: Tool-free accuracy by presented position, averaged over strong and moderate budget pressure. Means are unweighted over available domains, with missing domains omitted. Accuracy declines from position 1 to position 6 in every model–pressure series.

Figure 5: Online adaptation in the agentic setting. Rates are equal-domain macros over clean trajectories; the annotation rubrics are provided in Appendix K .

Figure 6: Primary causes of positive oracle gaps across models and budget pressures. Segment widths show equal-domain macro shares. Budget spent elsewhere dominates for several models under strong pressure, and partial progress becomes the largest category for most models under moderate pressure.

Model	Domain	Score
		Contest Reference	A	B	Oracle
DeepSeek-V4-Pro	Math	4.48	4.56	4.90	5.28
	Code	2.60	4.10	3.86	4.90
	AR	4.36	4.32	4.72	5.06
GLM-5.2	Math	4.48	4.04	4.20	4.58
	Code	2.02	2.92	2.66	3.90

Table 3: Online scheduler results for DeepSeek-V4-Pro, GLM-5.2, and Hy-3 in the Agentic setting at

\rho=0.2

. Bold marks the best non-oracle score in each model–domain row.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Stage 2 input	Finalization constraint
Mathematics / AR trace-only finalizer	Stage 1 reasoning and visible output; no original problem statements	Extracts trace-supported answers and returns a missing-answer marker when the trace is insufficient
Coding trace-only finalizer	Stage 1 reasoning and visible output; no original problem statements	Prints complete trace-supported C++ programs, or MISSING when the trace is insufficient

Appendix

Table 4: Two-stage protocols used by the reported Tool-free thinking-model contests. In all three domains, Stage 1 receives the run-specific budget

B^{\rho}_{m,d}

and Stage 2 output is logged separately rather than charged to that budget.

Domain	Pure-NL answer contract	Agentic final artifact	Correctness judge
Mathematics	\boxed{...} final answer	/logs/artifacts/answer.txt	Boxed-answer extraction followed by an offline model-based equivalence judge
Competitive programming	Complete standalone C++17 solution, usually in a fenced code block	/app/solution_A.cpp – /app/solution_F.cpp	LightCPVerifier over hidden tests; Accepted is correct
Abstract reasoning	<answer>...</answer> final answer	/logs/artifacts/answer.txt	Reasoning Gym answer extraction and dataset-specific scorers

Appendix

Table 5: Domain-specific answer contracts and correctness judges.

R^{3}

-Bench does not use one global judge for all domains.

Domain	Tool-Free final answer	Agentic contest artifact	Judge
Mathematics	Final Answer: \boxed{...}	/logs/artifacts/answer.txt	model-based equivalence judge
Code	C++17 code block	/app/solution_A.cpp – /app/solution_F.cpp	executable verifier
Abstract reasoning	Final Answer: <answer>...</answer>	/logs/artifacts/answer.txt	rule-based verifier

Appendix

Table 6: Domain-specific answer-format instructions shown in the prompts. All contest prompts require one independently parseable answer per problem.

Action type	Examples	Budget status	Role
Compute command	python , python3 , perl , awk , bc , node , Rscript , julia	Counted	Performs computation used for solving or verification.
Compilation or execution	gcc , g++ , javac , java , locally built executables, local tests	Counted	Compiles, executes, or tests a candidate solution.
Text or data processing	grep , rg , sed , wc , tr , sort , uniq , cut , paste , seq	Counted	Processes task-relevant text or data during solving.
Shell computation	Shell arithmetic, expr , let , or arithmetic loops	Counted	Performs computation directly through the shell.
Passive file operation	Direct file reads, copies, edits, or pure staging of source and scratch files	Free non-compute action	Inspects or records task state without executing computation.
Final artifact write	Writing /app/solution_X.cpp , /logs/artifacts/answer.txt , or an answer slot	Free finalization	Records an existing solution or answer without returning correctness feedback.

Appendix

Table 7: Action-accounting rules under the shared compute_tools policy. The same policy is used for mathematics, competitive programming, and abstract reasoning. Counted actions perform problem-solving computation. Free actions inspect or record state without revealing correctness.

		Tool-free reasoning		Agentic reasoning
Model	Domain	$B^{0.2}$	$B^{0.8}$	$A^{0.2}$	$A^{0.8}$
DeepSeek-V4-Pro	Math	19161	76642	2	8
DeepSeek-V4-Pro	Code	16800	67200	7	28
DeepSeek-V4-Pro	AR	15500	62000	2	7
Qwen3.7-Max	Math	8317	33265	2	10
Qwen3.7-Max	Code	25500	101989	3	12

Appendix

Table 8: Formal shared-budget caps for the six-model evaluation. Pure-NL budgets are output-token caps; agentic budgets are counted-action caps. Every value is read from the frozen without-difficulty cell inventory.

Setting	Model	Regime	Math			Code			AR
			Contest	Oracle	Gap Ratio	Contest	Oracle	Gap Ratio	Contest	Oracle	Gap Ratio
Tool-free	DeepSeek-Chat	$\rho{=}0.2$	2.42	3.24	25.31%	0.86	2.02	57.43%	1.80	3.00	40.00%
	DeepSeek-Chat	$\rho{=}0.8$	3.40	4.06	16.26%	1.68	2.24	25.00%	2.80	3.88	27.84%
	DeepSeek-Reasoner	$\rho{=}0.2$	2.42	3.68	34.24%	1.38	1.88	26.60%	2.14	3.64	41.21%
	DeepSeek-Reasoner	$\rho{=}0.8$	3.70	4.98	25.70%	2.44	3.34	26.95%	2.98	4.32	31.02%
	DeepSeek-V4-Pro	$\rho{=}0.2$	2.34	3.94	40.61%	1.38	3.12	55.77%	2.24	4.02	44.28%

Appendix

Table 9: Main results on

R^{3}

-Bench with domains as columns. For each domain, Contest and Oracle report the average number of correct answers per six-problem suite. Gap Ratio denotes

\Delta_{\mathrm{RR}}/\mathrm{Oracle}

, where

\Delta_{\mathrm{RR}}=\mathrm{Oracle}-\mathrm{Contest}

. Lower Gap Ratio is better. AR denotes abstract reasoning.

Model	$\rho$	Math				Code				AR
		E	M	H	All	E	M	H	All	E	M	H	All
DeepSeek-Chat	0.2	60.7	22.0	16.0	40.3	28.0	1.0	0.0	14.3	42.7	22.0	8.0	30.0
DeepSeek-Chat	0.8	76.0	43.0	26.0	56.7	54.7	2.0	0.0	28.0	58.0	44.0	18.0	46.7
DeepSeek-Reasoner	0.2	56.7	30.0	12.0	40.3	41.3	7.0	0.0	23.0	49.3	27.0	12.0	35.7
DeepSeek-Reasoner	0.8	73.3	57.0	36.0	61.7	65.3	22.0	4.0	40.7	66.7	42.0	14.0	49.7
DeepSeek-V4-Pro	0.2	58.7	26.0	6.0	39.0	36.0	14.0	2.0	23.0	50.7	31.0	10.0	37.3

Appendix

Table 10: Tool-free shared-budget contest accuracy by budget pressure, domain, and difficulty tier. Entries are percentages over the problems evaluated in each condition.

Model	$\rho$	Math				Code				AR
		E	M	H	All	E	M	H	All	E	M	H	All
DeepSeek-Chat	0.2	85.3	62.0	44.0	70.7	75.3	9.0	0.0	40.7	76.7	60.0	50.0	66.7
DeepSeek-Chat	0.8	89.3	71.0	58.0	78.0	93.3	49.0	8.0	64.3	83.3	72.0	70.0	77.3
DeepSeek-Reasoner	0.2	84.7	69.0	46.0	73.0	87.3	21.0	4.0	51.3	82.7	69.0	56.0	73.7
DeepSeek-Reasoner	0.8	90.0	70.0	62.0	78.7	96.7	57.0	6.0	68.3	82.7	75.0	76.0	79.0
DeepSeek-V4-Pro	0.2	91.3	66.0	42.0	74.7	70.0	24.0	2.0	43.3	81.3	64.0	64.0	72.7

Appendix

Table 11: Agentic shared-budget contest accuracy by budget pressure, domain, and difficulty tier. Entries are percentages over the problems evaluated in each condition.

Setting	Domain	$\rho$	Spearman	Perm. $p$
Tool-Free	Math	0.2	+0.10	0.95
Tool-Free	Code	0.2	-0.30	0.68
Tool-Free	AR	0.2	+0.20	0.78
Tool-Free	Math	0.8	-0.80	0.13
Tool-Free	Code	0.8	-0.50	0.45
Tool-Free	AR	0.8	+0.20	0.78

Appendix

Table 12: Discriminant validity: Spearman correlation between model ECI and

R^{3}

-Bench Gap Ratio in each evaluation cell, with exact permutation

p

-values (

n{=}5

models,

120

permutations). Capability ordering explains little of the allocation-gap ordering. Global: mean Spearman

=-0.29

(

p=0.37

); pooled within-cell

R^{2}=0.01

(

p=0.70

Model	Temperature	Top-p	Reasoning / thinking configuration	Independent runs
DeepSeek-Chat	$0.0$	$1.0$	Non-thinking chat completion.	$5$
DeepSeek-Reasoner	Omitted	Omitted	Native reasoning mode; sampling parameters are unsupported and have no effect.	$5$
DeepSeek-V4-Pro	Omitted	Omitted	Thinking enabled with reasoning_effort=high .	$5$
Qwen3.7-Max	$0.2$	$0.8$	Thinking enabled for reasoning runs and disabled for answer-only finalization.	$5$
GLM-5.2	$1.0$	$0.95$	Thinking enabled with high effort for reasoning and agentic tasks.	$5$
Hy-3	$0.9$	$1.0$	reasoning_effort=high for mathematics, coding, and abstract reasoning.	$5$

Appendix

Table 13: Model-call parameters used in the evaluation. Sampling parameters follow official provider recommendations or provider defaults and are not tuned on

R^{3}

-Bench . “Omitted” indicates that a parameter is unsupported or not applicable. GPT-5.5 and Claude-Opus-4.8 are evaluated without thinking because their chain-of-thought content is not visible through our evaluation interfaces. Each reported configuration uses five independent runs per instance.

Configuration item	Coding	Mathematics / abstract reasoning
Runtime container	Docker Harbor task with Terminus-2 shell runtime.	Docker Harbor job with Terminus-2 shell runtime.
Timeouts	Agent timeout $7200$ s; build timeout $600$ s.	Agent and verifier timeout $7200$ s.
Sandbox resources	Memory $2048$ MB; storage $10240$ MB.	$1$ CPU; memory $2048$ MB; storage $10240$ MB.
Tool environment	Local shell, file editing, C++ compilation, local execution, and local tests under the compute_tools policy.	Local shell plus computation-oriented tools under the compute_tools policy.
Trajectory logging	Model messages, tool calls, shell commands, outputs, exposed reasoning content, budget usage, blocked commands, and final artifacts.	Model messages, tool calls, shell commands, outputs, exposed reasoning content, budget usage, blocked commands, and final artifacts.

Appendix

Table 14: Agentic sandbox and logging configuration. The table reports runtime constraints material to the benchmark while omitting credentials, private server identifiers, and deployment-specific endpoint URLs.

Figure 7: Difficulty composition of the curve-oracle portfolio. Dotted cutoffs mark the fixed suite mix: 50% Easy, 33% Medium, and 17% Hard. This oracle-only diagnostic uses all three domains; missing domains are not imputed. Hy Math and AR oracle-dependent values inherit conservative lower-bound oracles.

Figure 8: Curve-oracle selected-miss rates by budget pressure and difficulty tier. Each domain rate uses all judged problems in the explicitly paired suites as its denominator. Values are available-domain macros and missing domains are not imputed. Hy Math and AR oracle-dependent values inherit conservative lower-bound oracles.

Figure 9: Cumulative attributed-output share by presented position for all six models at both budget pressures. The dotted line is a uniform

1/6

-per-position reference. Values are available-domain macros; missing domains are not imputed.

Figure 10: Six-model wasted attributed-output share at both budget pressures, stacked by difficulty. Wasted output is attributed output spent on ultimately incorrect problems. Values are available-domain macros. Missing domains are not imputed.

Figure 11: Available-domain decomposition of pure-NL problem slots at the two pressure levels. Bars separate answered/not-missing problems from budget truncation, formatting or alignment failures, unresolved attempts, and evidence-insufficient or other cases. Missing domains and trajectories are omitted rather than imputed. The allocation-decision category requires decision-time budget checkpoints and is therefore not identifiable here.

Figure 12: Pure-NL answered/not-missing and budget-truncation rates by presented position. Each panel is one model; solid lines show answered rates and dashed lines show truncation, with circles for

\rho=.2

and squares for

\rho=.8

. Each point is an available-domain macro. Missing domains and trajectories are omitted rather than imputed. Answered rates fall and truncation rates rise from the first to the last position in all 12 model–pressure series.

Problem	Tier	Oracle?	Lowest observed successful cost	Contest actions	Outcome	State	Diagnosis
A: 2120E	Medium	Yes	2	0	Missing	Never focused	Cheap oracle-selected miss
B: 2091B	Easy	Yes	2	0	Missing	Never focused	Cheap oracle-selected miss
C: 2107A	Easy	No	3	10	Wrong	Repeated construction, repair, compilation, and testing	Overcommitment target
D: 2029B	Easy	Yes	2	0	Missing	Never focused	Cheap oracle-selected miss
E: 1992F	Medium	Yes	2	0	Missing	Never focused	Cheap oracle-selected miss
F: 2107F2	Hard	No	No observed success	0	Missing	Never focused	No response-curve reachability evidence

Appendix

Table 15: Problem-level allocation in the overcommitment case. Costs are counted actions. The lowest observed successful cost is the cheapest grid level of the sampled single-problem response curve at which the problem was solved in at least one repeat, not the unknown theoretical minimum required to solve the problem.

Stage	Evidence and decision	Budget and consequence
Initial probe	The candidate is a palindrome, but the all-letters constraint is known to be violated; the agent attempts further work on Problem 3	4 remain; progress is substantive but incomplete
Coverage guard	A second paid action is requested before full initial coverage; the runtime blocks it and requires a switch	4 remain; the first switch is protocol-shaped
Shelving	The agent records the defect and continues the coverage pass	4 remain; the candidate is stored but unresolved
Coverage completion	Problems 4–6 each receive one paid probe; return to Problem 3 is now permitted	1 remains
Finalization	The agent notes that one compute step remains and considers revisiting Problem 3, but keeps the current answer	1 unused; the candidate receives non-full credit and is counted incorrect

Appendix

Table 16: Progress, protocol-shaped shelving, and finalization decisions in the abstract-reasoning case.

Problem	Contest reference	Strategy B	Allocation effect
A: 2108B (M)	3 / Wrong	2 / Wrong	One fewer action on the initial target; repeated low-return investment is curtailed
B: 2070B (E)	3 / Accepted	1 / Accepted	Existing success is preserved with two fewer actions
C: 2003D2 (H)	0 / Missing	1 / Wrong	Newly covered, but coverage does not guarantee success
D: 2070C (E)	0 / Missing	1 / Accepted	Newly covered; one answer is recovered
E: 2022E1 (M)	0 / Missing	1 / Accepted	Newly covered; a second answer is recovered
F: 2000C (E)	1 / Accepted	1 / Accepted	Existing success is preserved at unchanged cost

Appendix

Table 17: Matched scheduler comparison on the same coding contest. Both strategies execute seven counted actions. The contest reference scores

2/6

, while Strategy B scores

4/6

. Parenthetical letters denote benchmark tiers.

Model	Domain	Control	Stress	$\Delta$ (pp)
Agentic
DeepSeek-V4-Pro	AR	0.800	0.857	-5.7
	Code	0.917	0.867	5.0
	Math	0.867	0.867	0.0
Qwen3.7-Max	AR	0.835	0.851	-1.6
	Code	0.900	0.817	8.3

Appendix

Table 18: Target-in-suite context-stress results. Control and Stress are mean target scores;

\Delta=\text{Control}-\text{Stress}

in percentage points, so positive values indicate lower performance under context stress.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 924 papers

Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions
Aug 8, 2026Chenrui Fan, Yize Cheng, Ming Li +3LLM Evaluation Inference-Time Compute Allocation
University of Maryland, College Park · MBZUAI, UAE
ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition
Jan 7, 2026Muyang Zhao, Chunhao Tian, Qi Qi +1Cost-Aware Inference Metacognition in Language Models
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Uncertainty-Aware Budget Allocation for Adaptive Test-Time Reasoning
May 26, 2026Manh Nguyen, Sunil Gupta, Hung LeEfficient Language Model Reasoning Adaptive Sampling
Applied Artificial Intelligence Initiative Deakin University, Australia

Aug 8, 2026J/K move · Enter open · S save

Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions

Chenrui Fan, Yize Cheng, Ming Li, Yongyuan Liang+2

University of Maryland, College Park · MBZUAI, UAE

LLM Evaluation Inference-Time Compute Allocation Large Reasoning Models

Open paper AskPDF arXiv

Reasoning language models increasingly use test-time compute to improve performance, but existing evaluations typically study this compute one question at a time. Yet when multiple problems share an end-to-end cost or latency constraint, models must decide how to divide limited inference compute among them. We introduce an exam-style evaluation framework for studying this setting, in which a model must distribute one shared token budget across questions with different difficulty and point values to maximize its total score. Across several open and frontier reasoning models, we find that models fail to allocate a shared budget strategically across questions of varying difficulties and values. Models behave largely as greedy sequential solvers: they prioritize questions by presentation order, front-load effort on early questions, and remain insensitive to value, with these tendencies becoming more pronounced as the number of questions grows. Explicit planning prompts spread compute more evenly but do not produce value- or difficulty-aware prioritization. The same behavioral pattern extends from mathematical to code reasoning. These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation and remains a challenge for current reasoning models.

Abstract

LLM Reasoning

Explore similar work

Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions

ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition

Uncertainty-Aware Budget Allocation for Adaptive Test-Time Reasoning