Generative models increasingly expose a reasoning effort setting that lets users scale inference-time compute, trading higher cost for potentially improved performance. Users must choose an effort level before knowing whether extra reasoning will help, and even a successful run does not reveal whether the effort purchased was necessary. Reasoning effort is therefore a credence good: buyers observe the outcome but cannot assess the marginal value of the computation they paid for. In this work, we measure the counterfactual effect of effort levels on both proprietary and open-weight models on four benchmarks, spanning single-call reasoning and long-horizon agentic tasks. Effort, it turns out, does not consistently preserve tasks that lower effort solves, can overspend, and can change trajectory behavior. First, we find that although higher effort can raise accuracy in aggregate, it frequently fails on tasks that lower effort solved. Second, on the subset of easy tasks that every effort level solves, higher effort is generally more costly, with the highest effort level costing up to 42x as much as the lowest. Lastly, effort changes not only how many tokens are used, but also how agents solve tasks: at higher effort, agents verify their own work more heavily and more often search for solutions than work through problems directly, both of which lengthen trajectories without necessarily altering the outcome. These findings show that naively choosing the highest level is not only cost-inefficient but also offers no guarantee of the best instance-level performance. A simple alternative, starting at the lowest effort and escalating only after a verified failure, matches or improves on fixed top effort across all models and benchmarks at lower cost for most proprietary models.
Figures & tables
Figure 1: Instances gained and lost between adjacent reasoning-effort settings on the agentic benchmarks. Higher effort can solve additional instances while losing some lower-effort successes. AIME, GPQA, and a corrected version of SWE-Bench Pro are included in Appendix Figure 7 .
Figure 2: API cost of one task the model solves at every effort level, including none if supported, relative to the cost at low (dashed line; GLM-5.2 is anchored on high , its lowest reasoning level). Kimi K3 has no none setting. Each bar is split into input, hidden reasoning, and visible output tokens. The always-solved set is fixed per model and benchmark, so bars compare prices within a group and only vary across groups; the footer gives the dollar value of 1× . Bars taller than the axis are clipped and labelled.
Figure 3: GPT-5.6-sol on matched SWE-Bench Pro instances across reasoning effort levels. Left: reference-fix reads from official git history and online lookup attempts in the corrected environment, regardless of retrieval success. Right: instances resolved. Corrected removes the original git history and restricts network access to allowlisted package registries.
Figure 4: Changes in the mean per-trajectory share of annotated prose assigned to each Act·ONOMY behavior group, in percentage points, from the lowest to the highest annotated reasoning-enabled setting on matched tasks. A dagger ( † ) marks a change whose unadjusted 90% paired task-bootstrap confidence interval excludes zero (10,000 resamples); markers use unrounded estimates.
Figure 5: Self-routing and naïve laddering compared with fixed efforts, random allocation, and the oracle on Terminal-Bench 2.1; Appendix I shows the other models with saved self-routing comparisons. Each panel shows success against total API cost, normalized by the cost of that model’s fixed effort with the highest performance (dashed line, effort level noted in gray text). Naïve laddering (green triangles) starts at the lowest available reasoning effort and retries at the highest effort after an externally verified failure, charging both executions when it retries. All policies in each panel use the same task cohort. Lower cost and higher success are preferable.
Figure 6: GPT-5.6-terra on Terminal-Bench 2.1 at fixed high , fixed xhigh , and naïve laddering (green). Bars show mean cost per task (left axis) as the API bill and as full cost (Equation 1 ), labelled with their ratio to xhigh ; dots show tasks solved (right axis) with 95% Wilson intervals.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Evaluation size
Execution setting
AIME 2026 Dekoninck et al. (2026)
30 problems, 10 attempts each
Single API call per attempt
GPQA Diamond Rein et al. (2023)
198 questions, 4 attempts each
Single API call per attempt
Terminal-Bench 2.1 Merrill et al. (2026)
89 tasks
Multi-turn terminal agent using Terminus-2
SWE-Bench Pro Deng et al. (2025)
44-instance pilot
Multi-turn repository-level agent using mini-swe-agent
Appendix
Table 1: Benchmarks and execution settings. AIME uses ten attempts per problem and GPQA uses four attempts per question; the agentic benchmarks evaluate multi-turn executions with fixed harnesses.
Model
API
Effort settings
GPT-5.6-sol, GPT-5.6-terra, GPT-5.6-luna
OpenAI
none , low , medium , high , xhigh
Claude Sonnet 5
Anthropic
none , low , medium , high , xhigh
GLM-5.2
Together AI
none , high , max
Inkling
Together AI
none , low , medium , high , max
Kimi K3
Together AI
low , high , max
DeepSeek V4 Flash
Together AI
none , low , high , max
Appendix
Table 2: Serving API and evaluated reasoning-effort settings for each model.
Model / cost category
AIME
GPQA
SWE-Pro
TB
Total (USD)
Benchmark inference
GPT-5.6-sol
72.4
125.9
1,142.7
165.1
1,506.0
GPT-5.6-terra
29.2
47.6
312.1
131.8
520.8
GPT-5.6-luna
4.9
10.2
15.4
19.3
49.8
Claude Sonnet 5
217.2
230.3
1,045.2
322.3
1,815.0
GLM-5.2
44.7
126.7
804.9
255.7
1,232.1
Appendix
Table 3: Experimental cost in USD, rounded to one decimal place.
Model
none
low
medium
high
xhigh
max
AIME 2026 (30 problems; 10 attempts per problem)
GPT-5.6-luna
25.3 [14, 38]
75.7 [65, 85]
85.3 [75, 94]
94.7 [89, 99]
96.7 [93, 99]
–
GPT-5.6-terra
7.3 [2, 14]
89.7 [81, 96]
91.3 [82, 98]
92.3 [86, 98]
97.0 [94, 99]
–
GPT-5.6-sol
55.3 [42, 68]
96.7 [90, 100]
97.7 [95, 100]
99.7 [99, 100]
99.7 [99, 100]
–
Claude Sonnet 5
79.7 [68, 90]
72.3 [60, 83]
86.3 [78, 93]
96.7 [92, 99]
97.3 [92, 100]
–
GLM-5.2
46.0 [35, 57]
–
–
86.3 [76, 95]
–
90.7 [81, 98]
Appendix
Table 4: Overall accuracy (%) across reasoning-effort settings, with 95% confidence intervals in brackets, rounded to whole percentage points. AIME and GPQA report mean per-attempt correctness, with percentile bootstrap intervals (10,000 resamples) over problems that keep each problem’s attempts together; the agentic benchmarks report the fraction of tasks solved, with Wilson score intervals. Intervals describe each setting separately; because all settings of a model share the same tasks, paired tests between settings are more sensitive than a comparison of overlapping intervals. Bold marks the highest accuracy within each model–benchmark row, including ties; dashes indicate settings without a reported result. Effort names retain their model-specific meanings.
Figure 7: Instances gained and lost between adjacent reasoning-effort settings on AIME, GPQA, and the corrected SWE-Bench Pro environments. An instance counts as solved at a level if any of its attempts at that level is correct.
Figure 8: Self-routing and naïve laddering on Terminal-Bench 2.1 with user waiting cost added to the API bill (Equation 1 ), for the seven models with saved self-routing comparisons. Each panel shows success against full cost, normalized by the full cost of that model’s fixed effort with the highest performance (dashed line). The oracle selects the successful effort with the lowest full cost; router calls are charged their API cost only. Policies, markers, and task cohorts match Figure 5 . Lower cost and higher success are preferable.
Figure 9: Full cost of one task the model solves at every effort level, including none if supported, relative to the full cost at low (dashed line; GLM-5.2 is anchored on high ). Kimi K3 has no none setting. Full cost adds to the API bill the time a user waits for the output tokens at the model’s published throughput, priced at the BLS mean hourly wage of 33.54.Eachbarissplitintoinput,hiddenreasoning,visibleoutput,andwaiting.Thealways−solvedsetisfixedpermodelandbenchmark;thefootergivesthedollarvalueof1\times$ . Bars taller than the axis are clipped and labelled.
Solved only in
Model
Effort
Original (%)
Corrected (%)
Δ (pp)
Original
Corrected
p
GPT-5.6-luna
xhigh
84.1
56.8
−27.3
13
1
0.002
GPT-5.6-terra
xhigh
72.7
61.4
−11.4
7
2
0.180
GPT-5.6-sol
xhigh
77.3
59.1
−18.2
8
0
0.008
Claude Sonnet 5
xhigh
79.5
50.0
−29.5
13
0
< 0.001
GLM-5.2
max
70.5
43.2
−27.3
13
1
0.002
Appendix
Table 5: Solve rates on the original and corrected SWE-Bench Pro environments at each model’s highest evaluated reasoning-effort setting, using the same 44 instances in both environments. Corrected removes the task repository’s original git history and restricts network access to allowlisted package registries. Δ is corrected minus original solve rate in percentage points, computed before rounding. The Solved only in columns count discordant instances, and p is the two-sided exact McNemar test on those counts. The decrease is significant at α=0.05 for 5 of 8 models, and a sign test over the 8 models, all of which decrease, gives p=0.008 . Both conditions follow the source scoring rule that evaluator-passed executions costing more than $5 count as failures. Table 4 reports results at all effort levels.
Figure 10: The summarized trajectory of GPT-5.6-sol on the SWE-Bench Pro instance openlibrary-00bec1e7 at none (left, 15 calls, 0.19)andxhigh(right,27calls,1.00); both runs resolve the task. Each row is a phase of consecutive agent steps and bar length is the number of steps in that phase. Grey marks routine work, blue a genuine test run, orange a look into git history or validation against the copied file, and red a read or copy of the gold fix commit. At none the agent fixes the validator from the issue text and the code. At xhigh it does the same work, then runs git log --all , finds the gold fix commit still present in the container’s clone, overwrites its own file with git show <gold>:<file> , runs that commit’s tests, and confirms a byte-for-byte match before submitting.
Figure 11: Mean API cost per always-solved task in the corrected SWE-Bench Pro environment, using the same 44-instance sample as the original environment. Bars follow the order none , low , medium , high , and xhigh / max , leaving unsupported settings empty, and are split into input, hidden reasoning, and visible output charges. Bars taller than the axis are clipped and labelled.
L→L
L→H
H→H
Model
Success
Cost ($)
Success
Cost ($)
Success
Cost ($)
GPT-5.6-luna
2.8
0.029
57.1
0.308
70.6
0.425
GPT-5.6-terra
34.7
0.608
64.6
1.692
72.0
2.379
GPT-5.6-sol
59.6
1.688
76.8
3.796
79.9
6.044
Claude Sonnet 5
43.8
3.806
59.0
10.994
64.2
17.607
GLM-5.2
52.7
4.781
57.5
5.497
61.5
6.136
Appendix
Table 6: Retry control study on DeepSWE v1.1, averaged equally over 113 tasks per model. Each policy makes one initial attempt and at most one fresh retry after failure. Success is reported in percent; cost is mean API dollars per task, including retries only when needed. L/H denote low / xhigh , except for GLM-5.2 ( high / max ).
Figure 12: Self-routing and naïve laddering compared with fixed efforts, random allocation, and the oracle on Terminal-Bench 2.1 for GLM-5.2, Inkling, and Kimi K3. The layout and policies match Figure 5 . Naïve laddering starts at low for Inkling and Kimi K3 and at high for GLM-5.2, retrying verified failures at max . Lower cost and higher success are preferable.
(a) Prompt template
You are about to be given one ⟨ unit ⟩ to solve on your own. Setting: ⟨ setting ⟩ Before you start, choose the reasoning effort you will run at. Your options, from least to most thinking: ⟨ option 1 ⟩ — ⟨ description 1 ⟩ … ⟨ option k⟩ — ⟨ description k⟩ Thinking longer costs more money and more time, and on a hard task it is often what makes the difference between solving it and not. Thinking longer on an easy task buys nothing. Choose the LOWEST level you believe will still let you solve this particular ⟨ unit ⟩ . Do not attempt the task now. Do not explain your choice. --- BEGIN ⟨ UNIT ⟩ --- ⟨ task description ⟩ --- END ⟨ UNIT ⟩ --- Reply with exactly one line, nothing else: EFFORT: <one of: ⟨ option 1 ⟩ , …, ⟨ option k⟩ >
Appendix
Table 7: Prompt used for self-routing. Each selection sends the template in (a) as a single user message, with the fields in angle brackets filled in and the unit name written in capitals in the task delimiters. The task description is the text the model would receive at the start of the run. The selection call runs at low effort, or at high for GLM-5.2, which has no low setting. The chosen level is read from the EFFORT: line, or otherwise from the last option named in the response.
Figure 13: Rollouts passed on each DeepSWE task at every reasoning-effort setting. Each row is an effort setting and each column is a task; columns are sorted separately for each model by total rollouts passed, so a column position does not identify the same task across models. A task solved at a lower setting but not at a higher one appears as a shaded cell above an unshaded cell in the same column. Orange dots mark the eight cells with fewer than four successful rollouts in the official DeepSWE release.
Figure 14: API cost of one DeepSWE task the model solves at every plotted effort level, relative to the cost at low (dashed line; GLM-5.2 is anchored on high ). A task’s cost at a level is the mean over all its passing rollouts. The footer gives the dollar value of 1× . Bars taller than the axis are clipped and labelled.
Figure 15: Full cost of one DeepSWE task the model solves at every plotted effort level, relative to the full cost at low .
Inference-time compute allocation decides where, when, and how much computation models spend during prediction to improve quality under resource limits.
Large Reasoning Models (LRMs) generate chain-of-thought traces whose length tracks human reaction times across cognitive tasks, but recent debate questions whether this alignment reflects genuine computational structure or surface verbosity. We test whether the alignment varies with inference-time reasoning effort. Across GPT-OSS-20B and GPT-OSS-120B, three effort levels, and six reasoning tasks, within-task and cross-task alignment remain invariant: Bayes Factors lean toward the null, and mean alignment is numerically near-identical across conditions. A manipulation check reveals that the effort parameter sets an upper budget on generation rather than driving real-time allocation, suggesting that the allocation policy is crystallized at training time. Arithmetic complexity contrasts further show that token allocation tracks fine-grained, format-dependent human difficulty patterns, with model scale improving the match. Cognitive cost alignment between LRMs and humans appears to be a training-time achievement, robust to inference-time perturbations, supporting a compiled rather than online account of LRM problem-solving.
Yueqing Hu, Tianhong Wang
Institute of Neuroscience, Chinese Academy of Sciences, Shanghai, China · School of Philosophy, Anhui University, Hefei, China
Increasing the thinking budget of AI models can significantly improve accuracy, but not all questions warrant the same amount of reasoning. Users may prefer to allocate different amounts of reasoning effort depending on how they value output quality versus latency and cost. To leverage this tradeoff effectively, users need fine-grained control over the amount of thinking used for a particular query, but few approaches enable such control. Existing methods require users to specify the absolute number of desired tokens, but this requires knowing the difficulty of the problem beforehand to appropriately set the token budget for a query. To address these issues, we propose Adaptive Effort Control, a self-adaptive reinforcement learning method that trains models to use a user-specified fraction of tokens relative to the current average chain-of-thought length for each query. This approach eliminates dataset- and phase-specific hyperparameter tuning. Users can dynamically adjust the cost-accuracy trade-off through a continuous effort parameter specified at inference time. We observe that the model automatically learns to allocate resources proportionally to the task difficulty. Across model scales ranging from 1.5B to 32B parameters, our approach reduces chain-of-thought length on math tasks by 1.9-3.7x at matched accuracy relative to the base model used for RL training.
Michael Kleinman, Matthew Trager, Alessandro Achille +2
Real-effort tasks, in which participants perform cognitively costly activities whose outcomes depend on actual performance, are widely used in experimental economics. Their validity, however, rests on the assumption that a human performs them. We study whether this assumption still holds in the era of Artificial Intelligence (AI) and Large Language Models (LLMs). Using 8 canonical real-effort tasks and 23 LLMs from three major providers, we show that most tasks can now be solved accurately and at a negligible cost, while only a few resist automation. Performance improves with each model generation, and midtier models are rapidly closing the gap with frontier ones, broadening the set of widely accessible models that can automate these tasks. Additionally, we show that verbally offering monetary incentives has no effect on LLM performance. Our findings establish a boundary condition for the use of real-effort tasks in unsupervised settings: when participants can cheaply outsource task completion to an LLM, observed performance may no longer reflect genuine human effort.
Federico Belotti, Stefano Coniglio, Antonio Cosma +1