cs.AIOct 25, 2026
SaveWhat Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents
Organizations: Johns Hopkins University · Together AI · Cornell University
Abstract
Generative models increasingly expose a reasoning effort setting that lets users scale inference-time compute, trading higher cost for potentially improved performance. Users must choose an effort level before knowing whether extra reasoning will help, and even a successful run does not reveal whether the effort purchased was necessary. Reasoning effort is therefore a credence good: buyers observe the outcome but cannot assess the marginal value of the computation they paid for. In this work, we measure the counterfactual effect of effort levels on both proprietary and open-weight models on four benchmarks, spanning single-call reasoning and long-horizon agentic tasks. Effort, it turns out, does not consistently preserve tasks that lower effort solves, can overspend, and can change trajectory behavior. First, we find that although higher effort can raise accuracy in aggregate, it frequently fails on tasks that lower effort solved. Second, on the subset of easy tasks that every effort level solves, higher effort is generally more costly, with the highest effort level costing up to 42x as much as the lowest. Lastly, effort changes not only how many tokens are used, but also how agents solve tasks: at higher effort, agents verify their own work more heavily and more often search for solutions than work through problems directly, both of which lengthen trajectories without necessarily altering the outcome. These findings show that naively choosing the highest level is not only cost-inefficient but also offers no guarantee of the best instance-level performance. A simple alternative, starting at the lowest effort and escalating only after a verified failure, matches or improves on fixed top effort across all models and benchmarks at lower cost for most proprietary models.
Figures & tables
Figure 1: Instances gained and lost between adjacent reasoning-effort settings on the agentic benchmarks. Higher effort can solve additional instances while losing some lower-effort successes. AIME, GPQA, and a corrected version of SWE-Bench Pro are included in Appendix Figure 7 .
Figure 2: API cost of one task the model solves at every effort level, including none if supported, relative to the cost at low (dashed line; GLM-5.2 is anchored on high , its lowest reasoning level). Kimi K3 has no none setting. Each bar is split into input, hidden reasoning, and visible output tokens. The always-solved set is fixed per model and benchmark, so bars compare prices within a group and only vary across groups; the footer gives the dollar value of . Bars taller than the axis are clipped and labelled.
Figure 3: GPT-5.6-sol on matched SWE-Bench Pro instances across reasoning effort levels. Left: reference-fix reads from official git history and online lookup attempts in the corrected environment, regardless of retrieval success. Right: instances resolved. Corrected removes the original git history and restricts network access to allowlisted package registries.
Figure 4: Changes in the mean per-trajectory share of annotated prose assigned to each Act·ONOMY behavior group, in percentage points, from the lowest to the highest annotated reasoning-enabled setting on matched tasks. A dagger ( ) marks a change whose unadjusted 90% paired task-bootstrap confidence interval excludes zero (10,000 resamples); markers use unrounded estimates.
Figure 5: Self-routing and naïve laddering compared with fixed efforts, random allocation, and the oracle on Terminal-Bench 2.1; Appendix I shows the other models with saved self-routing comparisons. Each panel shows success against total API cost, normalized by the cost of that model’s fixed effort with the highest performance (dashed line, effort level noted in gray text). Naïve laddering (green triangles) starts at the lowest available reasoning effort and retries at the highest effort after an externally verified failure, charging both executions when it retries. All policies in each panel use the same task cohort. Lower cost and higher success are preferable.
Figure 6: GPT-5.6-terra on Terminal-Bench 2.1 at fixed high , fixed xhigh , and naïve laddering (green). Bars show mean cost per task (left axis) as the API bill and as full cost (Equation 1 ), labelled with their ratio to xhigh ; dots show tasks solved (right axis) with 95% Wilson intervals.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Evaluation size | Execution setting |
|---|---|---|
| AIME 2026 Dekoninck et al. (2026) | 30 problems, 10 attempts each | Single API call per attempt |
| GPQA Diamond Rein et al. (2023) | 198 questions, 4 attempts each | Single API call per attempt |
| Terminal-Bench 2.1 Merrill et al. (2026) | 89 tasks | Multi-turn terminal agent using Terminus-2 |
| SWE-Bench Pro Deng et al. (2025) | 44-instance pilot | Multi-turn repository-level agent using mini-swe-agent |
Table 1: Benchmarks and execution settings. AIME uses ten attempts per problem and GPQA uses four attempts per question; the agentic benchmarks evaluate multi-turn executions with fixed harnesses.
| Model | API | Effort settings |
|---|---|---|
| GPT-5.6-sol, GPT-5.6-terra, GPT-5.6-luna | OpenAI | none , low , medium , high , xhigh |
| Claude Sonnet 5 | Anthropic | none , low , medium , high , xhigh |
| GLM-5.2 | Together AI | none , high , max |
| Inkling | Together AI | none , low , medium , high , max |
| Kimi K3 | Together AI | low , high , max |
| DeepSeek V4 Flash | Together AI | none , low , high , max |
Table 2: Serving API and evaluated reasoning-effort settings for each model.
| Model / cost category | AIME | GPQA | SWE-Pro | TB | Total (USD) |
| Benchmark inference | |||||
| GPT-5.6-sol | 72.4 | 125.9 | 1,142.7 | 165.1 | 1,506.0 |
| GPT-5.6-terra | 29.2 | 47.6 | 312.1 | 131.8 | 520.8 |
| GPT-5.6-luna | 4.9 | 10.2 | 15.4 | 19.3 | 49.8 |
| Claude Sonnet 5 | 217.2 | 230.3 | 1,045.2 | 322.3 | 1,815.0 |
| GLM-5.2 | 44.7 | 126.7 | 804.9 | 255.7 | 1,232.1 |
Table 3: Experimental cost in USD, rounded to one decimal place.
| Model | none | low | medium | high | xhigh | max |
|---|---|---|---|---|---|---|
| AIME 2026 (30 problems; 10 attempts per problem) | ||||||
| GPT-5.6-luna | 25.3 [14, 38] | 75.7 [65, 85] | 85.3 [75, 94] | 94.7 [89, 99] | 96.7 [93, 99] | – |
| GPT-5.6-terra | 7.3 [2, 14] | 89.7 [81, 96] | 91.3 [82, 98] | 92.3 [86, 98] | 97.0 [94, 99] | – |
| GPT-5.6-sol | 55.3 [42, 68] | 96.7 [90, 100] | 97.7 [95, 100] | 99.7 [99, 100] | 99.7 [99, 100] | – |
| Claude Sonnet 5 | 79.7 [68, 90] | 72.3 [60, 83] | 86.3 [78, 93] | 96.7 [92, 99] | 97.3 [92, 100] | – |
| GLM-5.2 | 46.0 [35, 57] | – | – | 86.3 [76, 95] | – | 90.7 [81, 98] |
Table 4: Overall accuracy (%) across reasoning-effort settings, with 95% confidence intervals in brackets, rounded to whole percentage points. AIME and GPQA report mean per-attempt correctness, with percentile bootstrap intervals (10,000 resamples) over problems that keep each problem’s attempts together; the agentic benchmarks report the fraction of tasks solved, with Wilson score intervals. Intervals describe each setting separately; because all settings of a model share the same tasks, paired tests between settings are more sensitive than a comparison of overlapping intervals. Bold marks the highest accuracy within each model–benchmark row, including ties; dashes indicate settings without a reported result. Effort names retain their model-specific meanings.
Figure 7: Instances gained and lost between adjacent reasoning-effort settings on AIME, GPQA, and the corrected SWE-Bench Pro environments. An instance counts as solved at a level if any of its attempts at that level is correct.
Figure 8: Self-routing and naïve laddering on Terminal-Bench 2.1 with user waiting cost added to the API bill (Equation 1 ), for the seven models with saved self-routing comparisons. Each panel shows success against full cost, normalized by the full cost of that model’s fixed effort with the highest performance (dashed line). The oracle selects the successful effort with the lowest full cost; router calls are charged their API cost only. Policies, markers, and task cohorts match Figure 5 . Lower cost and higher success are preferable.
Figure 9: Full cost of one task the model solves at every effort level, including none if supported, relative to the full cost at low (dashed line; GLM-5.2 is anchored on high ). Kimi K3 has no none setting. Full cost adds to the API bill the time a user waits for the output tokens at the model’s published throughput, priced at the BLS mean hourly wage of 1\times$ . Bars taller than the axis are clipped and labelled.
| Solved only in | |||||||
|---|---|---|---|---|---|---|---|
| Model | Effort | Original (%) | Corrected (%) | (pp) | Original | Corrected | |
| GPT-5.6-luna | xhigh | 84.1 | 56.8 | 13 | 1 | 0.002 | |
| GPT-5.6-terra | xhigh | 72.7 | 61.4 | 7 | 2 | 0.180 | |
| GPT-5.6-sol | xhigh | 77.3 | 59.1 | 8 | 0 | 0.008 | |
| Claude Sonnet 5 | xhigh | 79.5 | 50.0 | 13 | 0 | 0.001 | |
| GLM-5.2 | max | 70.5 | 43.2 | 13 | 1 | 0.002 | |
Table 5: Solve rates on the original and corrected SWE-Bench Pro environments at each model’s highest evaluated reasoning-effort setting, using the same 44 instances in both environments. Corrected removes the task repository’s original git history and restricts network access to allowlisted package registries. is corrected minus original solve rate in percentage points, computed before rounding. The Solved only in columns count discordant instances, and is the two-sided exact McNemar test on those counts. The decrease is significant at for 5 of 8 models, and a sign test over the 8 models, all of which decrease, gives . Both conditions follow the source scoring rule that evaluator-passed executions costing more than $5 count as failures. Table 4 reports results at all effort levels.
Figure 10: The summarized trajectory of GPT-5.6-sol on the SWE-Bench Pro instance openlibrary-00bec1e7 at none (left, 15 calls, 1.00); both runs resolve the task. Each row is a phase of consecutive agent steps and bar length is the number of steps in that phase. Grey marks routine work, blue a genuine test run, orange a look into git history or validation against the copied file, and red a read or copy of the gold fix commit. At none the agent fixes the validator from the issue text and the code. At xhigh it does the same work, then runs git log --all , finds the gold fix commit still present in the container’s clone, overwrites its own file with git show <gold>:<file> , runs that commit’s tests, and confirms a byte-for-byte match before submitting.
Figure 11: Mean API cost per always-solved task in the corrected SWE-Bench Pro environment, using the same 44-instance sample as the original environment. Bars follow the order none , low , medium , high , and xhigh / max , leaving unsupported settings empty, and are split into input, hidden reasoning, and visible output charges. Bars taller than the axis are clipped and labelled.
| Model | Success | Cost ($) | Success | Cost ($) | Success | Cost ($) |
|---|---|---|---|---|---|---|
| GPT-5.6-luna | 2.8 | 0.029 | 57.1 | 0.308 | 70.6 | 0.425 |
| GPT-5.6-terra | 34.7 | 0.608 | 64.6 | 1.692 | 72.0 | 2.379 |
| GPT-5.6-sol | 59.6 | 1.688 | 76.8 | 3.796 | 79.9 | 6.044 |
| Claude Sonnet 5 | 43.8 | 3.806 | 59.0 | 10.994 | 64.2 | 17.607 |
| GLM-5.2 | 52.7 | 4.781 | 57.5 | 5.497 | 61.5 | 6.136 |
Table 6: Retry control study on DeepSWE v1.1, averaged equally over 113 tasks per model. Each policy makes one initial attempt and at most one fresh retry after failure. Success is reported in percent; cost is mean API dollars per task, including retries only when needed. denote low / xhigh , except for GLM-5.2 ( high / max ).
Figure 12: Self-routing and naïve laddering compared with fixed efforts, random allocation, and the oracle on Terminal-Bench 2.1 for GLM-5.2, Inkling, and Kimi K3. The layout and policies match Figure 5 . Naïve laddering starts at low for Inkling and Kimi K3 and at high for GLM-5.2, retrying verified failures at max . Lower cost and higher success are preferable.
| (a) Prompt template |
| You are about to be given one unit to solve on your own. Setting: setting Before you start, choose the reasoning effort you will run at. Your options, from least to most thinking: option 1 — description 1 … option — description Thinking longer costs more money and more time, and on a hard task it is often what makes the difference between solving it and not. Thinking longer on an easy task buys nothing. Choose the LOWEST level you believe will still let you solve this particular unit . Do not attempt the task now. Do not explain your choice. --- BEGIN UNIT --- task description --- END UNIT --- Reply with exactly one line, nothing else: EFFORT: <one of: option 1 , …, option > |
Table 7: Prompt used for self-routing. Each selection sends the template in (a) as a single user message, with the fields in angle brackets filled in and the unit name written in capitals in the task delimiters. The task description is the text the model would receive at the start of the run. The selection call runs at low effort, or at high for GLM-5.2, which has no low setting. The chosen level is read from the EFFORT: line, or otherwise from the last option named in the response.
Figure 13: Rollouts passed on each DeepSWE task at every reasoning-effort setting. Each row is an effort setting and each column is a task; columns are sorted separately for each model by total rollouts passed, so a column position does not identify the same task across models. A task solved at a lower setting but not at a higher one appears as a shaded cell above an unshaded cell in the same column. Orange dots mark the eight cells with fewer than four successful rollouts in the official DeepSWE release.
Figure 14: API cost of one DeepSWE task the model solves at every plotted effort level, relative to the cost at low (dashed line; GLM-5.2 is anchored on high ). A task’s cost at a level is the mean over all its passing rollouts. The footer gives the dollar value of . Bars taller than the axis are clipped and labelled.
Figure 15: Full cost of one DeepSWE task the model solves at every plotted effort level, relative to the full cost at low .
More in this topic
Inference-Time Compute Allocation
Inference-time compute allocation decides where, when, and how much computation models spend during prediction to improve quality under resource limits.
See all 82 papers