Developers and consumers increasingly choose reasoning models (RMs) based on their listed API prices. However, how accurately do these prices reflect actual inference costs? We conduct the first systematic study of this question, evaluating 8 frontier RMs across 12 diverse tasks covering competition math, science QA, code generation, and multi-domain agents. We uncover the pricing reversal phenomenon: in 32% of model-pair comparisons, the model with a lower listed price actually incurs a higher total cost, with reversal magnitude reaching up to 28x. For example, Gemini 3 Flash's listed price is 80% cheaper than GPT-5.4's, yet its actual cost across all tasks is 38% higher. We build a formal cost attribution framework based on Shapley value, and leverage it to trace the dominating contributors to vast heterogeneity in thinking token consumption and number of interaction turns: on the same query, one model may use 900% more thinking tokens than another, or 10x more turns of environment interactions. We further show that per-query cost prediction is fundamentally difficult: repeated runs of the same query yield thinking token variation up to 9.7x, establishing an irreducible noise floor for any predictor. Thus, we propose cost distribution prediction as an open challenge. Our findings demonstrate that listed API pricing is an unreliable proxy for actual cost, calling for cost-aware model selection and transparent per-request cost monitoring.
Figures & tables
Figure 1: The phenomenon of mismatch between RM pricing and their actual costs. The actual cost is aggregated over all 12 tasks, including AIME, MMLUPro, Human Last Exam, SimpleQA, GAIA, and Terminal-Bench 2.0. Overall, models with lower listed prices may incur much higher expenses than those with higher prices. For example, Gemini 3 Flash’s list price (3.5/1milliontokens)is8017.5), but its actual cost (705)isactually38509). This dramatically changes the cost ranking and poses a pressing challenge to cost-sensitive users. For example, one might choose Gemini 3 Flash over Kimi K2.6 due to its lower price, but recognize later that it is much more expensive.
Figure 2: The pricing reversal phenomenon. (a) Across 8 models evaluated on 12 diverse tasks, we observe that the listed price rankings systematically mismatch the actual costs. On average, 32% of model pairs exhibit pricing reversal, i.e., a model with a lower listed price ends up with a higher cost. The actual cost rankings vary substantially across tasks, too. (b) For example, on MMLU-pro, Gemini 3.1 Flash’s actual cost is the highest, though it is ranked 7th by listed price. (c) However, Kimi K2.6 becomes the most expensive one on Cybench despite its low listed price. This suggests that naively assessing cost by a fixed API pricing Table is thus not reliable.
Symbol
Description
Symbol
Description
T
Total turns
nˉthink
Avg. thinking tokens/turn
nˉin
Avg. fresh input tokens/turn
pin
Price of fresh input tokens
nˉout
Avg. output tokens/turn
pout
Price of output (incl. thinking)
nˉcache
Avg. cached tokens/turn
pcache
Price of cached tokens
Table 1: The eight cost factors.
Figure 3: Cost contributions assigned by our cost attribution framework. Overall, the number of thinking tokens is the dominating factor for single-turn tasks. More than 95% of the cost difference among reversed model pairs is attributed to thinking tokens. For multi-turn tasks, on the other hand, both the number of turns and the cached input tokens contribute substantially. This suggests that the historical context difference is a unique factor in agentic systems.
Figure 4: Cost attribution breakdown by task difficulty. On easy tasks, the cost gap is often small, as most models do not need too much effort. On hard questions, the gap becomes substantial, especially for multi-turn tasks. In particular, the impact of cached tokens also becomes more visible.
Figure 5: Per-query cost distributions for a fixed (model, query) pair on a signle-turn task (AIME) and a multi-turn task (Terminal-Bench 2.0). Even after fixing both, repeated rollouts span up to 9.7× between the cheapest and most expensive run. This suggests that the cost itself is a big
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Category
# tasks
Description
Single-turn workloads
AIME Art of Problem Solving [2025]
Math
60
Competition math (AMC/AIME)
ARC-AGI Chollet [2019]
Reasoning
400
Visual / abstract reasoning
ArenaHard Li et al. [2024]
Chat
750
Open-ended hard prompts
GPQA Rein et al. [2023]
Science
198
Graduate-level QA (diamond)
HLE Phan et al. [2025]
Mixed
500
Humanity’s Last Exam (subset of 2,056 )
Appendix
Table 2: Details of evaluated datasets.
Model
Provider
API model code
pi
po
pcache
Lm
GPT-5.4
OpenAI
gpt-5.4
2.50
15.00
0.25
17.50
GPT-5.4 Mini
OpenAI
gpt-5.4-mini
0.75
4.50
0.075
5.25
Gemini 3.1 Pro
Google
gemini-3.1-pro-preview
2.00
12.00
0.20
14.00
Gemini 3 Flash
Google
gemini-3-flash-preview
0.50
3.00
0.05
3.50
Claude Opus 4.7
Anthropic
claude-opus-4-7
5.00
25.00
0.50
30.00
Claude Haiku 4.5
Anthropic
claude-haiku-4-5
1.00
5.00
0.10
6.00
Appendix
Table 3: API model identifiers and pricing (USD per MTok), recorded May 1, 2026 .
Model
T
top- p
max_tokens
Reasoning configuration
GPT-5.4
default
default
default
reasoning_effort="high"
GPT-5.4 Mini
default
default
default
reasoning_effort="high"
Gemini 3.1 Pro
default
default
default
built-in adaptive thinking
Gemini 3 Flash
default
default
default
built-in adaptive thinking
Claude Opus 4.7
default
default
32,000
adaptive thinking
Claude Haiku 4.5
default
default
default
extended thinking
Appendix
Table 4: Decoding and reasoning configuration. We use default decoding parameters. For OpenAI models, we set a high reasoning effort since these tasks are often considered challenging. Single-turn requests use a 1,800 s per-call timeout; multi-turn requests use a 5-hour timeout per task.
Figure 6: Listed-price vs. actual-cost ranking on all 12 benchmarks ( 9 single-turn +3 multi-turn). Crossings between left- and right-column ranks are reversed model pairs.
Model
AIME
ARC
Arena
GPQA
HLE
LCB
LMB
MMLU
SQA
GPT-5.4
389
4,596
1,090
585
17,309
3,284
289
1,804
4,722
GPT-5.4 Mini
272
3,617
1,140
482
8,766
2,097
284
2,964
3,530
Gemini 3.1 Pro
646
5,340
2,061
1,147
35,735
8,231
586
8,007
30,122
Gemini 3 Flash
994
12,579
2,087
3,619
57,797
53,338
1,754
71,489
5,024
Claude Opus 4.7
321
2,797
1,019
614
13,599
2,405
279
1,827
1,336
Claude Haiku 4.5
1,074
7,267
2,146
1,751
6,047
9,023
1,086
9,301
121
Appendix
Table 5: Total thinking tokens (in thousands) by (model, single-turn dataset). Abbreviations: ARC = ARC-AGI, Arena = ArenaHard, HLE=Human Last Exam, LCB = LiveCodeBench, LMB = LiveMathBench, MMLU = MMLU-Pro, SQA = SimpleQA.
Figure 7: Per-dataset Shapley breakdown across all 12 benchmarks. The dominant cost-driving factor varies systematically with workload.
Figure 8: Per-query cost distribution by (model, dataset). Long-tailed in every cell.
Dataset
MEAN
Q1
Q10
Q20
Q40
Q50
Q70
Q90
Q99
Single-turn
AIME
32.1
50.0
53.6
57.1
53.6
50.0
42.9
42.9
17.9
ARC-AGI
14.3
50.0
42.9
42.9
25.0
21.4
17.9
17.9
17.9
ArenaHard
10.7
28.6
17.9
17.9
17.9
14.3
10.7
10.7
17.9
GPQA
35.7
50.0
53.6
50.0
60.7
60.7
53.6
25.0
17.9
HLE
28.6
32.1
42.9
50.0
46.4
42.9
28.6
21.4
17.9
Appendix
Table 6: Per-dataset listed-price reversal rate (%) by cost statistic. Last row is the pooled (unweighted) mean across the 12 datasets, matching Fig. 9 of the main text.
Figure 9: Listed-price reversal rate vs. which cost statistic the user cares about, pooled over 12×28=336 ordered model pairs. (a) Global pooled across all 12 datasets: mean reversal is 31.5% , but the body Q15 – Q40 rises to a 39 – 40% plateau (peak Q40 : 39.6% ); the high tail Q99 is the lowest at 25.0% , so the mean tracks the high tail rather than the body. (b–d) Local workload regimes underlying the pooled curve: reasoning (single-turn: AIME, GPQA, HLE, LiveMathBench, LiveCodeBench) is a clean inverted-U; short-answer / chat (single-turn: ARC-AGI, ArenaHard, MMLU-Pro, SimpleQA) is roughly flat near 30% ; agentic (multi-turn: TerminalBench 2, Cybench, GAIA) is bimodal with a body peak at Q40 – Q85 . The mean is a lossy summary in every regime.
API buyers purchase a dated contract, not a model name alone: the contract includes the requested and served model, reasoning-effort term or its omission, output rail, service product, prompt, and price schedule. We study the reasoning-effort term through a registered paired contrast of Sonnet 5 with explicit high effort against the same model with effort omitted, using 30 AIME 2026 items and five calls per item. Every paid attempt was assigned one frozen terminal category, and inference resampled items while retaining their repeated calls. Mean delivered cost was $0.01031 per call higher under the explicit-high contract than under the omitted contract [+$0.00204, +$0.01974]. The corresponding accuracy contrast was +0.0133 [-0.0267, +0.0467]; we did not detect an accuracy difference, and the interval permits a gain of up to 4.67 percentage points that this design cannot rule out. Cost per correct answer was $0.08665 under the high-effort contract and $0.07662 under the omitted contract, as registered point estimates. A dated contract census, Models-API metadata, and preregistered raw-response probes further documented model-specific omission semantics, including within a provider; claims remained at documentation grade when raw structure was indeterminate. The request registry, parser, terminal taxonomy, statistical plan, and analysis pipeline were frozen before outcomes were examined; the resulting claims are bounded to the model, task, and collection date studied.
Yeabin Moon
Brandeis School of Business and Economics Brandeis University
Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on these platforms typically use a fixed price per token, preventing users from achieving the most competitive price for their tasks. In this work, we design a procurement platform where token prices for each task are driven by provider competition, enabling users to secure competitive pricing for guaranteed quality levels. To this end, the platform sequentially routes queries via a reverse second-price auction that incentivizes model providers to truthfully bid their best estimate of the average cost to serve a user's query. As it routes queries, the platform learns the quality offered by each provider and progressively routes queries to the most cost-competitive provider among those meeting a desired quality threshold. To validate our design, we conduct experiments with multiple LLMs from the Llama and Qwen families on popular mathematical reasoning and question-answering benchmarks. The results show that the pricing margin of the most cost-competitive provider on our platform varies significantly---from 10% to 71%---depending on the task and quality threshold. This suggests a substantial inefficiency in the current fixed-price market, and it demonstrates that our platform may enable users to capture maximum savings whenever competitive market conditions permit.
Dimitrios Rontogiannis, Ander Artola Velasco, Manuel Gomez Rodriguez
Max Planck Institute for Software Systems Kaiserslautern, Germany
Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control token. A Collaborate Engine then sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Across five mathematical reasoning benchmarks, PyroDash supports different accuracy-cost operating points. With λ=0.05, it achieves 64.04 percent average accuracy, 6.36 percentage points above the LLM-only baseline, while reducing cost by 20.4 percent. With λ=0.6, it achieves 54.55 percent accuracy with a 1.90 percent LLM token ratio and 0.012 LLM calls per example, reducing total cost from USD 49.36 to USD 1.78. These results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.