What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute

Zhixu Du; Weijia Han; Hai Helen Li; Yiran Chen

cs.AIOct 1, 2026

What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute

Authors: Zhixu Du, Weijia Han, Hai Helen Li, Yiran Chen

Organizations: Duke University · University of Washington

Abstract

Large language models spend the same amount of computation on every token they generate, regardless of how difficult each token is to produce. Methods such as speculative decoding and model routing are built on the premise that much of this computation is unnecessary, yet the computation an individual token actually requires has not been measured. We measure it through a Mixture-of-Agents (MoA) lens: a panel of fifteen language models of increasing capacity, drawn from three families, in which every agent attempts to reproduce a reference sequence token by token, conditioned on the correct preceding tokens. We define the inference cost of the smallest agent that succeeds as the token's sufficient compute, which upper-bounds what the token requires. On three core benchmarks, a 0.5B agent reproduces 92--95% of reference tokens. Across Qwen, OLMo, and R1-distilled panels, the most expensive 10% account for 64--80% of estimated FLOPs. On all 500 MATH-500 problems, the MoA-derived map helps model routing reduce projected latency from 7.59 to 5.12 seconds while slightly improving accuracy, relative to the best confidence-routing baseline. The MoA-map helps drafting use 32.6% fewer draft tokens and approximately 20% lower projected latency than fixed-window drafting at similar accuracy. These comparisons reveal remaining allocation headroom, motivating controllers that exploit sufficient-compute structure.

Figures & tables

Figure 1: Measuring which model suffices at each token. (a) Each column is a token from a GSM8K response. Every model receives the same reference prefix; green cells reproduce the next token and outlines mark the smallest matching model. (b) Within the Qwen2.5 family, the 0.5B model reproduces 91.9–95.2% of tokens. The “unresolved” means that none matches.

Figure 2: Minimum-cost policy selection. At the illustrated prefix, the 3B, 7B, and 32B agents all reproduce the reference token “for” and reach the same next state. Their remaining cost

J_{t+1}

is therefore identical, so the controller selects the agent with the lowest immediate prediction FLOPs: 3B in this example. Repeating this selection gives the model sequence shown along the bottom.

Figure 3: Token reproduction and estimated FLOP concentration across Qwen2.5, OLMo-2, and DeepSeek-R1-Distill-Qwen. Top : distributions of the least-cost matching model for GSM8K, MATH-500, and HumanEval. Labels on the bar give the smallest-model share. Hatched gray segments denote unresolved tokens (Unres.), which might require larger models to reproduce. Bottom : cumulative estimated FLOPs over all reference tokens, ordered from most to least expensive, with a log horizontal axis. Resolved tokens retain their measured architecture-derived FLOP costs; each unresolved token is assigned twice the prediction FLOPs of the largest model in its panel. Vertical dashed lines and markers identify the top 1% and 10% of all tokens.

Model families	Qwen references	Llama references
Qwen2.5 / OLMo-2	$0.322\pm 0.003$	$0.417\pm 0.003$
Qwen2.5 / Falcon3	$0.380\pm 0.003$	$0.475\pm 0.003$
OLMo-2 / Falcon3	$0.398\pm 0.003$	$0.447\pm 0.003$

Table 1: Model families share a reproduction ordering. Partial Spearman correlations of aligned word-level reproduction indices, pooled across GSM8K, MATH-500, and HumanEval, controlling for generator surprisal, probe surprisal, and position. Columns distinguish reference generators. Values are estimates

\pm

one item-bootstrap SD. A 0 indicates no rank association.

Figure 4: Accuracy drops with the smaller model continuation. Differences from a 72B continuation of the same verified prefix. (a) : a small model generates up to

L

tokens, then 72B resumes. Shading shows

\pm

one item-bootstrap SD. (b) : full-suffix generation from reference character fraction

\tau

Figure 5: Accuracy versus latency on MATH-500. (a) Confidence routing selects 0.5B when its maximum next-token probability is at least

\theta

. MoA-map routing uses the sufficient-compute map. (b) Draft lengths follow a fixed window, model confidence, or consecutive sufficient-compute matches. Gray squares show the 72B-only baseline. Vertical bars show one item-bootstrap standard deviation.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Attempted	Qwen verified	Llama verified	Intersection
GSM8K	1,319	1,281	1,288	1,275
MATH-500	500	375	363	347
HumanEval	164	144	146	136
LiveCodeBench subset	219	108	105	84
NQ-Open	3,610	1,090	1,355	862
MBPP+	378	310	294	282

Appendix

Table A1 : Benchmark coverage before and after verification. The first three rows define the core. Counts refer to items.

Role	Model family	Sizes (B)
Measurement panel	Qwen2.5-Instruct	0.5, 1.5, 3, 7, 14, 32, 72
Measurement panel	OLMo-2-Instruct	1, 7, 13, 32
Reasoning-distilled panel	R1-Distill-Qwen	1.5, 7, 14, 32
Cross-family replication	Falcon3-Instruct	1, 3, 7, 10
Probability probe	Llama-3.1-Instruct	8
Reference generator	Qwen2.5-Instruct	72

Appendix

Table A2 : Model roles and sizes. Measurement panels supply candidate next-token policies; the Falcon panel tests cross-family agreement. Reference generators supply candidate responses for task verification; the additional MoE generator tests sensitivity to the reference source. The probability probe supplies surprisal controls. Sizes are nominal billions of parameters; the final row distinguishes total and active parameters. Qwen-72B serves both as a panel member and a generator.

Figure 6: Cross-family agreement across benchmark subsets. Partial Spearman correlation of Qwen/OLMo word indices after controlling generator surprisal, probe surprisal, and position; bars show

\pm

one item-bootstrap SD. Zero denotes no remaining rank association, and larger positive values indicate stronger shared ordering. Pooling uses the three core datasets. The right column gives each generator’s verified item count (Qwen/Llama), including six/nine AIME items; Table A1 specifies the evaluated populations.

Dataset	Qwen text	Llama text
GSM8K	$0.321\pm 0.004$	$0.385\pm 0.003$
MATH-500	$0.327\pm 0.006$	$0.462\pm 0.005$
HumanEval	$0.440\pm 0.021$	$0.474\pm 0.017$
LiveCodeBench subset	$0.383\pm 0.014$	$0.381\pm 0.013$
NQ-Open	$0.439\pm 0.022$	$0.555\pm 0.016$
MBPP+	$0.286\pm 0.018$	$0.394\pm 0.015$

Appendix

Table A3 : Per-dataset Qwen/OLMo partial correlations with bootstrap SDs (standard errors).

Variant	Qwen text	Llama text
Primary controls	$0.322\pm 0.003$	$0.417\pm 0.003$
+ word length and token counts	$0.321\pm 0.003$	$0.416\pm 0.003$
+ frequency and approximate entropy	$0.299\pm 0.003$	$0.406\pm 0.003$
+ both families’ loss gaps	$0.276\pm 0.003$	$0.409\pm 0.003$
+ smallest-model surprisals	$0.322\pm 0.003$	$0.409\pm 0.003$
High-confidence reference words	$0.287\pm 0.004$	$0.400\pm 0.003$

Appendix

Table A4 : Sensitivity of cross-family agreement, with bootstrap SDs (standard errors). The frequency/entropy and loss-gap rows extend the primary controls successively. Other rows specify separate controls or populations.

Text	Panel	GSM8K	MATH-500	HumanEval
Qwen	Qwen	0.547 (0.76%)	0.451 (0.49%)	0.514 (0.14%)
Qwen	Olmo	0.330 (2.80%)	0.365 (2.99%)	0.316 (2.08%)
Llama	Qwen	0.543 (4.15%)	0.547 (4.76%)	0.243 (0.73%)
Llama	Olmo	0.385 (5.29%)	0.467 (8.72%)	0.218 (1.42%)

Appendix

Table A5 : Gini coefficients of architecture-derived prediction FLOPs among resolved tokens; parentheses give unresolved-token percentages among all tokens. These values exclude the imputed costs used in Figure 3 .

Panel	Dataset	Unresolved	Resolved only	All tokens
Qwen	GSM8K	0.76	41.76 / 60.11	56.86 / 79.54
Qwen	MATH-500	0.49	35.84 / 50.85	60.14 / 71.98
Qwen	HumanEval	0.14	45.18 / 56.72	52.94 / 63.66
OLMo	GSM8K	2.80	14.09 / 41.09	17.22 / 68.57
OLMo	MATH-500	2.99	15.15 / 44.39	16.25 / 69.37
OLMo	HumanEval	2.08	14.84 / 39.55	19.87 / 63.75

Appendix

Table A6 : Concentration on Qwen-generated verified references under two cost conventions. Unresolved is the percentage of all native tokens with no matching panel model. Resolved-only columns give FLOP shares for the top 1% / top 10% of resolved tokens. All-token columns give the corresponding shares among all tokens after assigning each unresolved position twice the largest model’s prediction FLOPs. All entries are percentages.

Task	Words	Qwen panel	OLMo panel
copy	24,770	$0.061\pm 0.008$	$0.062\pm 0.008$
continue	7,755	$0.277\pm 0.049$	$0.267\pm 0.048$
summarize	3,840	$0.283\pm 0.014$	$0.294\pm 0.013$
repeat	3,864	$0.130\pm 0.005$	$0.120\pm 0.004$
add	5,121	$0.289\pm 0.013$	$0.291\pm 0.013$
product	7,533	$0.405\pm 0.011$	$0.389\pm 0.012$

Appendix

Table A7 : Agreement with FlexiDepth layer allocations, with bootstrap SDs. Each split has 100 responses; the first word is excluded.

Maximum tokens	0.5B	1.5B	7B
1	$-0.09\pm 0.19$	$-0.30\pm 0.16$	$-0.06\pm 0.15$
2	$-0.17\pm 0.22$	$-0.27\pm 0.18$	$-0.04\pm 0.16$
4	$-0.30\pm 0.24$	$-0.17\pm 0.19$	$-0.00\pm 0.18$
8	$-0.86\pm 0.26$	$-0.71\pm 0.21$	$-0.18\pm 0.19$
16	$-2.19\pm 0.30$	$-1.37\pm 0.26$	$-0.22\pm 0.21$
32	$-4.53\pm 0.39$	$-2.43\pm 0.30$	$-0.55\pm 0.23$

Appendix

Table A8 : Handover-length accuracy differences (percentage points), with bootstrap SDs (standard errors).

Dataset	$\tau$	0.5B	1.5B	7B	32B
GSM8K	0.0	-52.88	-23.39	-4.51	-0.16
GSM8K	0.25	-48.25	-21.17	-4.12	-0.50
GSM8K	0.5	-35.08	-13.92	-3.17	-0.74
GSM8K	0.75	-20.60	-7.15	-1.77	-0.34
GSM8K	0.9	-10.62	-3.07	-0.72	-0.08
MATH-500	0.0	-55.76	-28.85	-5.08	+1.17

Appendix

Table A9 : Full-suffix handover accuracy differences (percentage points). Prefix fractions use characters, snapped to a word boundary.

Policy	Accuracy (%)	Projected time (s)	Compute (TFLOPs)
0.5B only	$28.20\pm 2.04$	$1.84\pm 0.04$	$0.67\pm 0.01$
72B only	$69.20\pm 2.11$	$19.03\pm 0.38$	$101.92\pm 1.87$
Confidence routing, $\theta=0.5$	$40.00\pm 2.25$	$2.80\pm 0.06$	$86.57\pm 1.90$
Confidence routing, $\theta=0.8$	$60.00\pm 2.21$	$4.45\pm 0.10$	$99.50\pm 1.85$
Confidence routing, $\theta=0.95$	$66.60\pm 2.21$	$5.91\pm 0.13$	$99.16\pm 1.84$
Confidence routing, $\theta=0.99$	$68.40\pm 2.14$	$7.59\pm 0.18$	$100.38\pm 1.89$

Appendix

Table A10 : All ten allocation policies on the full MATH-500 evaluation set (500 problems). Accuracy is independently checked for each generated response. Projected time includes prompt processing and cached model calls under Appendix E.2 ; it excludes oracle-map acquisition and is not measured deployment latency. Values are means

\pm

one item-bootstrap SD. Confidence thresholds control small-model selection; adaptive drafting stops below probability 0.4 with a 16-token cap. MoA-map policies use advance reference-token agreement.

Baseline	$\Delta$ accuracy (pp)	$\Delta$ projected (s)
Routing, $\theta=0.5$	$29.60\pm 2.16$	$2.32\pm 0.24$
Routing, $\theta=0.8$	$9.60\pm 1.60$	$0.67\pm 0.23$
Routing, $\theta=0.95$	$3.00\pm 1.04$	$-0.79\pm 0.22$
Routing, $\theta=0.99$	$1.20\pm 0.62$	$-2.46\pm 0.21$
Fixed drafting	$0.40\pm 0.29$	$-1.00\pm 0.04$
Adaptive drafting	$0.40\pm 0.29$	$-0.71\pm 0.03$

Appendix

Table A11 : Paired differences between each MoA-map policy and the indicated baseline on the same 500 problems (map minus baseline). Positive accuracy differences favor the map; negative time differences favor the map. Accuracy is in percentage points (pp), and time differences are in seconds. Values show the mean difference

\pm

one item-bootstrap SD; projected time follows Appendix E.2 .

LLM Inference Efficiency

Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.

See all 352 papers

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
Apr 24, 2026Longju Bai, Zhemin Huang, Xingyao Wang +5Coding Agents Cost-Aware Inference
University of Michigan · Stanford University · Microsoft AI +3
TokenCast: Forecasting Token Consumption During LLM Agent Execution
Sep 28, 2026Chaoqian Ouyang, Ling Yue, Libin Zheng +7Cost-Aware Inference Efficient Language Model Inference
Sun Yat-Sen University · Rensselaer Polytechnic Institute · Southeast University +1
Thinking Costs Tokens: When More Structure is Worth the Price
Aug 27, 2026Thomas Nolasque, John Grey, Calista Pham +1Efficient Language Model Reasoning Test-Time Scaling
Royal Bank of Canada

Apr 24, 2026J/K move · Enter open · S save

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks

Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun+4

University of Michigan · Stanford University · Microsoft AI+3

Coding Agents Cost-Aware Inference

Open paper AskPDF arXiv

The wide adoption of AI agents in complex human workflows is driving rapid growth in LLM token consumption. When agents are deployed on tasks that require a significant amount of tokens, three questions naturally arise: (1) Where do AI agents spend the tokens? (2) Which models are more token-efficient? and (3) Can agents predict their token usage before task execution? In this paper, we present the first systematic study of token consumption patterns in agentic coding tasks. We analyze trajectories from eight frontier LLMs on SWE-bench Verified and evaluate models' ability to predict their own token costs before task execution. We find that: (1) agentic tasks are uniquely expensive, consuming 1000x more tokens than code reasoning and code chat, with input tokens rather than output tokens driving the overall cost; (2) token usage is highly variable and inherently stochastic: runs on the same task can differ by up to 30x in total tokens, and higher token usage does not translate into higher accuracy; instead, accuracy often peaks at intermediate cost and saturates at higher costs; (3) models vary substantially in token efficiency: on the same tasks, Kimi-K2 and Claude-Sonnet-4.5, on average, consume over 1.5 million more tokens than GPT-5; (4) task difficulty rated by human experts only weakly aligns with actual token costs, revealing a fundamental gap between human-perceived complexity and the computational effort agents actually expend; and (5) frontier models fail to accurately predict their own token usage (with weak-to-moderate correlations, up to 0.39) and systematically underestimate real token costs. Our study offers new insights into the economics of AI agents and can inspire future research in this direction.

Abstract

LLM Inference Efficiency

Explore similar work

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks

TokenCast: Forecasting Token Consumption During LLM Agent Execution

Thinking Costs Tokens: When More Structure is Worth the Price