Organizations: University of Illinois Urbana-Champaign · Northeastern University · Independent Researcher · University of California, Los Angeles · UT Austin
AI agents may consume massive tokens across a wide range of tasks, but how many of those tokens are really necessary? Although prior work has shown that agents can incur unexpectedly high token costs, we still lack a systematic way to identify and measure token waste. In this work, we introduce a unified framework for measuring token waste across heterogeneous agent systems. Our framework records and normalizes complete agent trajectories, identifies removable spans using an LLM-as-a-Judge (LLMaJ), and maps the identified text spans back to the original trajectory for token-level measurement. We validate the identified waste through direct interventions by removing marked spans, resuming agent execution, and measuring changes in task outcomes and subsequent actions. In online pruning experiments, we remove 41-46% of the model's reasoning while largely preserving task performance across two distinct agent benchmarks. Using this framework, we analyze agents across multiple settings. Our analysis shows that token waste has a system-level structure that aggregate token usage alone cannot reveal. First, waste varies more across tasks and agent settings than across models. Second, reasoning and visible execution can exhibit different waste patterns. Third, different harnesses can introduce distinct sources of inefficiency even under the same model and task. Overall, our framework transforms token waste into an actionable optimization signal, enabling researchers and practitioners to train more efficient models, design better reasoning strategies, and optimize agent harnesses toward reducing unnecessary computation rather than merely reducing token usage.
Figures & tables
Figure 1: Overview of the proposed framework. The framework minimally instruments agent benchmarks, routes model interactions through a unified proxy for trajectory tracing, and uses LLM-based judges to annotate agent actions and identify token waste, followed by span-level verification and token measurement.
Figure 2: Examples of the waste patterns drawn from real agent trajectories.
removed
solved (%)
Benchmark
Intervention
median
% of
original
pruned
worse /
tokens
prefix
better
SWE-bench Verified
25%
350
19.8
81.7
81.7
3 / 3
50%
649
14.2
86.0
86.0
1 / 1
75%
813
15.9
91.3
91.3
0 / 0
MCP-Atlas
1st (turn 1)
111
47.5
75.3
65.6
9 / 0
Table 1: Effect of removing LLMaJ-identified strict waste and resuming agent execution. “% of prefix” denotes the fraction of model-generated text that is removed, excluding tool observations. “Worse / better” counts units that change from solved to unsolved and vice versa. For MCP-Atlas, the trajectories are shorter, so we only intervene after the first, second, and third turns.
Benchmark
Intervention
Changed turns (%)
SWE-bench Verified
Marked waste deleted
3.0
All reasoning deleted
25.0
MCP-Atlas
Marked waste deleted
10.5
All reasoning deleted
37.9
Table 2: Effect of removing previous reasoning on the agent’s next-action distribution. Each turn is evaluated using 16 rollouts per condition and an energy-distance permutation test. “Changed turns” counts turns for which the intervention produces a significant shift in the action distribution at p<0.05 .
SWE-bench Verified
MCP-Atlas
Pruning Method
deleted (%)
solved (%)
deleted (%)
solved (%)
nothing deleted
0
89.0
0
65.0
restated and padding deleted
41
89.5
46
62.3
all reasoning deleted
100
73.0
100
56.0
Table 3: Online pruning results on SWE-bench Verified and MCP-Atlas.
Figure 3: Averaged waste ratio comparison across the evaluated models. Waste ratio is calculated by dividing the cost of wasted output by the cost of all of its audited output, and we average this waste ratio across the four benchmarks.
Figure 4: Performance-Waste Pareto shows that models with less waste can still achieve higher success rates than models with more waste.
Figure 5: Waste pattern distribution for different models. Although the distributions differ, detours dominate the waste across almost all of them.
Figure 6: Comparing waste ratio across the benchmarks, waste ratios vary much more across benchmarks than across model.
Figure 7: Waste ratio against the context length. Across benchmarks, trajectories with larger contexts can waste more, less or roughly the same share of their output.
Figure 8: As the trajectory evolves, the waste ratio can either decrease or increase.
Figure 9: Success rate of Kimi-K3 across reasoning-effort settings. Increasing reasoning effort does not consistently improve task performance.
Figure 10: Overall waste ratio of Kimi-K3 across reasoning-effort settings, showing no consistent monotonic trend.
Figure 11: Waste ratio of reasoning across trajectory stages under different reasoning efforts.
Figure 12: Token usage (top) and waste ratio (bottom) for reasoning versus visible responses and tool calls across reasoning-effort settings.
Figure 13: Waste composition across reasoning-effort settings.
Figure 14: Change in output tokens between adjacent reasoning-effort levels, paired task by task and grouped by task-outcome changes. None → Low reduces token usage, while higher effort levels increase it.
Figure 15: Success rate versus token-waste ratio across agent harnesses with the same Kimi-K3 model and reasoning effort. Lower waste does not consistently correspond to higher task performance.
Figure 16: Median token usage per trajectory across agent harnesses, separated into chain-of-thought and visible responses and tool calls.
Figure 17: Distribution of action categories across agent harnesses, shown separately for chain-of-thought (top) and visible responses and tool calls (bottom).
Figure 18: Token-waste ratio across agent harnesses, decomposed into strict waste and detours on SWE-bench Verified and MCP-Atlas.
Figure 19: Two exemplar sources of harness-induced waste.
Figure 20: Diagnostics of harness-related token waste across complementary aspects of harness design and agent execution.
Figure 21: Task success rate (left) and token-waste ratio (right) across agent harnesses on SWE-bench Verified, with and without network access.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 22: Average number of turns (top) and output tokens (bottom) per trajectory across reasoning-effort settings.
Figure 23: Distribution of action categories in chain-of-thought (top) and visible responses and tool calls (bottom) across reasoning-effort settings.
Figure 24: Waste ratio within each action category across reasoning-effort settings, shown separately for chain-of-thought (top) and visible responses and tool calls (bottom).
The wide adoption of AI agents in complex human workflows is driving rapid growth in LLM token consumption. When agents are deployed on tasks that require a significant amount of tokens, three questions naturally arise: (1) Where do AI agents spend the tokens? (2) Which models are more token-efficient? and (3) Can agents predict their token usage before task execution? In this paper, we present the first systematic study of token consumption patterns in agentic coding tasks. We analyze trajectories from eight frontier LLMs on SWE-bench Verified and evaluate models' ability to predict their own token costs before task execution. We find that: (1) agentic tasks are uniquely expensive, consuming 1000x more tokens than code reasoning and code chat, with input tokens rather than output tokens driving the overall cost; (2) token usage is highly variable and inherently stochastic: runs on the same task can differ by up to 30x in total tokens, and higher token usage does not translate into higher accuracy; instead, accuracy often peaks at intermediate cost and saturates at higher costs; (3) models vary substantially in token efficiency: on the same tasks, Kimi-K2 and Claude-Sonnet-4.5, on average, consume over 1.5 million more tokens than GPT-5; (4) task difficulty rated by human experts only weakly aligns with actual token costs, revealing a fundamental gap between human-perceived complexity and the computational effort agents actually expend; and (5) frontier models fail to accurately predict their own token usage (with weak-to-moderate correlations, up to 0.39) and systematically underestimate real token costs. Our study offers new insights into the economics of AI agents and can inspire future research in this direction.
Longju Bai, Zhemin Huang, Xingyao Wang +5
University of Michigan · Stanford University · Microsoft AI +3
Agentic coding workflows are now widely deployed in real-world systems. With long-horizon reasoning and tool use, token usage has become an important consideration for both cost and efficiency. Two engineers using AI will solve the same problem differently. How the specification of a task shapes an agent's token spend, and whether that spend can be predicted in advance, are open questions. Here, we study the effects of different task specifications on agentic token spend with the Kimi K3 model at three thinking efforts. Across 2,700 runs, we show that reducing a full task specification to a bare user story raises token spend by 29.7%, while run-to-run variance remains unaffected by any prompt changes. We show that prompt-sensitivity is task-dependent, running from 13% to 115%. We fit a simple predictor that can price a full distribution of task specifications and thinking effort configurations from a single cheap probe on an unseen task within 36%, improving over prior work in predicting token spend. Our work provides initial results quantifying the effects of task specification on agentic token spend and introduces a method that can be used to systematically evaluate the cost of AI coding workflows.
When a large language model (LLM) agent executes the same task, token consumption can vary by over an order of magnitude across runs. The agent chooses its next steps based on tool feedback and intermediate results, while the growing context steadily inflates the input size of every subsequent call. The total consumption of a task is therefore hard to predict before execution and the prediction must be revised as the run unfolds. In this paper, we propose TokenCast, which learns a composable cost representation for each execution segment, recording its own consumption and the context growth it introduces. Composing adjacent segments yields a cumulative estimate that captures the extra input cost incurred when context from earlier segments is re-read by every later call. As execution unfolds, newly observed evidence refreshes the forecast, requiring no additional LLM calls and incurring a mean cumulative prediction time of 32.8 ms per run on SWE-bench Verified. Across 4 task suites and 6 agent models, TokenCast's mean absolute error reduction against the strongest comparator averages 14.5% over 96 evaluated combinations. In offline budget-control replay, TokenCast uses 21.3% fewer tokens on average than a fixed-budget policy at matched trace completion. The code is available at https://github.com/DEFENSE-SEU/TokenCast.
Chaoqian Ouyang, Ling Yue, Libin Zheng +7
Sun Yat-Sen University · Rensselaer Polytechnic Institute · Southeast University +1