Organizations use engineers' token spend to monitor AI adoption, set budgets, and measure productivity, but the metric's relationship with engineering output is unclear. We link five weeks of token usage in coding agents to the commits of 404 engineers at JetBrains, and analyze a company survey of 412 employees. We find that token spend varied mostly between engineers, and which engineers spent the most stayed largely the same over the five weeks. Among weeks with similar commit counts, the 90th percentile of spend was 39 to 240 times the 10th. Within an engineer, doubling spend was associated with about 10% more commits. Available work metrics explained 38% of the variation in spend between engineers and 13% of the week-to-week variation within an engineer. In the survey, respondents in technical roles reported in nearly nine of ten agent evaluations using agents for tasks beyond coding, such as analysis, research, and planning. Token spend therefore measures how much engineers use agents across many kinds of work and not the outcome and value of that work. We recommend using token spend at most for resource accounting, and evaluating agent-assisted work with evidence about tasks and outcomes.
Figures & tables
Fig. 1: Within each weekly-commit quintile, the 90th percentile of token spend is 39 to 240 times the 10th (numbers at the top). Lines mark the 10th, 50th, and 90th percentiles; spend is relative to the median engineer-week. Ties are split at quintile edges, so adjacent ranges share an endpoint.
Fig. 2: Token spend relative to the median, per metered person-week (left, n=1,599 ) and per engineer (right, n=397 ), over the five analysis weeks. Solid lines: medians. Dotted lines: 10th and 90th percentiles, at 0.05 and 5.3 times the median for person-weeks and 0.06 and 4.3 for engineers. Values under a thousandth of the median are pooled in the first bin.
Fig. 3: Spend vs. commits, both on log10 scales (slope 1 is proportional; lines are least-squares fits; squares are means of 5% slices). (A) Engineers’ average weeks. (B) Engineer-weeks relative to the engineer’s own mean.
Fig. 4: Share of variation in weekly token use that each set of predictors explains. “Which person” is fit in sample; “same, within person” is tested on held-out weeks; every other set on held-out people. “Added” bars are separate fits. A negative share means the set predicts held-out people worse than the week’s average spend alone.
Fig. 5: Tokens relative to the engineer’s average week, by the week’s commit count: observed (dots) and predicted by each curve shape (lines), with the share of tokens attributed to zero output under each shape in parentheses.
Fig. 6: Share of each group’s tokens not tied to output, by the spline. Dot area is the group’s share of all tokens; the dashed line marks all users.
Fig. 7: Activities for which technical respondents reported using the two coding agents in the past work week. ( n=167 ; multiple selections allowed).
Fig. 8: Self-reported time savings or losses from coding-agent use among technical respondents. (A) Hours saved or lost in the past work week, grouped by the number of non-coding activity types reported. Each dot is one evaluation and the bar its group’s median; the vertical axis is compressed above 2 hours. (B) Associations with reported time savings from a regression model including all seven predictors simultaneously (134 evaluations; R² = 0.43). Predictors were standardized, and the outcome was asinh-transformed. Points show coefficient estimates; lines show 95% bootstrap confidence intervals. Blue intervals exclude zero.
Fig. 9: Thematic coding of the free-text answers on what cost time and on enabled work. Figure 13 in SM B shows the other two questions.
Fig. 10: Effect of the coding agents on six work aspects, tech. respondents.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 11: Within-person correlation between an engineer’s tokens on a working day and their commits k working days later, both as log(1+x) with person and day means removed. Small negative values at long lags are expected once each engineer’s mean over five weeks is removed.
Fig. 12: (A) Rank correlation of weekly tokens between every pair of analysis weeks, over engineers active in both. (B) Tokens per commit over weeks 0, 2, and 4 vs. weeks 1 and 3; filled dots are in the top decile of both halves (17 engineers), open dots in the top decile of only one (34 engineers).
Fig. 13: Thematic coding of answers by respondents in technical roles to open-ended questions about what saved them time while working with agents and the agent’s main perceived value (AI-assisted coding, checked by one author), as in Figure 9 .
Agentic coding workflows are now widely deployed in real-world systems. With long-horizon reasoning and tool use, token usage has become an important consideration for both cost and efficiency. Two engineers using AI will solve the same problem differently. How the specification of a task shapes an agent's token spend, and whether that spend can be predicted in advance, are open questions. Here, we study the effects of different task specifications on agentic token spend with the Kimi K3 model at three thinking efforts. Across 2,700 runs, we show that reducing a full task specification to a bare user story raises token spend by 29.7%, while run-to-run variance remains unaffected by any prompt changes. We show that prompt-sensitivity is task-dependent, running from 13% to 115%. We fit a simple predictor that can price a full distribution of task specifications and thinking effort configurations from a single cheap probe on an unseen task within 36%, improving over prior work in predicting token spend. Our work provides initial results quantifying the effects of task specification on agentic token spend and introduces a method that can be used to systematically evaluate the cost of AI coding workflows.
The wide adoption of AI agents in complex human workflows is driving rapid growth in LLM token consumption. When agents are deployed on tasks that require a significant amount of tokens, three questions naturally arise: (1) Where do AI agents spend the tokens? (2) Which models are more token-efficient? and (3) Can agents predict their token usage before task execution? In this paper, we present the first systematic study of token consumption patterns in agentic coding tasks. We analyze trajectories from eight frontier LLMs on SWE-bench Verified and evaluate models' ability to predict their own token costs before task execution. We find that: (1) agentic tasks are uniquely expensive, consuming 1000x more tokens than code reasoning and code chat, with input tokens rather than output tokens driving the overall cost; (2) token usage is highly variable and inherently stochastic: runs on the same task can differ by up to 30x in total tokens, and higher token usage does not translate into higher accuracy; instead, accuracy often peaks at intermediate cost and saturates at higher costs; (3) models vary substantially in token efficiency: on the same tasks, Kimi-K2 and Claude-Sonnet-4.5, on average, consume over 1.5 million more tokens than GPT-5; (4) task difficulty rated by human experts only weakly aligns with actual token costs, revealing a fundamental gap between human-perceived complexity and the computational effort agents actually expend; and (5) frontier models fail to accurately predict their own token usage (with weak-to-moderate correlations, up to 0.39) and systematically underestimate real token costs. Our study offers new insights into the economics of AI agents and can inspire future research in this direction.
Longju Bai, Zhemin Huang, Xingyao Wang +5
University of Michigan · Stanford University · Microsoft AI +3
Advances in frontier coding agents have sharply increased the token costs of AI-assisted software engineering. Their token efficiency is typically evaluated on isolated tasks, whereas real-world developers use them to continuously build, update, and maintain large-scale projects. This mismatch motivates us to evaluate agents over sequences of realistic coding tasks, where we find that agents exhibit recurring environment-dependent token costs that existing methods struggle to reduce. To minimize these costs, we introduce TokenEconomizer, an external controller that cuts environment-specific token costs over a sequence of tasks. These cuts come from determining (i) which operational micro-skills to inject, (ii) which persistently irrelevant tools to prune, and (iii) when to intervene in unproductive failure loops. Across Claude Code, Codex, and GitHub Copilot CLI paired with frontier models, TokenEconomizer reduces the average token cost by 18.4-30.9% with comparable task success. Savings hold in every repository under every tested agent-model setting and generalize to diverse domains such as sequential web development and security auditing, where costs fall by 6.4-21.2% depending on task structure. Our work establishes adaptive black-box cost control as a promising direction for reducing token costs of coding agents in realistic settings.
Aaron J. Li, Weijia Xu, Chandan Singh +8
UC Berkeley · Microsoft Research · Microsoft Security