When Is Smaller Greener? Right-Sizing Language Models for Agentic AI

Sajid Siraj; Maria Barbati

cs.AIOct 14, 2026

When Is Smaller Greener? Right-Sizing Language Models for Agentic AI

Authors: Sajid Siraj, Maria Barbati

Organizations: Leeds University Business School, University of Leeds, Leeds, United Kingdom · Ca’Foscari University of Venice, Venice, Italy

Abstract

AI agents use large language models (LLMs) to plan and act over multiple steps, yet models are usually compared per call rather than per completed task. This paper asks: which model should an agent use to complete tasks reliably while minimising energy, cost and time? We show that per-call comparisons can be misleading for agentic AI. Because a task succeeds only if all required steps succeed, small differences in per-step reliability can produce large differences in overall task success. Failed attempts also consume resources. We therefore evaluate models using energy, cost and time per successfully completed task, treating the completed task as the functional unit. On this basis, we develop a decision model for choosing LLMs in agentic workflows. The model applies to tasks with binary or graded outcomes and to settings with or without retries. We show that each resource has a crossover task length at which a smaller, less reliable model ceases to be the greener, cheaper or faster option. These crossovers divide tasks into three regimes: for short tasks smaller models are preferred, for long tasks larger models are preferred, and only between these extremes do preferences among objectives matter. We embed these results in a two-stage framework that separates efficiency from preference. A Pareto screen first removes models that are never competitive; a weighted decision is required only when a genuine trade-off remains. A pilot study of 15 open-weight models from five developers, ranging from 0.6B to 27B parameters, on two multi-step agent tasks with measured GPU energy supports the predicted structure. The model with the lowest energy per completed task changes with task length and task type, and observed crossover points generally match those predicted from per-step measurements. These findings provide a practical framework for right-sizing language models in agentic AI.

Figures & tables

Gap	What is missing	How we address it
G1. Unit of comparison	Models are chosen on per-call measures, while energy per successful task is only measured, not used for choice.	A decision model built on resources per successful task (Section 4.4 ).
G2. When is smaller better?	Task success is known to fall with task length, but there is no rule for when this reverses the ranking of models on energy, cost or time.	Crossover lengths and three regimes (Section 4.5 ).
G3. Role of preferences	Existing multi-criteria methods apply weights to all alternatives, even where no real trade-off exists.	A preference-free screen, with weights used only where models are both efficient (Sections 4.5 and 4.8 ).
G4. Kinds of task	Methods do not distinguish binary from graded outcomes, or actions that can be retried from actions that cannot.	Four special cases of one general model (Section 4.6 ).

Table 1: Gaps in the literature on the selection question, and how this paper addresses them.

Panel of Figure 1 — Figure 1: The framework. Shaded steps use no preference information; Stage 2 (dashed) uses the weights of the task profile and runs only when more than one model is efficient. Dotted arrows show which inputs each step uses.

Quantity	Role	Measurement
$\rho_{a}$ : per-step reliability	objective (max)	share of steps completed correctly in benchmark runs of the task family
$r_{a,e}$ : energy per step	objective (min)	measured on known hardware for open-weight models; estimated with bounds for closed models
$r_{a,c}$ : cost per step	objective (min)	tokens per step multiplied by the provider’s price
$r_{a,t}$ : time per step	objective (min)	measured response time per step
Context capacity	constraint	provider specification
Safety	constraint	published safety evaluations and robustness tests

Table 2: Model quantities used by the framework. Per-step quantities are measured for a task family.

Symbol	Set by	Meaning and example
$\rho_{a}$	measurement	Per-step reliability of model $a$ on the task family.
$r_{a,e},r_{a,c},r_{a,t}$	measurement	Energy, cost and time per step of model $a$ .
$L$	task profile	Number of steps in one attempt; follows from the expected length of the task.
$\underline{q}$	decision maker	Quality floor; follows from the required accuracy. $\underline{q}=1$ for tasks that are simply done or not done.
$K$	decision maker	Maximum number of attempts; follows from whether actions can be undone. $K=1$ for irreversible actions, $K=\infty$ for free retries.
$\beta$	decision maker	Required probability that the task is completed; follows from the risk of a failed action.

Table 3: Parameters of the general model ( 5 ).

	Failures can be retried ( $K=\infty$ )	Irreversible or limited retries ( $K$ finite)
Binary outcome ( $\underline{q}=1$ )	V1. Minimise energy, cost and time per successful task. No constraint. Example: data extraction with an automatic check.	V2. Same objectives, subject to the chance constraint ( 6 ). Example: changing an airline booking.
Graded outcome ( $\underline{q}<1$ )	V3. Success means reaching the quality floor. Quality above the floor has no extra value. Example: a summary that must contain a set of required facts.	V4. Quality floor as a constraint and expected quality $\bar{q}_{a}$ as a fourth objective. Example: drafting a report, where better quality has real value.

Table 4: Four special cases of the general model ( 5 ).

Model	$\rho$	Energy	Cost	Time	Context
A	0.995	0.400	3.00	2.7	400
B	0.990	0.220	1.20	2.1	200
C	0.970	0.060	0.25	1.5	128
D	0.960	0.075	0.30	1.8	128
E	0.985	0.320	1.80	3.6	1000
F	0.930	0.030	0.10	0.9	32

Table 5: Illustrative candidate models. Per-step reliability

\rho

; energy in Wh per step; cost in US cents per step; time in seconds per step; context in thousands of tokens.

Task length $L$	Efficient models
1 to 12	F
13 to 16	C, F
17 to 21	B, C, F
22 to 49	B, C
50 to 76	A, B, C
77 to 181	A, B

Table 6: Efficient models per successful task, by task length, for the candidates in Table 5 (variant V1).

Figure 2: Resources per successful task of model C relative to model B, on a log scale, against task length. Below zero, C uses less. Each line crosses zero at the crossover length of its resource. In the shaded band neither model is better on every resource.

Quantity	How it is measured
Task success	Automatic check of the final state of each task against the correct state, as in $\tau$ -bench ( Yao et al., 2024 ; Barres et al., 2025 ) .
Per-step reliability	Estimated from complete runs, by the step at which each run first fails.
Success curve $S_{a}(L)$	Share of successful runs, grouped by the number of steps a task needs.
Energy	For open-weight models, measured per inference on known hardware, using published measurements ( Chung et al., 2025 ) and checked against an estimate based on measured speed. For closed models, estimated with stated bounds.
Cost	Tokens used, multiplied by the provider’s published price.
Time	Measured response time for each step, summed over the task.

Table 7: Measurements used in the evaluation.

Panel of Figure 3 — Figure 3: The 15 models on two per-step measures: GPU energy per step, and the expected number of steps before the first error, $1/(1-\hat{\rho})$ , both on log scales. Colours and markers show the developer. Hollow markers are models removed by the per-step screen. Better models lie towards the top left.

Panel of Figure 4 — Figure 4: GPU energy per successfully completed task against task length, on a log scale, for the models that use the least energy per completed task at some length, and for the most reliable model. Lines use the estimated per-step reliability and the measured energy of each step; points are the observed values, every five steps, in the colour and marker of their model. Hollow points rest on fewer than three successful runs out of 15, so they are uncertain.

LLM Agent Workflow Optimization

LLM agent workflow optimization studies how to design and run multi-step agents for better task success, reliability, latency, and cost.

See all 243 papers

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs
Oct 15, 2026Kunlun Zhu, Tao Feng, Xu Han +5LLM Agent Harnesses LLM Agents
University of Illinois Urbana-Champaign · Princeton University · Google
Small Language Models are the Future of Agentic AI
Jun 2, 2025Peter Belcak, Greg Heinrich, Shizhe Diao +5Small Language Models LLM Agents
NVIDIA Research · Georgia Institute of Technology
Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows
Oct 2, 2026Merve Astekin, Yan Naing Tun, Arda Goknil +5Software Engineering Agents LLM Agent Evaluation
SINTEF, Oslo, Norway · Singapore Management University, Singapore, Singapore · Ozyegin University, Istanbul, Turkiye

Oct 15, 2026J/K move · Enter open · S save

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Kunlun Zhu, Tao Feng, Xu Han, Heng Wang+4

University of Illinois Urbana-Champaign · Princeton University · Google

LLM Agent Harnesses LLM Agents Tool Selection for Language Model Agents LLM Decision-Making AI Agent Benchmarks LLM Agent Training+1

Open paper AskPDF arXiv

Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.

Abstract

LLM Agent Workflow Optimization

Explore similar work

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Small Language Models are the Future of Agentic AI

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows