When Is Smaller Greener? Right-Sizing Language Models for Agentic AI

Sajid Siraj; Maria Barbati

cs.AIOct 14, 2026

When Is Smaller Greener? Right-Sizing Language Models for Agentic AI

Authors: Sajid Siraj, Maria Barbati

Organizations: Leeds University Business School, University of Leeds, Leeds, United Kingdom · Ca’Foscari University of Venice, Venice, Italy

Abstract

AI agents use large language models (LLMs) to plan and act over multiple steps, yet models are usually compared per call rather than per completed task. This paper asks: which model should an agent use to complete tasks reliably while minimising energy, cost and time? We show that per-call comparisons can be misleading for agentic AI. Because a task succeeds only if all required steps succeed, small differences in per-step reliability can produce large differences in overall task success. Failed attempts also consume resources. We therefore evaluate models using energy, cost and time per successfully completed task, treating the completed task as the functional unit. On this basis, we develop a decision model for choosing LLMs in agentic workflows. The model applies to tasks with binary or graded outcomes and to settings with or without retries. We show that each resource has a crossover task length at which a smaller, less reliable model ceases to be the greener, cheaper or faster option. These crossovers divide tasks into three regimes: for short tasks smaller models are preferred, for long tasks larger models are preferred, and only between these extremes do preferences among objectives matter. We embed these results in a two-stage framework that separates efficiency from preference. A Pareto screen first removes models that are never competitive; a weighted decision is required only when a genuine trade-off remains. A pilot study of 15 open-weight models from five developers, ranging from 0.6B to 27B parameters, on two multi-step agent tasks with measured GPU energy supports the predicted structure. The model with the lowest energy per completed task changes with task length and task type, and observed crossover points generally match those predicted from per-step measurements. These findings provide a practical framework for right-sizing language models in agentic AI.

Figures & tables

Gap	What is missing	How we address it
G1. Unit of comparison	Models are chosen on per-call measures, while energy per successful task is only measured, not used for choice.	A decision model built on resources per successful task (Section 4.4 ).
G2. When is smaller better?	Task success is known to fall with task length, but there is no rule for when this reverses the ranking of models on energy, cost or time.	Crossover lengths and three regimes (Section 4.5 ).
G3. Role of preferences	Existing multi-criteria methods apply weights to all alternatives, even where no real trade-off exists.	A preference-free screen, with weights used only where models are both efficient (Sections 4.5 and 4.8 ).
G4. Kinds of task	Methods do not distinguish binary from graded outcomes, or actions that can be retried from actions that cannot.	Four special cases of one general model (Section 4.6 ).

Table 1: Gaps in the literature on the selection question, and how this paper addresses them.

Panel of Figure 1 — Figure 1: The framework. Shaded steps use no preference information; Stage 2 (dashed) uses the weights of the task profile and runs only when more than one model is efficient. Dotted arrows show which inputs each step uses.

Quantity	Role	Measurement
$\rho_{a}$ : per-step reliability	objective (max)	share of steps completed correctly in benchmark runs of the task family
$r_{a,e}$ : energy per step	objective (min)	measured on known hardware for open-weight models; estimated with bounds for closed models
$r_{a,c}$ : cost per step	objective (min)	tokens per step multiplied by the provider’s price
$r_{a,t}$ : time per step	objective (min)	measured response time per step
Context capacity	constraint	provider specification
Safety	constraint	published safety evaluations and robustness tests

Table 2: Model quantities used by the framework. Per-step quantities are measured for a task family.

Symbol	Set by	Meaning and example
$\rho_{a}$	measurement	Per-step reliability of model $a$ on the task family.
$r_{a,e},r_{a,c},r_{a,t}$	measurement	Energy, cost and time per step of model $a$ .
$L$	task profile	Number of steps in one attempt; follows from the expected length of the task.
$\underline{q}$	decision maker	Quality floor; follows from the required accuracy. $\underline{q}=1$ for tasks that are simply done or not done.
$K$	decision maker	Maximum number of attempts; follows from whether actions can be undone. $K=1$ for irreversible actions, $K=\infty$ for free retries.
$\beta$	decision maker	Required probability that the task is completed; follows from the risk of a failed action.

Table 3: Parameters of the general model ( 5 ).

	Failures can be retried ( $K=\infty$ )	Irreversible or limited retries ( $K$ finite)
Binary outcome ( $\underline{q}=1$ )	V1. Minimise energy, cost and time per successful task. No constraint. Example: data extraction with an automatic check.	V2. Same objectives, subject to the chance constraint ( 6 ). Example: changing an airline booking.
Graded outcome ( $\underline{q}<1$ )	V3. Success means reaching the quality floor. Quality above the floor has no extra value. Example: a summary that must contain a set of required facts.	V4. Quality floor as a constraint and expected quality $\bar{q}_{a}$ as a fourth objective. Example: drafting a report, where better quality has real value.

Table 4: Four special cases of the general model ( 5 ).

Model	$\rho$	Energy	Cost	Time	Context
A	0.995	0.400	3.00	2.7	400
B	0.990	0.220	1.20	2.1	200
C	0.970	0.060	0.25	1.5	128
D	0.960	0.075	0.30	1.8	128
E	0.985	0.320	1.80	3.6	1000
F	0.930	0.030	0.10	0.9	32

Table 5: Illustrative candidate models. Per-step reliability

\rho

; energy in Wh per step; cost in US cents per step; time in seconds per step; context in thousands of tokens.

Task length $L$	Efficient models
1 to 12	F
13 to 16	C, F
17 to 21	B, C, F
22 to 49	B, C
50 to 76	A, B, C
77 to 181	A, B

Table 6: Efficient models per successful task, by task length, for the candidates in Table 5 (variant V1).

Figure 2: Resources per successful task of model C relative to model B, on a log scale, against task length. Below zero, C uses less. Each line crosses zero at the crossover length of its resource. In the shaded band neither model is better on every resource.

Quantity	How it is measured
Task success	Automatic check of the final state of each task against the correct state, as in $\tau$ -bench ( Yao et al., 2024 ; Barres et al., 2025 ) .
Per-step reliability	Estimated from complete runs, by the step at which each run first fails.
Success curve $S_{a}(L)$	Share of successful runs, grouped by the number of steps a task needs.
Energy	For open-weight models, measured per inference on known hardware, using published measurements ( Chung et al., 2025 ) and checked against an estimate based on measured speed. For closed models, estimated with stated bounds.
Cost	Tokens used, multiplied by the provider’s published price.
Time	Measured response time for each step, summed over the task.

Table 7: Measurements used in the evaluation.

Panel of Figure 3 — Figure 3: The 15 models on two per-step measures: GPU energy per step, and the expected number of steps before the first error, $1/(1-\hat{\rho})$ , both on log scales. Colours and markers show the developer. Hollow markers are models removed by the per-step screen. Better models lie towards the top left.

Panel of Figure 4 — Figure 4: GPU energy per successfully completed task against task length, on a log scale, for the models that use the least energy per completed task at some length, and for the most reliable model. Lines use the estimated per-step reliability and the measured energy of each step; points are the observed values, every five steps, in the colour and marker of their model. Hollow points rest on fewer than three successful runs out of 15, so they are uncertain.

LLM Agent Workflow Optimization

LLM agent workflow optimization studies how to design and run multi-step agents for better task success, reliability, latency, and cost.

See all 243 papers

Oct 15, 2026

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.

Kunlun Zhu, Tao Feng, Xu Han +5

University of Illinois Urbana-Champaign · Princeton University · Google

LLM Agent Harnesses LLM Agents Tool Selection for Language Model Agents LLM Decision-Making AI Agent Benchmarks LLM Agent Training+1

Ask PDF

Jun 2, 2025

Small Language Models are the Future of Agentic AI

Large language models (LLMs) are often praised for exhibiting near-human performance on a wide range of tasks and valued for their ability to hold a general conversation. The rise of agentic AI systems is, however, ushering in a mass of applications in which language models perform a small number of specialized tasks repetitively and with little variation. Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI. Our argumentation is grounded in the current level of capabilities exhibited by SLMs, the common architectures of agentic systems, and the economy of LM deployment. We further argue that in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice. We discuss the potential barriers for the adoption of SLMs in agentic systems and outline a general LLM-to-SLM agent conversion algorithm. Our position, formulated as a value statement, highlights the significance of the operational and economic impact even a partial shift from LLMs to SLMs is to have on the AI agent industry. We aim to stimulate the discussion on the effective use of AI resources and hope to advance the efforts to lower the costs of AI of the present day. Calling for both contributions to and critique of our position, we commit to publishing all such correspondence at https://research.nvidia.com/labs/lpr/slm-agents.

Peter Belcak, Greg Heinrich, Shizhe Diao +5

NVIDIA Research · Georgia Institute of Technology

Small Language Models LLM Agents

Ask PDF

Oct 2, 2026

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows

Large language models (LLMs) are increasingly used in software engineering, including agentic systems that coordinate multiple agents, but impose higher computational and environmental costs. In this paper, we present a comprehensive empirical study of agentic LLM systems across five software engineering tasks: code generation, technical debt identification, code vulnerability detection, log parsing, and log analysis. For each task, we compare LLM configurations that range from a non-agentic single-query baseline to multi-agent workflows, using six open-weight LLMs, two prompt strategies, and three hardware platforms. We assess each configuration in terms of accuracy, inference latency, and energy consumption. Our results reveal substantial trade-offs between agentic complexity and energy efficiency: multi-agent designs consume on average 6.36

\times

as much energy and run 6.07

\times

as long as the non-agentic baseline, with worst-case slowdowns of up to 160

\times

for individual task--hardware pairs. Accuracy gains from additional agents are limited and task-specific: multi-agent improves average vulnerability-detection accuracy, but lightweight non-agentic and single-agent configurations still dominate the Pareto front, accounting for 59 of 66 Pareto-optimal configurations. Model and prompt choice act as task-specific levers whose effective direction varies between tasks rather than as global defaults. We translate these findings into design guidelines for sustainable, task-aware LLM-based development tools.

Merve Astekin, Yan Naing Tun, Arda Goknil +5

SINTEF, Oslo, Norway · Singapore Management University, Singapore, Singapore · Ozyegin University, Istanbul, Turkiye

Software Engineering Agents LLM Agent Evaluation Multi-Agent LLM Systems LLM Inference Efficiency Energy-Efficient ML

Ask PDF

Abstract

LLM Agent Workflow Optimization

Explore similar work

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Small Language Models are the Future of Agentic AI

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows