Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems

Aikaterini Maria Panteleaki; Varatheepan Paramanayakam; Spyros Tragoudas; Iraklis Anagnostopoulos

cs.AISep 11, 2026

Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems

Authors: Aikaterini Maria Panteleaki, Varatheepan Paramanayakam, Spyros Tragoudas, Iraklis Anagnostopoulos

Organizations: School of Electrical, Computer and Biomedical Engineering Southern Illinois University Carbondale

Abstract

Large Language Models (LLMs) with function-calling capabilities are becoming critical for modern agentic AI systems. Nevertheless, current deployments typically route inferences to powerful cloud-based models, incurring significant energy use and carbon emissions. We address this sustainability challenge with a carbon-aware routing framework that distributes function-calling queries across a three-tier edge-cloud architecture, combining edge and cloud LLMs on heterogeneous hardware. At its core, a lightweight k-NN predictor operating in a unified semantic-lexical embedding space estimates query-specific accuracy, delay, and power consumption on each edge tier. These predictions are then combined with real-time grid carbon intensity to route every query to the lowest-emission tier capable of executing it successfully. Evaluated on state-of-the-art function-calling benchmarks and LLM families, our framework matches cloud-level accuracy while reducing operational carbon emissions by $4\times$ on average.

Figures & tables

Model	Success Rate	General Energy/Query	Carbon Emissions [ 6 ]
Llama3.1-70B (Cloud)	97.8%	4.35 Wh [ 6 ]	$\sim$ 1.30 $\mathrm{gCO_{2}eq}$
Llama3.1-8B (Edge)	56.5%	0.17 Wh [ 7 ]	$\sim$ 0.05 $\mathrm{gCO_{2}eq}$
Llama3.2-1B (Edge)	13.6%	0.04 Wh [ 7 ]	$\sim$ 0.01 $\mathrm{gCO_{2}eq}$

TABLE I : Success Rate on BFCL V2 [ 1 ] , Energy Consumption per query and estimated Carbon Emissions for LLMs

Fig. 1 : Carbon-aware routing framework architecture

Panel of Figure 3 — Fig. 2 : Query Distribution across three-tier architecture comparing our framework and TensorOpera [ 10 ]

Panel of Figure 4 — Fig. 3 : Performance comparison on BFCL V2 and GeoEngine: Success Rate (%), Normalized Carbon Footprint, Carbon-per-Success, and Normalized Inference Delay. All normalized metrics use Cloud Only as the reference.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 346 papers

GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization
May 12, 2026Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha +1LLM Inference Efficiency Constrained Optimization
Manipal University Jaipur · TCS Research
Small Reasoning Models are Instruction Followers in Function Calling
Aug 23, 2026Yalda Taheri, Mohammad Hassan Heydari, Erfan Naaman +1LLM Tool Use
Department of Engineering Islamic Azad University · Department of Computer Engineering University of Isfahan
CarbonEdge: Carbon-Aware Deep Learning Inference Framework for Sustainable Edge Computing
Mar 28, 2026Guilin Zhang, Wulan Guo, Ziqi Tan +2Edge Inference Edge Computing
Department of Engineering Management and Systems Engineering, George Washington University, USA · Department of Computer Science, Information, and Engineering Technology, Youngstown State University, USA

May 12, 2026J/K move · Enter open · S save

GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization

Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha, Tirthankar Dasgupta

Manipal University Jaipur · TCS Research

LLM Inference Efficiency Constrained Optimization LLM Routing Carbon-Aware Computing

Open paper AskPDF arXiv

The growing deployment of large language models (LLMs) makes per-request routing essential for balancing response quality and computational cost across heterogeneous model pools. Current routing methods rarely consider sustainable energy use and CO2 emissions as optimization objectives, despite grid carbon intensity varying by time and region, and models differing significantly in energy consumption. To address this gap, we introduce Green-Aware Routing (GAR), a constrained multi-objective optimization framework that minimizes per-request CO2 emissions subject to explicit accuracy floors and p95-latency service-level objectives (SLOs). GAR employs adaptive constraint optimization through per-dataset floor tuning and incorporates lightweight estimators for correctness, tail latency, and carbon emissions, enabling real-time routing decisions without additional inference passes. We present GAR-PD, a practical online primal-dual routing algorithm for rolling carbon budgets, alongside heuristic variants that achieve high feasibility coverage while limiting accuracy degradation. Comprehensive experiments across standard NLP benchmarks with heterogeneous LLM pools (7B-70B) demonstrate that GAR achieves substantial carbon reductions while maintaining competitive accuracy and p95 latency guarantees, providing a practical, theoretically grounded approach to sustainable LLM inference.

Abstract

LLM Routing

Explore similar work

GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization

Small Reasoning Models are Instruction Followers in Function Calling

CarbonEdge: Carbon-Aware Deep Learning Inference Framework for Sustainable Edge Computing