Should SLMs be in the Driver's Seat? A Study of Local Models as Interpreters in Cloud-Written Agent Programs

Boris Radovič; Marco Canini; Veljko Pejović

cs.AIOct 9, 2026

Should SLMs be in the Driver's Seat? A Study of Local Models as Interpreters in Cloud-Written Agent Programs

Authors: Boris Radovič, Marco Canini, Veljko Pejović

Organizations: King Abdullah University of Science and Technology · University of Ljubljana, Faculty of Computer and Information Science

Abstract

The most accurate agents executing user-provided tasks are driven by large, cloud-hosted models, which solve them by reading the private data of the user. Agents powered by models small enough to run on edge devices keep that data local, but yield far worse results. A third option is to have cloud models write a program that devices execute: the data stays local, but as the program is written before any data is seen, it fails whenever a step depends on interpreting content that appears only at run time. We explore an intermediate architecture, in which the cloud-written program defers bounded questions over user data to a small local model through an ordinary function call. On AppWorld this improves the success rate of a code-only agent in all six combinations of backbone model and test partition we evaluate, by up to 14.9pp, recovering a substantial part of the gap to a cloud-hosted ReAct loop while keeping user data on the device.

Figures & tables

Figure 1: Success rate of the ReAct agent on the AppWorld benchmark on the Test Normal (TN) and Test Challenge (TC) partitions, w.r.t. the number of LM parameters.

Agent	Interprets run-time data	Local LM cost	User data to cloud	Cloud tokens	Cloud-device round trips
Cloud ReAct	✓	none	every observation	high	one per step
Edge ReAct	✓	high	none	none	none
Full-Code (FC)	✗	none	request, traceback	moderate	$\leq 5$
Ours	✓	low	request, traceback	moderate	$\leq 5$

Table 1: Qualitative comparison of the execution strategies.

SLM size	Qwen-coder	Qwen-122B	Qwen-27B
None (FC)	$33.3$ $\pm 3.6$	$42.3$ $\pm 3.8$	$45.8$ $\pm 3.9$
0.8B	$33.9$ $\pm 3.8$	$48.8$ $\pm 3.9$	$52.4$ $\pm 3.9$
2B	$36.9$ $\pm 3.7$	$51.2$ $\pm 3.9$	$61.3$ $\pm 3.7$
9B	$38.7$ $\pm 3.7$	$52.4$ $\pm 3.9$	$60.7$ $\pm 3.8$

Table 3: SR on TN w.r.t. SLM size.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Quantization	GPU
Qwen-27B (Qwen3.6-27B)	UD-Q8_K_XL	A100-80GB
Qwen-122B (Qwen3.5-122B-A10B)	UD-Q4_K_XL	A100-80GB
Qwen-coder (Qwen3-Coder-Next)	Q6_K	A100-80GB
Qwen3.5-0.8B/2B/4B/9B	UD-Q8_K_XL	V100
Gemma-4 E2B–31B	UD-Q8_K_XL	A100-80GB

Appendix

Table 5: Employed models. All weights are GGUF files (UD = Unsloth Dynamic quantization).

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 843 papers

Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents
Oct 6, 2026Abolfazl YounesiSmall Language Models LLM Agents
Department of Computer Engineering Sharif University of Technology Tehran, Iran
Sibyl: An Efficient Small-large Model Collaboration Framework for Long-horizon Tasks
Oct 4, 2026Zhewei Fang, Yuxin Zhang, Zhenwei Shao +8LLM Agent Training Long-Horizon LLM Agents
Hangzhou Dianzi University · Fudan University · The Hong Kong University of Science and Technology +3
EdgeWise: Bringing Long-Horizon Agents to the Edge via Selective Cloud Delegation
Oct 13, 2026Zeru Shi, Jindong Jiang, Ryo Hachiuma +10Edge-Cloud Collaborative Inference LLM Agents
NVIDIA · Rutgers University

Oct 6, 2026J/K move · Enter open · S save

Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents

Abolfazl Younesi

Department of Computer Engineering Sharif University of Technology Tehran, Iran

Small Language Models LLM Agents LLM Routing LLM Agent Reliability

Open paper AskPDF arXiv

Small language models (SLMs) are attractive as local agent controllers because they reduce remote inference, latency, and deployment footprint, yet structured tool errors can cause an agent step to fail. Existing routers typically select a model once per query. However, agents expose sequential decision points whose difficulty dynamically changes based on intermediate observations. We propose STEPGATE, an uncertainty-aware handoff framework that scores each local SLM action and selectively escalates challenging steps to a stronger model. On a 52-task held-out single-step BFCL-derived test split, the Qwen2.5-1.5B/7B pair attains 82.7% task success with 30.8% escalation, versus 67.3% local-only and 75.4% random escalation (which uses 33.8% escalation). In a separate multi-turn evaluation, STEPGATE achieves 69.0% trajectory success and 84.0% action success using only 30.0% cloud actions, compared with 48.0%/70.5% local-only, 60.0%/78.2% random escalation, and 57.0%/77.1% query-level routing (strong-only achieves 82.0% trajectory success at 100% cloud actions). These results suggest that step-level escalation recovers a large share of the performance gap to the stronger Qwen2.5-7B backend at a matched cloud-action rate while transmitting fewer tokens remotely. However, our evaluation is limited to one model family, a single stronger backend, and scripted tasks. Furthermore, the test sets are small, multi-turn comparisons rely on paired intervals and statistical tests, and our risk tiers serve as research annotations rather than formal safety guarantees.

Abstract

LLM Agent Reliability

Explore similar work

Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents

Sibyl: An Efficient Small-large Model Collaboration Framework for Long-horizon Tasks

EdgeWise: Bringing Long-Horizon Agents to the Edge via Selective Cloud Delegation