Sibyl: An Efficient Small-large Model Collaboration Framework for Long-horizon Tasks

Zhewei Fang; Yuxin Zhang; Zhenwei Shao; Mengze Li; Zheng Lin; Long Chen; Zhou Yu; Zhe Chen

cs.AIOct 4, 2026

Sibyl: An Efficient Small-large Model Collaboration Framework for Long-horizon Tasks

Authors: Zhewei Fang, Yuxin Zhang, Zhenwei Shao, Mengze Li, Zheng Lin, Long Chen, Zhou Yu, Zhe Chen, +3 more

Organizations: Hangzhou Dianzi University · Fudan University · The Hong Kong University of Science and Technology · University of Luxembourg · Simon Fraser University · Alibaba Group

Abstract

Small language models (SLMs) offer a promising foundation for on-device agents through low-latency, resource-efficient inference, yet limited reasoning and planning capabilities constrain their performance on long-horizon tasks requiring multi-step interaction with the environment. Step-level collaboration between SLMs and larger cloud-hosted models can bridge this gap, but identifying states that warrant cloud assistance remains challenging: the contribution of each cloud call is entangled with subsequent actions and can be assessed only from the final task outcome. Compounding this challenge, the SLM must balance two competing objectives: maximizing task success and minimizing cloud calls. To address this, we propose Sibyl, an algorithm that trains SLM agents to selectively consult cloud models at the step level and internalize their guidance for subsequent decisions, achieving strong task performance with minimal cloud reliance. Sibyl follows a three-stage training pipeline that (1) builds a robust base policy through consultation-free self-evolving reinforcement learning (RL); (2) cold-starts consultation behavior via decisive-disagreement state mining; and (3) jointly optimizes consultation decisions and guidance internalization through consultation-aware RL. Experiments on ALFWorld and WebShop demonstrate that Sibyl, using only a 0.6B-parameter model, outperforms state-of-the-art baselines, including agent training and routing methods, by 95.2% and 80.4% in success rate while averaging only 0.8 and 3.9 cloud calls per trajectory, respectively.

Figures & tables

Figure 1: Sibyl enables step-level device–cloud collaboration on long-horizon tasks, combining high success with low latency and cost. Right: success-rate comparisons with representative baselines; see § 5 for further results.

Figure 2: Three-stage training pipeline of Sibyl, which trains an SLM to act locally, consult the cloud at critical steps, and internalize its guidance to achieve high task success with few cloud calls.

Method	ALFWorld							WebShop
Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.
Closed-Source Model
$\diamondsuit$ Qwen3.7-Max	$88.6_{\pm 2.9}$	$100.0_{\pm 0.0}$	$80.3_{\pm 7.7}$	$66.7_{\pm 3.6}$	$64.0_{\pm 4.0}$	$95.8_{\pm 4.2}$	$82.4_{\pm 0.8}$	$52.7_{\pm 0.2}$	$46.4_{\pm 0.6}$
$\diamondsuit$ Gemini 3.1 Pro	$100.0_{\pm 0.0}$	$100.0_{\pm 0.0}$	$96.3_{\pm 3.7}$	$100.0_{\pm 0.0}$	$100.0_{\pm 0.0}$	$95.8_{\pm 4.1}$	$98.6_{\pm 1.4}$	$65.2_{\pm 0.2}$	$60.0_{\pm 0.4}$
Qwen3-0.6B assisted by Qwen3.7-Max
$\heartsuit$ GRPO	$12.4_{\pm 1.7}$	$25.6_{\pm 19.4}$	$6.2_{\pm 5.7}$	$4.2_{\pm 3.6}$	$6.7_{\pm 2.3}$	$4.2_{\pm 4.2}$	$9.1_{\pm 2.3}$	$67.1_{\pm 0.1}$	$16.0_{\pm 0.4}$

Table 1: Results on ALFWorld and WebShop, averaged over three runs.

\heartsuit

\diamondsuit

\clubsuit

, and

\spadesuit

: device-only, cloud-only, cloud-assisted training, and collaborative inference, respectively. The Qwen3-1.7B results for GRPO and SEED are from ( Wu et al., 2026 ) .

Method	ALFWorld				WebShop
Method	Succ. $\uparrow$	Call ratio $\downarrow$	Calls / Ep. $\downarrow$	Cost $\downarrow$	Score $\uparrow$	Succ. $\uparrow$	Call ratio $\downarrow$	Calls / Ep. $\downarrow$	Cost $\downarrow$
Closed-Source Model
Qwen3.7-Max	82.4	100.0	15.2	$8.6	52.7	46.4	100.0	10.8	$48.5
Gemini-3.1 Pro	98.6	100.0	9.1	$5.8	65.2	60.0	100.0	8.4	$58.6
Qwen3-0.6B assisted by Qwen3.7-Max
RouteLLM	65.7	64.5	12.5	$5.3	24.6	22.8	85.5	5.1	$18.3

Table 2: Task performance and test-time cloud cost on ALFWorld and WebShop. Call ratio is the percentage of executed steps delegated to the cloud, Calls/Ep. is the mean number of cloud calls per episode, and Cost is the total API cost over the full test set, calculated from cloud input/output token usage and API prices.

	Training API Cost (Stage 1/2/3)	Training GPU Hours	Inference API Cost (full test set)
ALFWorld	$10.7/$ 1.7/$92.7	149.7	$0.4
WebShop	$13.0/$ 6.5/$126.0	205.7	$16.1

Table 3: Training and inference costs of Sibyl on ALFWorld and WebShop.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task type	Success (%)
Pick	91.7
Look	100.0
Clean	100.0
Heat	95.7
Cool	95.2
Pick2	94.1

Appendix

Table 5: Unseen-split success by task type.

Task	Seen success (%)	Seen calls/ep.	Unseen success (%)	Unseen calls/ep.
Pick	100.0	0.1	91.7	0.5
Look	92.3	0.5	100.0	0.7
Clean	100.0	0.5	100.0	0.7
Cool	92.0	1.2	95.2	0.7
Heat	87.5	1.5	95.7	0.8
Pick2	87.5	1.3	94.1	0.6

Appendix

Table 6: Task-wise success and average cloud calls per episode on the Seen and Unseen splits.

Case	Edge-only branch	Cloud’s critical action	Edge continuation and outcome
ALFWorld: clean butterknife	30 steps, no cloud calls, and failure. The on-device SLMs takes an ordinary knife , cleans and returns it, and never reaches the target butterknife .	The cloud steers the search to diningtable 3 at step 20, where butterknife 1 is visible.	The on-device SLMs takes the butterknife, cleans it at the sink, and moves it to sidetable 1 . The collaborative run succeeds in 25 steps with 3 cloud calls.
ALFWorld: two credit cards	30 steps, no cloud calls, and failure while searching for the second card after placing the first.	The cloud first requests inventory , then chooses diningtable 1 to locate creditcard 1 , and finally armchair 1 as the destination.	The on-device SLMs takes the card and executes the final move locally. The collaborative run succeeds in 14 steps with 3 cloud calls.
WebShop: random-color bath mitt	From the same product page, the forced edge action is click[buy now] ; the purchase has empty options and scores $0.75$ .	The cloud selects the required option click[random color 1pc] .	The identical edge continuation then executes click[buy now] , producing the correct option and score $1.00$ .
WebShop: ottoman size	From the same product page, the forced edge action is click[buy now] ; the purchase has empty options and scores $0.80$ .	The cloud selects the required size click[100x40x40cm] .	The identical edge continuation executes click[buy now] , producing the correct size and score $1.00$ .

Appendix

Table 7: Representative edge–cloud rescue cases. The cloud provides the bottleneck action shown in the third column; subsequent actions are executed by the on-device SLM. The two benchmarks follow their respective evaluation protocols.

Explore similar work

Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents
May 23, 2026Yuxin Zhang, Mengxue Hu, Zheng Lin +8Personalized Large Language Model Agents Long-Horizon Task Planning
1Fudan University · 2Alibaba Group · 3The University of Hong Kong +4
Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents
Oct 6, 2026Abolfazl YounesiSmall Large Language Models
Department of Computer Engineering Sharif University of Technology Tehran, Iran
Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Models
Apr 20, 2026Hang Zeng, Xiangyu Liu, Yong Hu +5Collaboration Assistant
Shanghai Jiao Tong University, Shanghai, China · WeChat Tencent, Beijing, China · State University of New York at Buffalo, New York, United States.

May 23, 2026, cs.AIJ/K move · Enter open · S save

Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents

Yuxin Zhang, Mengxue Hu, Zheng Lin, Xiaoyi Fan+7

1Fudan University · 2Alibaba Group · 3The University of Hong Kong+4

Personalized Large Language Model Agents Long-Horizon Task Planning Cloud

Open paper AskPDF arXiv

Large language model (LLM) agents excel at solving complex long-horizon tasks through autonomous interaction with environments. However, their real-world deployment faces a fundamental device--cloud dilemma: on-device models are efficient but often brittle, while cloud models are stronger but costly in computation. State-of-the-art LLM device--cloud routers usually make coarse task-level decisions, which cannot adapt to the changing difficulty of multi-step agent interactions. To address this issue, we present Hera, a step-level device--cloud LLM agent coordinator for long-horizon tasks achieving a strong performance--cost Pareto frontier. Hera adopts a novel two-stage training paradigm: (1) imitation learning for cold-start, followed by (2) reinforcement learning that jointly optimizes task success and cloud usage efficiency. The first stage casts step-level routing as a supervised classification problem: the device agent is replayed on cloud trajectories, with each state labeled by the agreement between device and cloud actions. In the second stage, we perform cost-aware reinforcement learning by grouping identical states across trajectories and updating Hera with labels favoring higher expected return and fewer future cloud calls. We evaluate Hera on ALFWorld, WebShop, and AppWorld, where it consistently outperforms prior methods, achieving 92.5% of the cloud-only success rate with cloud use in only 46.3% of steps.

Abstract

Explore similar work

Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents

Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents

Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Models