Skill-SLM: Agent Skill-driven Small Language Models for Reliable Robot Operation

Wenhao Wang; Yanyan Li; Jiawei Yuan

cs.ROOct 7, 2026

Skill-SLM: Agent Skill-driven Small Language Models for Reliable Robot Operation

Authors: Wenhao Wang, Yanyan Li, Jiawei Yuan

Organizations: Department of Computer & Information Science, University of Massachusetts Dartmouth · Department of Computer Science & Engineering, California State University San Marcos

Abstract

Small language models (SLMs) have been increasingly adopted for onboard robot operation because they enable intelligent decision-making. However, existing approaches are mainly distillation-oriented and rely on enumerating representative task-solution pairs. This makes dataset construction difficult and limits generalization to diverse robot tasks whose possible forms grow rapidly. This paper proposes Skill-SLM, a framework that reformulates SLM-driven robot operation as a task-decomposition and skill-composition problem. Given a natural language task instruction, Skill-SLM decomposes the task into subtasks, selects appropriate skills from the skill library, and orchestrates the selected skills into executable robot operations. First, to support the skill-driven workflow, we propose a novel robot operational skill aware context-free grammar (CFG) to extract the skills required to accomplish tasks and build the skill library accordingly. Then, we configure LLM teachers to induce and synthesize training datasets for the SLMs, enabling SLMs to decompose tasks and orchestrate skills reliably. Additionally, we employ a progressive skill orchestration strategy to improve the reliability of skill implementation and overall robot operation. Experiments on UAV operation tasks indicate that Skill-SLM substantially outperforms distillation-oriented baselines, especially on unseen tasks that require generalization of capabilities. Additional experiments on ground vehicle tasks further demonstrate that Skill-SLM can be applied to different robot platforms.

Figures & tables

Fig. 1: Comparison of SLM-driven robot operation methods: (a) memorizing enumerated examples limits generalization to unseen tasks, whereas (b) Skill-SLM decomposes tasks by operational skills and recomposes them to solve unseen tasks.

Fig. 2: Overall design of Skill-SLM with an illustrative example.

Fig. 3: Process of skill library construction.

Panel of Figure 4 — Fig. 4: UAV and ground vehicle used in the experiment.

Task Set	Example Task Instructions
Seen-Pattern	“ Find the nearest car and inspect it.”
	“ Go to the bridge and return home .”
New-Combination	“ Find the nearest bridge, inspect it, and return home .”
New-Capability	“ Find the nearest tree, inspect it, then repeat the inspection reversely for opposite-perspective coverage.”

TABLE I: Example Task Instructions in the Experiments.

		Seen-Pattern		New-Combination		New-Capability
Method	Model	TSR	OSR	TSR	OSR	TSR	OSR
Ro-SLM [ 4 ]	Llama-3.2-3B-Instruct	$96.0\%$	$96.4\%$	$44.0\%$	$70.8\%$	$6.0\%$	$34.1\%$
	Llama-3.1-8B-Instruct	$96.0\%$	$98.0\%$	$28.0\%$	$62.3\%$	$26.0\%$	$52.9\%$
	Qwen3-4B	$88.0\%$	$95.4\%$	$44.0\%$	$75.8\%$	$34.0\%$	$54.3\%$
	Qwen3-8B	$84.0\%$	$89.6\%$	$34.0\%$	$69.1\%$	$30.0\%$	$52.1\%$
PRISM [ 5 ]	Llama-3.2-3B-Instruct	$78.0\%$	$90.3\%$	$28.0\%$	$75.8\%$	$56.0\%$	$81.4\%$

TABLE II: Results on the UAV task sets.

		Seen-Pattern		New-Combination		New-Capability
Ablation	Model	TSR	OSR	TSR	OSR	TSR	OSR
Direct Skill	Llama-3.2-3B-Instruct	$92.0\%$	$93.5\%$	$90.0\%$	$93.0\%$	$46.0\%$	$57.1\%$
Extraction	Llama-3.1-8B-Instruct	$96.0\%$	$97.2\%$	$92.0\%$	$94.7\%$	$68.0\%$	$75.0\%$
	Qwen3-4B	$94.0\%$	$96.0\%$	$98.0\%$	$99.5\%$	$58.0\%$	$67.7\%$
	Qwen3-8B	$98.0\%$	$99.0\%$	$98.0\%$	$99.0\%$	$88.0\%$	$91.0\%$
No	Llama-3.2-3B-Instruct	$4.0\%$	$11.6\%$	$0.0\%$	$3.0\%$	$0.0\%$	$5.1\%$

TABLE III: Ablation results on the UAV task sets.

	Seen-Pattern		New-Combination		New-Capability
Model	TSR	OSR	TSR	OSR	TSR	OSR
Llama-3.2-3B-Instruct	$90.0\%$	$94.0\%$	$85.0\%$	$88.3\%$	$70.0\%$	$77.3\%$
Llama-3.1-8B-Instruct	$100.0\%$	$100.0\%$	$90.0\%$	$92.0\%$	$85.0\%$	$91.1\%$
Qwen3-4B	$100.0\%$	$100.0\%$	$95.0\%$	$97.0\%$	$80.0\%$	$97.2\%$
Qwen3-8B	$100.0\%$	$100.0\%$	$95.0\%$	$97.0\%$	$100.0\%$	$100.0\%$

TABLE IV: Skill-SLM results on the ground vehicle task sets.

Explore similar work

May 20, 2026cs.RO

To Select or not to Select, that is the Question: Distilling Robot Skill Prediction into a Small Ensemble

As robot fleets become more heterogeneous, including humanoids, rovers, quadrupeds, and drones, selecting the right robot for a task becomes a core systems problem. We study robot skill prediction: mapping a natural-language task description to the physical capabilities required to execute it, such as fly, wheels, legs, surface water, under water and hands. Since labelled data that maps natural-language task descriptions to robot's physical capabilities does not exist, we construct a synthetic task-to-skill dataset using LLM-assisted generation and targeted label auditing. Trained on this data, a ~133M-parameter ensemble of two fine-tuned sentence encoders (mpnet + MiniLM) reaches 83.5% task-to-skill matching on a stratified 200 task dataset, outperforming Kimi K2 (1T MoE) at 72.0%, GPT-OSS-120B at 71.5%, and Llama-4-Scout-17B at 69.0% under the same zero-shot prompt. These results suggest that, for fixed robot skill taxonomies, small specialized models trained on synthetic data can outperform much larger general-purpose LLMs for fleet-level task routing.

Haechan Mark Bong, Simon Roy, Euhid Aman +1

Department of Computer Engineering and Software Engineering, Polytechnique Montréal, QC., Canada. · MILA, QC., Canada. · National Taiwan University of Science and Technology (NTUST), Taipei City, Taiwan.

Topic pending

Ask PDF

Jun 6, 2026cs.RO

CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning

Enabling robots to understand and execute tasks from natural language commands while maintaining data efficiency remains challenging. Foundation models such as vision-language-action (VLA) and vision-language models (VLMs) provide intuitive interaction channels but require extensive data; task-parameterized imitation learning achieves data efficiency but lacks natural language grounding. This work bridges this gap through a modular architecture combining task-parameterized kernelized movement primitives (TP-KMPs) with pretrained VLMs. During learning, skills are acquired from 2 to 5 kinesthetic demonstrations, and the VLM generates skill schemas describing each skill's parameters and preconditions. During execution, the VLM interprets commands to select skills, reason about parameter bindings, and create novel behaviors through covariance-weighted composition. When no skill or composition suffices, the system identifies capability gaps and requests targeted demonstrations, all without fine-tuning. Validation on a 7-DoF manipulator shows success rates of 73.3%-100% in scenarios requiring skill selection, composition, and active learning.

Markus Knauer, Valentin Gieraths, Tai Mai +4

German Aerospace Center (DLR), Institute of Robotics and Mechatronics (RMC), Münchener Str. 20, 82234 Weßling, Germany. · School of Computation, Information and Technology (CIT), Technical University of Munich (TUM), Arcisstr. 21, 80333 Munich, Germany.

Robot Skill Learning Language-Conditioned Robot Manipulation Imitation Learning Robot Imitation Learning

Ask PDF

Aug 8, 2026cs.AI

SkillSmith: Enhancing Locally Deployed Agents via Automatic Skill Construction and Evolution

LLM-based agent frameworks now act as personal assistants for multi-step tasks. Existing agent frameworks such as OpenClaw commonly follow the Cloud Agent depolyment mode using closed-source cloud LLMs as backbone model, which may expose private user information and incur repeated LLM-calling costs. Local Agents address these deployment concerns by depolying frontier open-source SLMs on user-controlled devices, but their task effectiveness still lags far behind Cloud Agents. Through diagnostic analysis, we reveal that the limited effectiveness of Local Agents with frontier SLM backbones mainly comes from missing environment knowledge caused by limited backbone model scale including environment rules and operation procedures. To supply such knowledge non-parametrically, context-efficiently, and without expert authoring, we present SkillSmith, a Cloud--Local Agent collaboration framework that uses Skill as a context-efficient knowledge carrier, automatic constructs Skill from Cloud Agent task exploration and evolves Skill using Local Agent execution feedback to enhance a frozen Local Agent. Experiments on daily agent task datasets AppWorld and WorkBench show that the automatically generated Skill enables the Local Agent with Qwen3.6-27B(SLM) to achieve task effectiveness comparable to Cloud Agents with frontier LLMs, outperform the strongest non-parametric baselines, reduce average actions per task from 36.1 to 9.9 on AppWorld-Normal, and generalize to other SLM backbone models without rerunning Skill construction.

Xinle Jiang, Remy Xie, Ming Tang

Southern University of Science and Technology, China · AOE Tech Labs Ltd., Hong Kong

LLM Agent Self-Improvement LLM Agent Skill Learning LLM Agents

Ask PDF