SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming

John Woods; Hasti Seifi

cs.ROAug 14, 2026

SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming

Authors: John Woods, Hasti Seifi

Organizations: School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ

Abstract

Natural-language interfaces can lower the barrier to programming robots, but existing systems struggle when users request complex tasks. While large language models (LLMs) perform well with simple commands, they often struggle to generate code for multi-step tasks, decompose high-level instructions, or reuse prior solutions. We present SkillComposer, an interactive natural-language robot programming system for simulation environments that continually learns reusable program abstractions. SkillComposer uses a generate-test architecture in which an LLM iteratively generates and revises robot programs before execution. Successful programs are stored and processed by an online library-learning algorithm that compresses recurring function sequences into reusable macro skills for future tasks. We evaluate SkillComposer through ablation experiments and a user study with 12 participants to determine its effectiveness on manipulation and robot caregiving tasks. The results show that evaluator-guided generation and learned abstractions improve success rates and usability while reducing user effort in natural-language robot programming.

Figures & tables

Fig. 1 : An overview of the SkillComposer architecture. Blue modules are natural language models, the orange module is the library learning algorithm, yellow modules are static algorithms, and green modules are data structures.

Fig. 2 : The SkillComposer user interface, which contains a live camera feed from the simulation and a chat interface for interacting with the system.

Fig. 3 : An equivalent portion of code in the JSON representation (top) and s-expression representation (bottom). Default arguments are added into the s-expression based on the defaults listed in the function schema.

Panel of Figure 4 — Fig. 4 : Three simulation environments used in the experiments. (a) Objects: evaluates object manipulation, spatial reasoning, multi-step actions, and goal planning (prompts 1–20). (b) Feeding: evaluates simple and multi-step HRI tasks (prompts 21–40) and is used in the main user study. (c) Bananas: used for the user study practice task.

Prompt Category	Example Prompt	System	Success Rate	Program Length	Macro Usage (%)	Runtime (s)
Object Manipulation	Grab a yellow fruit	Coder LLM	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	–	2.26 $\pm$ 0.80
		Coder LLM + Eval LLM	0.96 $\pm$ 0.09	1.32 $\pm$ 0.44	–	9.58 $\pm$ 9.86
		Coder LLM + Stitch	1.00 $\pm$ 0.00	1.08 $\pm$ 0.11	0.00 $\pm$ 0.00	2.24 $\pm$ 0.67
		SkillComposer	0.96 $\pm$ 0.09	1.32 $\pm$ 0.36	8.00 $\pm$ 17.89	13.97 $\pm$ 9.80
Spatial Reasoning	Move rightmost banana to the left of the can	Coder LLM	0.24 $\pm$ 0.43	0.96 $\pm$ 0.41	–	11.94 $\pm$ 4.05
		Coder LLM + Eval LLM	0.76 $\pm$ 0.43	2.82 $\pm$ 0.76	–	50.43 $\pm$ 17.02

TABLE I: Summary of ablation experiments for each prompt category, averaged over all trials and environments. A representative prompt from each category is provided for reference.

Fig. 5 : Top: Average SkillComposer macro usage across five runs in the Objects (left) and Feeding (right) environments versus the number of input prompts. Bottom: Average number of learned macros across five runs versus input prompts.

System	Task	Success Rate	Num. Prompts	Task Time (m:ss)
Baseline	Rearrange Objects	0.50 $\pm$ 0.52	5.8 $\pm$ 2.2	6:09 $\pm$ 2:57
	Meal Preparation	0.92 $\pm$ 0.29	8.2 $\pm$ 5.6	6:10 $\pm$ 3:19
	Feed Care Recipient	0.75 $\pm$ 0.45	6.6 $\pm$ 2.7	3:48 $\pm$ 1:43
SkillComposer	Rearrange Objects	0.92 $\pm$ 0.29	1.9 $\pm$ 1.2	3:20 $\pm$ 1:20
	Meal Preparation	0.92 $\pm$ 0.29	3.2 $\pm$ 2.6	3:59 $\pm$ 1:48
	Feed Care Recipient	0.67 $\pm$ 0.49	3.5 $\pm$ 1.6	4:16 $\pm$ 1:57

TABLE II: Summary of user study task metrics, averaged over all participants.

Fig. 6 : Post-questionnaire results from the user study. SUS is measured on a 100-point scale [ 25 ] , while the other four questions are measured on a 7-point Likert scale, with 7 being the highest. Error bars report

\mu\pm 2\sigma

Code Generation for Robotics

See all 20 papers

ASPIRE: Agentic /Skills Discovery for Robotics
Jun 30, 2026Runyu Lu, Yubo Wu, Ethan Kou +11Robot Policy Learning Long-Horizon Robotic Manipulation
NVIDIA · UMich · UIUC +2
CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning
Jun 6, 2026Markus Knauer, Valentin Gieraths, Tai Mai +4Robot Skill Learning Language-Conditioned Robot Manipulation
German Aerospace Center (DLR), Institute of Robotics and Mechatronics (RMC), Münchener Str. 20, 82234 Weßling, Germany. · School of Computation, Information and Technology (CIT), Technical University of Munich (TUM), Arcisstr. 21, 80333 Munich, Germany.
Robotic Ultra-Long-Horizon Manipulation Skills via Human-guided Lifelong Code Generation
Sep 23, 2025Yuan Meng, Zhenguo Sun, Max Fest +4Code Generation for Robotics Long-Horizon Robotic Manipulation
School of Computation, Information and Technology Technical University of Munich Munich, Germany · Beijing Academy of Artificial Intelligence (BAAI) Beijing, China · The State Key Laboratory for Novel Software Technology Nanjing University Suzhou, China

Jun 30, 2026J/K move · Enter open · S save

ASPIRE: Agentic /Skills Discovery for Robotics

Runyu Lu, Yubo Wu, Ethan Kou, Letian Fu+10

NVIDIA · UMich · UIUC+2

Robot Policy Learning Long-Horizon Robotic Manipulation Continual Robot Learning Robot Skill Learning Multi-Task Robotic Manipulation AI Control+7

Open paper AskPDF arXiv

Traditional robot programming is challenging: it requires orchestrating multimodal perception, managing physical contact dynamics, and handling diverse configurations and execution failures. We introduce ASPIRE (Agentic Skill Programming through Iterative Robot Exploration), a continual learning system that autonomously writes and refines robot control programs in a code-as-policy paradigm while compounding experience into a reusable skill library. ASPIRE discovers skills that persist across tasks, simulation and real-world settings, and embodiments. It operates in an open-ended loop with three components: (1) a closed-loop robot execution engine that exposes fine-grained multimodal traces, enabling autonomous failure diagnosis, repair synthesis, and validation; (2) a continually expanding skill library that distills validated fixes into reusable, transferable knowledge; and (3) evolutionary search that generates diverse task sequences and control programs to explore beyond single-trajectory refinement. ASPIRE surpasses prior methods by up to 77% on LIBERO-Pro manipulation under perturbation, 72% on Robosuite bimanual handover, and 32% on BEHAVIOR-1K long-horizon household tasks. Its accumulated library also enables zero-shot generalization to unseen long-horizon tasks: on LIBERO-Pro Long, ASPIRE achieves 31% success versus 4% for prior methods despite their use of test-time reasoning and retries. Finally, simulation-discovered skills provide initial evidence of sim-to-real transfer, substantially reducing real-robot programming effort across different embodiments and robot APIs.

Abstract

Code Generation for Robotics

Explore similar work

ASPIRE: Agentic /Skills Discovery for Robotics

CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning

Robotic Ultra-Long-Horizon Manipulation Skills via Human-guided Lifelong Code Generation