Demonstration-Free Skill Composition for Navigation and Manipulation
Authors: Madeleine Nouri, Gelareh Mohammadi, Francisco Cruz
Organizations: School of Computer Science and Engineering, University of New South Wales, Sydney, Australia · Escuela de Ingeniería, Universidad Central de Chile, Santiago, Chile
Hierarchical reinforcement learning enables sequential tasks to be represented as compositions of separately trained skills. This paper evaluates demonstration-free hierarchical pipelines for MiniGrid navigation and robotic manipulation, comparing their training requirements and task performance with flat reinforcement learning baselines. The evaluation concerns complete pipelines with task-specific observation, reward and control structures. In navigation, two parameterised skills are reused across unseen FourRooms layouts under manually specified high-level plans. In manipulation, reaching, grasp alignment and lifting are trained separately, with downstream skills initialised from states collected during predecessor rollouts. Across unseen navigation layouts, the hierarchical pipeline substantially outperforms the evaluated flat DQN baselines while requiring fewer training interactions. In manipulation, the hierarchical chain requires fewer training steps than flat SAC and exhibits lower observed variation across seeds. The frozen Pick and Lift skills are further reused for stacking by training an additional Place skill.
Figures & tables
Figure 1: MiniGrid FourRooms. The agent (red triangle) navigates from its start to the goal (green tile), crossing one or more doorways. Door and goal positions vary across layout seeds. The hierarchical plan is a token sequence of GoTo / CrossDoor subtasks ending at the goal.
Hyperparameter
Value
Training budget, flat baselines
500,000
Training budget, GoTo
50,000
Training budget, CrossDoor
50,000
Learning rate
3×10−4
Replay buffer size
200,000
Learning starts
5,000
Table 1: DQN hyperparameters for the discrete experiment.
Figure 2: Discrete navigation results ( n=20 seeds, mean, 95 % CI). (a) HRL skill training return and (b) episode length for the GoTo and CrossDoor subpolicies. Both subpolicies converge within ∼ 50,000 training steps, with CrossDoor converging earlier given its ∼ 5-step horizon. (c) Success rates across previously unseen layouts for HRL and the flat DQN. Dotted lines mark the overall means ( 0.55 vs 0.02 ).
Figure 3: Execution of the hierarchical Pick and Lift chain in Robosuite using a Panda robot arm: left — approaching the cube (Reach), centre — aligning and closing the gripper around the cube (Grasp), and right — raising the cube to the target height (Lift).
Figure 4: Learning curves on the Pick and Lift task ( n=15 seeds, mean, 95 % CI). Left: evaluation success rate versus environment steps for the three hierarchical skills, converging on average within ∼ 132k (reach), ∼ 47k (grasp), and ∼ 16k (lift) steps. Right: The flat baseline reaches the training stopping criterion in 14 of 15 seeds ( ∼ 292k steps on average). One seed does not reach the desired success rate within its 1M-step budget.
Method
Steps
Success (mean ± std)
95% CI
Flat SAC
∼ 292k
0.914±0.253
±0.140
HRL Reach
∼ 132k
0.978±0.051
±0.028
HRL Grasp
∼ 47k
0.940±0.055
±0.031
HRL Lift (10 cm)
∼ 16k
0.987±0.028
±0.015
Full chain
∼ 195k
0.858±0.103
±0.057
Table 2: Per-skill evaluation success rates (mean ± std, 95 % CI, 15 seeds) and full-chain result over 450 rollouts (30 per seed). Chain failures: reach 20, grasp 38, lift 6. Flat SAC is reported on the full task. The mean training step count for flat SAC is calculated over the 14 seeds that reached the success criterion, and one seed did not reach this criterion within the budget of 1,000,000 steps.
Metric
Result
Pick and Lift completion
385/450 (85.6%)
Place success given handoff
322/385 (83.6%)
Full stacking success
0.716±0.105
Table 3: Stacking with frozen Pick and Lift policies and an additionally trained Place skill. Evaluation uses 15 policy seeds and 30 shared initialisations per seed. Full stacking success is the mean ± sample standard deviation across seeds. The other metrics are pooled episode ratios.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
In embodied manipulation, Vision Language Action (VLA) models can execute atomic skills, but long-horizon manipulation requires recombining these skills without demonstrations for every skill combination. The full instruction route treats skill chaining as long-horizon imitation. In a controlled study, full instruction training on limited skill combination sequences reaches 40.0% ordered success on covered combinations but only 7.7% on recombination, suggesting memorization of sequence templates rather than robust skill recombination. We introduce Hierarchical Subtask Execution (HSE), a diagnostic framework that exposes skill recombination through a shared modular interface over the planner, trigger, and policy, decomposing long-horizon failures into separable modes. We introduce RoboCombine, a benchmark for skill recombination without combined skill training trajectories, with metrics for ordered success and transition conversion across subtask boundaries. RoboCombine reveals that atomic competence alone does not ensure ordered execution. Failures concentrate at transitions where downstream subtasks launch from handoff states from preceding subtasks rather than canonical atomic starts. We identify this mismatch as the start state gap. To address it, we introduce Pose Robust Task Initialization (PRTI), a lightweight atomic task post-training strategy that broadens start state coverage beyond canonical starts. Under oracle timed HSE on RoboCombine, PRTI reaches 18.16% ordered success, compared with 4.21% under random perturbation and 9.47% under a robot-reset oracle, without combined skill training trajectories. These results show that combinatorial generalization requires handoff robust atomic skills that convert first stage progress into reliable downstream execution, not merely decomposition into known subtasks. Code: https://github.com/AutoLab-SAI-SJTU/RoboCombine.
Xupeng Zhang, Yantai Yang, Chang Guo +2
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University
Adapting robotic manipulation to new objects and tasks often requires additional demonstrations or manual engineering. Reusable manipulation skills can reduce this effort, but adapting these skills to new scenes remains challenging. We present ManiSkillFormer, a framework for demonstration-free and compositional manipulation that connects perception and action through explicit geometric contracts. Building on reusable skill schemas, LLM agents generate contracts specifying the geometry primitives required by each skill, together with corresponding motion templates for semantic objects and task contexts. These contracts guide a perception module to ground task-relevant 3D geometry from observations, which is then used to instantiate reusable motion templates in a skill library. We evaluate ManiSkillFormer on a dual-arm robot across demonstration-free pick-and-place with 30 instances from 8 object categories, functional manipulation including unscrewing, pouring, pressing, and folding, and three long-horizon tasks. ManiSkillFormer achieves an average success rate of 88.97% for pick-and-place, 75.00% for functional manipulation, and completion rates of 50--80% across the long-horizon tasks, outperforming the evaluated baselines and two ablated pipelines. These results demonstrate the potential of explicit geometric contracts to support skill reuse and composition across objects and tasks without per-object policy fine-tuning or additional robot demonstrations. The project website is at https://patricia1019.github.io/ManiSkillFormer/.
Peiqi Yu, Mosam Dabhi, Shangtao Li +3
Carnegie Mellon University, Pittsburgh, PA 15213, USA
Cross-task generalization is a core challenge in open-world robotic manipulation, and the key lies in extracting transferable manipulation knowledge from seen tasks. Recent in-context learning approaches leverage seen task demonstrations to generate actions for unseen tasks without parameter updates. However, existing methods provide only low-level continuous action sequences as context, failing to capture composable skill knowledge and causing models to degenerate into superficial trajectory imitation. We propose Decompose and Recompose, a skill reasoning framework using atomic skill-action pairs as intermediate representations. Our approach decomposes seen demonstrations into interpretable skill--action alignments, enabling the model to recompose these skills for unseen tasks through compositional reasoning. Specifically, we construct a task-adaptive dynamic demonstration library via visual-semantic retrieval combined with skill sequences from a planning agent, complemented by a coverage-aware static library to fill missing skill patterns. Together, these yield skill-comprehensive demonstrations that explicitly elicit compositional reasoning for skill composition and execution ordering. Experiments on the AGNOSTOS benchmark and real-world environments validate our method's zero-shot cross-task generalization capability.
Xitie Zhang, Aming Wu, Yahong Han
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China