cs.ROJan 16, 2026

Skill-Aware Diffusion for Generalizable Robotic Manipulation

Authors: Aoshen Huang, Jiaming Chen, Jiyu Cheng, Ran Song, Wei Pan, Wei Zhang

Organizations: School of Control Science and Engineering, Shandong University, Jinan 250061, China. · Department of Computer Science, The University of Manchester, M13 9PL Manchester, U.K.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Robust generalization in robotic manipulation is crucial for robots to adapt flexibly to diverse environments. Existing methods usually improve generalization by scaling data and networks, but model tasks independently and overlook skill-level information. Observing that tasks within the same skill share similar motion patterns, we propose Skill-Aware Diffusion (SADiff), which explicitly incorporates skill-level information to improve generalization. SADiff learns skill-specific representations through a skill-aware encoding module with learnable skill tokens, and conditions a skill-constrained diffusion model to generate object-centric motion flow. A skill-retrieval transformation strategy further exploits skill-specific trajectory priors to refine the mapping from 2D motion flow to executable 3D actions. Furthermore, we introduce IsaacSkill, a high-fidelity dataset containing fundamental robotic skills for comprehensive evaluation and sim-to-real transfer. Experiments in simulation and real-world settings show that SADiff achieves good performance and generalization across various manipulation tasks. Dataset and videos are available at https://sites.google.com/view/sa-diff.

Figures & tables

Fig. 1: Overview of the proposed Skill-Aware Diffusion (SADiff) framework. It is structured into three phases: (1) The encoding phase, where the skill-aware encoding module uses learnable skill tokens to interact with multimodal inputs and extract skill-specific information; (2) The generation phase, in which a skill-constrained diffusion model generates object-centric motion flow conditioned on the skill-aware token sequences and is optimized using a denoising loss and two skill-specific auxiliary losses; and (3) The execution phase, which employs a skill-retrieval transformation strategy to translate the generated 2D motion flow into executable 3D trajectories by leveraging skill-specific priors.
Fig. 2: Architecture of the skill-aware encoding module. The skill-aware encoding module integrates image, language, and bounding boxes of relevant objects with learnable skill tokens through attention-based interactions, producing skill-aware token sequences.
Fig. 3: Overview of the skill-constrained diffusion model. The diffusion model generates motion flow by jointly optimizing skill classification loss, skill contrastive loss, and denoising loss to ensure accurate skill selection, semantic alignment, and precise flow generation.
Fig. 4: Overview of the constructed IsaacSkill dataset. The dataset comprises five fundamental manipulation skills: “ Pouring ”, “ Picking & Placing ”, “ Pushing ”, “ Slide Opening ”, and “ Hinge Opening ”. Each skill includes three different tasks involving different objects.
Fig. 5: Overview of the generalization evaluation settings. Left: The original collected demonstration. Right: The model is evaluated under four different types of variations involving distribution shifts in backgrounds, intra-category instances, cross-category objects, and embodiment.
MethodPouringPicking& PlacingPushingSlide OpeningHinge OpeningAverage SR
R3M [ 10 ]64.064.052.060.060.060.0
AVDC [ 17 ]64.068.084.092.072.076.0
Track2Act [ 14 ]88.080.076.080.088.082.4
Im2Flow2Act [ 15 ]84.092.088.092.084.088.0
SADiff (Ours)92.096.092.096.088.092.8
TABLE I: Comparison of Success Rates (%) Across Different Skills in Within-Distribution Comparative Experiment
Fig. 6: Average success rate (%) of baselines and SADiff under different generalization scenarios. SADiff consistently achieves the highest average success rate across all four generalization settings.
Fig. 7: Examples of the instruction-guided adaptation scenario. The model is evaluated on its ability to switch manipulation skills within an unchanged visual environment solely by following modified language instructions.
MethodTask 1Task 2Task 3Task 4Average SR
R3M [ 10 ]0.00.00.00.00.0
AVDC [ 17 ]0.00.00.00.00.0
Track2Act [ 14 ]44.052.032.036.041.0
Im2Flow2Act [ 15 ]20.016.012.016.016.0
SADiff (Ours)88.084.088.080.085.0
TABLE II: Comparison of Success Rates (%) of Instruction-Guided Skill Adaptation Experiment
Fig. 8: Visualization of real-world experiments. This figure demonstrates the predicted object-centric motion flow and the corresponding execution trajectories for five manipulation skills in the real world.
MethodBackground GeneralizationInstance GeneralizationCross-Category GeneralizationAverage SR
SADiff89.686.482.486.1
w/o LST84.874.468.075.7
w/o SCL86.481.675.281.1
w/o STP73.662.464.866.9
TABLE III: Comparison of Success Rates (%) of Ablation Studies
Fig. 9: Qualitative evaluation under environmental variations. We subject the model to a variety of test scenarios featuring diverse backgrounds, complex lighting conditions, and unseen object instances to assess its visual robustness.
MethodPouringPicking& PlacingPushingSlide OpeningHinge OpeningAverage SR
Track2Act [ 14 ]44.052.056.048.052.050.4
Im2Flow2Act [ 15 ]56.060.056.060.040.054.4
SADiff (Ours)72.080.080.076.072.076.0
TABLE IV: Comparison of Success Rates (%) Across Different Skills in Real-World Experiment
PhasesPouringPicking& PlacingPushingSlide OpeningHinge Opening
Planning Time (s)18.423.117.216.917.8
Execution Time (s)28.221.910.614.515.2
TABLE V: Planning and Execution Time for Five Skills in Real-World Experiments
Fig. 10: Visualization of the scalability of SADiff to unseen skills. We show that SADiff can successfully perform new skills (e.g., “ Stacking ” and “ Wiping ”) unseen during training in both simulated and real-world environments.
Fig. 11: Visualization of the composability of SADiff for long-horizon tasks. We assess the model’s capability to execute multi-stage manipulation sequences by decomposing complex instructions into sequential subtasks.
Figure 17
Figure 18
Figure 19
Figure 20
Figure 21
Figure 22
More in this topic

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

Explore similar work

CardsList

  1. DROM: A Language-Guided Diffusion Framework for Multi-Skill Robotic Manipulation

    Sep 29, 2026Vincenzo Pomponi, Rocco Felici, Paolo Franceschi +5Long-Horizon Robotic ManipulationRobot Skill Learning

    Institute of Systems and Technologies for Sustainable Production (ISTePS), Department of Innovative Technologies, University of Applied Science and Arts of Southern Switzerland (SUPSI), Via la Santa 1, Lugano, CH-6900, Ticino, Switzerland. · Istituto Dalle Molle di studi sull’intelligenza artificiale (IDSIA), Department of Innovative Technologies, University of Applied Science and Arts of Southern Switzerland (SUPSI), Via la Santa 1, Lugano, CH-6900, Ticino, Switzerland. · Mechanical Department, Politecnico di Milano (PoliMi), Via Giuseppe Candiani, Milan, 20155, Italy. +1

  2. RoboReact: Agentic Skill Distillation from Generated Egocentric Videos for Generalizable Whole-Body Manipulation

    Aug 4, 2026Shuliang He, Shuai Wang, Bo Yue +3Robot Skill LearningGeneralization in Robotic Manipulation

    The Chinese University of Hong Kong, Shenzhen · JD Technology · Tsinghua University

  3. Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation

    May 2, 2026Xitie Zhang, Aming Wu, Yahong HanICL for RoboticsZero-Shot Robotic Manipulation

    School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China