Robot Skill Learning

Ask about this topic

Overview

Robot skill learning asks how machines can acquire reusable behaviors—such as grasping, insertion, locomotion, and tool use—and apply them in new situations. Common approaches include imitation learning from robot or human demonstrations, reinforcement learning in simulation and on hardware, and learned motion primitives or diffusion and vision-language-action policies. Researchers organize behaviors into skill libraries, hierarchies, or programs so robots can compose them for longer tasks, while using tactile feedback, language, and human corrections to adapt execution. A central challenge is making skills transfer reliably across objects, scenes, and robot bodies without losing earlier abilities; success on individual skills still does not guarantee robust long-horizon composition.

Latest papers

CardsList

323 papers48 papers in the last four weeks, up 92% on the four weeks before. 0.3% of all new papers.

Mar 3, 2026

Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulation

While skill-centric approaches leverage foundation models to enhance generalization in compositional tasks, they often rely on fixed skill libraries, limiting adaptability to new tasks without manual intervention. To address this, we propose Uni-Skill, a Unified Skill-centric framework that supports skill-aware planning and facilitates automatic skill evolution. Unlike prior methods that restrict planning to predefined skills, Uni-Skill requests for new skill implementations when existing ones are insufficient, ensuring adaptable planning with self-augmented skill library. To support automatic implementation of diverse skills requested by the planning module, we construct SkillFolder, a VerbNet-inspired repository derived from large-scale unstructured robotic videos. SkillFolder introduces a hierarchical skill taxonomy that captures diverse skill descriptions at multiple levels of abstraction. By populating this taxonomy with large-scale, automatically annotated demonstrations, Uni-Skill shifts the paradigm of skill acquisition from inefficient manual annotation to efficient offline structural retrieval. Retrieved examples provide semantic supervision over behavior patterns and fine-grained references for spatial trajectories, enabling few-shot skill inference without deployment-time demonstrations. Comprehensive experiments in both simulation and real-world settings verify the state-of-the-art performance of Uni-Skill over existing VLM-based skill-centric approaches, highlighting its advanced reasoning capabilities and strong zero-shot generalization across a wide range of novel tasks.

Senwei Xie, Yuntian Zhang, Ruiping Wang +1

Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China · University of Chinese Academy of Sciences, Beijing, 100049, China

Robot Skill LearningRobot LearningGeneralization in Robotic ManipulationLarge Language Model-Based Robot PlanningVLMs for Robotics
AskPDF
Mar 2, 2026

Towards Robot Skill Learning and Adaptation with Gaussian Processes

General robot skill adaptation requires expressive representations robust to varying task configurations. While recent learning-based skill adaptation methods refined via Reinforcement Learning (RL), have shown success, existing skill models often lack sufficient representational capacity for anything beyond minor environmental changes. In contrast, Gaussian Process (GP)-based skill modelling provides an expressive representation with useful analytical properties; however, adaptation of GP-based skills remains underexplored. This paper proposes a novel, robust skill adaptation framework that utilises GPs with sparse via-points for compact and expressive modelling. The model considers the trajectory's poses and leverages its first and second analytical derivatives to preserve the skill's kinematic profile. We present three adaptation methods to cater for the variability between initial and observed configurations. Firstly, an optimisation agent that adjusts the path's via-points while preserving the demonstration velocity. Second, a behaviour cloning agent trained to replicate output trajectories from the optimisation agent. Lastly, an RL agent that has learnt to modify via-points whilst maintaining the kinematic profile and enabling online capabilities. Evaluated across three tasks (drawer opening, cube-pushing and bar manipulation) in both simulation and hardware, our proposed methods outperform every benchmark in success rates. Furthermore, the results demonstrate that the GP-based representation enables all three methods to attain high cosine similarity and low velocity magnitude errors, indicating strong preservation of the kinematic profile. Overall, our formulation provides a compact representation capable of adapting to large deviations from a single demonstrated skill.

A K M Nadimul Haque, Fouad Sukkar, Sheila Sutjipto +3

Robot Skill LearningRobot LearningRobotic ManipulationTrajectory Representation Learning
AskPDF
Feb 18, 2026

SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation

The ability to manipulate tools significantly expands the set of tasks a robot can perform. Yet, tool manipulation represents a challenging class of dexterity, requiring grasping thin objects, in-hand object rotations, and forceful interactions. Since collecting teleoperation data for these behaviors is challenging, sim-to-real reinforcement learning (RL) is a promising alternative. However, prior approaches typically require substantial engineering effort to model objects and tune reward functions for each task. In this work, we propose SimToolReal, taking a step towards generalizing sim-to-real RL policies for tool manipulation. Instead of focusing on a single object and task, we procedurally generate a large variety of tool-like object primitives in simulation and train a single RL policy with the universal goal of manipulating each object to random goal poses. This approach enables SimToolReal to perform general dexterous tool manipulation at test-time without any object or task-specific training. We demonstrate that SimToolReal outperforms prior retargeting and fixed-grasp methods by 37% while matching the performance of specialist RL policies trained on specific target objects and tasks. Finally, we show that SimToolReal generalizes across a diverse set of everyday tools, achieving strong zero-shot performance over 120 real-world rollouts spanning 24 tasks, 12 object instances, and 6 tool categories.

Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg +1

Cornell University · Stanford University

Zero-Shot Robotic ManipulationRobot Skill LearningSim-to-Real TransferGeneralization in Robotic ManipulationRobotic Tool UseRobotic Manipulation+1
AskPDF
Feb 17, 2026

Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation

Learning generalist policies capable of accomplishing a wide range of everyday tasks remains an open challenge in dexterous manipulation. In particular, collecting large-scale manipulation data via real-world teleoperation is expensive and difficult to scale. While learning in simulation provides a feasible alternative, designing multiple task-specific environments and rewards for training is similarly challenging. We propose Dex4D, a framework that instead leverages simulation for learning task-agnostic dexterous skills that can be flexibly recomposed to perform diverse real-world manipulation tasks. Specifically, Dex4D learns a domain-agnostic 3D point track conditioned policy capable of transforming a rigid object from a current pose to a target pose. We train this 'Anypose-to-Anypose' policy in simulation across thousands of objects with diverse pose configurations, covering a broad space of robot-object interactions that can be composed at test time. At deployment, this policy can be zero-shot transferred to real-world tasks without finetuning, simply by prompting it with desired object-centric point tracks extracted from generated videos. During execution, Dex4D uses online point tracking for closed-loop perception and control. Extensive experiments in simulation and on real robots show that our method enables zero-shot deployment for diverse dexterous manipulation tasks and yields consistent improvements over prior baselines. Furthermore, we demonstrate strong generalization to novel objects, scene layouts, backgrounds, and trajectories, highlighting the robustness and scalability of the proposed framework. Project page: https://dex4d.github.io/.

Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki +1

Carnegie Mellon University

Robotic ManipulationRobot Skill LearningZero-Shot Robotic ManipulationSim-to-Real Transfer
AskPDF
Feb 17, 2026

Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching

While recent advances in humanoid locomotion have achieved stable walking on varied terrains, capturing the agility and adaptivity of highly dynamic human motions remains an open challenge. In particular, agile parkour in complex environments demands not only low-level robustness, but also human-like motion expressiveness, long-horizon skill composition, and perception-driven decision-making. In this paper, we present Perceptive Humanoid Parkour (PHP), a modular framework that enables humanoid robots to autonomously perform long-horizon, vision-based parkour across challenging obstacle courses. Our approach first leverages motion matching, formulated as nearest-neighbor search in a feature space, to compose retargeted atomic human skills into long-horizon kinematic trajectories. This framework enables the flexible composition and smooth transition of complex skill chains while preserving the elegance and fluidity of dynamic human motions. Next, we train motion-tracking reinforcement learning (RL) expert policies for these composed motions, and distill them into a single depth-based, multi-skill student policy, using a combination of DAgger and RL. Crucially, the combination of perception and skill composition enables autonomous, context-aware decision-making: using only onboard depth sensing and a discrete 2D velocity command, the robot selects and executes whether to step over, climb onto, vault or roll off obstacles of varying geometries and heights. We validate our framework with extensive real-world experiments on a Unitree G1 humanoid robot, demonstrating highly dynamic parkour skills such as climbing tall obstacles up to 1.25m (96% robot height), as well as long-horizon multi-obstacle traversal with closed-loop adaptation to real-time obstacle perturbations.

Zhen Wu, Xiaoyu Huang, Lujie Yang +8

Amazon FAR · UC Berkeley · CMU +1

Humanoid Robot LocomotionRobot Skill Learning
AskPDF
Feb 16, 2026

Simulation-based Learning of Electrical Cabinet Assembly Using Robot Skills

This paper presents a simulation-driven approach for automating the force-controlled assembly of electrical terminals on DIN-rails, a task traditionally hindered by high programming effort and product variability. The proposed method integrates deep reinforcement learning (DRL) with parameterizable robot skills in a physics-based simulation environment. To realistically model the snap-fit assembly process, we develop and evaluate two types of joining models: analytical models based on beam theory and rigid-body models implemented in the MuJoCo physics engine. These models enable accurate simulation of interaction forces, essential for training DRL agents. The robot skills are structured using the pitasc framework, allowing modular, reusable control strategies. Training is conducted in simulation using Soft Actor-Critic (SAC) and Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithms. Domain randomization is applied to improve robustness. The trained policies are transferred to a physical UR10e robot system without additional tuning. Experimental results demonstrate high success rates (up to 100%) in both simulation and real-world settings, even under significant positional and rotational deviations. The system generalizes well to new terminal types and positions, significantly reducing manual programming effort. This work highlights the potential of combining simulation-based learning with modular robot skills for flexible, scalable automation in small-batch manufacturing. Future work will explore hybrid learning methods, automated environment parameterization, and further refinement of joining models for design integration.

Arik Laemmle, Balázs András Bálint, Philipp Tenbrock +4

formerly Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Stuttgart, Germany · formerly Institut für Strahlwerkzeuge IFSW, University of Stuttgart, Stuttgart, Germany · Research Laboratory on Engineering and Management Intelligence, Institute for Computer Science and Control SZTAKI, Budapest, Hungary +1

Robotic AssemblyRobot Skill LearningForce-Controlled Robotic ManipulationRobotic RLSim-to-Real Transfer
AskPDF
Feb 16, 2026

TWISTED-RL: Hierarchical Skilled Agents for Knot-Tying without Human Demonstrations

Robotic knot-tying represents a fundamental challenge in robotics due to the complex interactions between deformable objects and strict topological constraints. We present TWISTED-RL, a framework that improves upon the previous state-of-the-art in demonstration-free knot-tying (TWISTED), which smartly decomposed a single knot-tying problem into manageable subproblems, each addressed by a specialized agent. Our approach replaces TWISTED's single-step inverse model that was learned via supervised learning with a multi-step Reinforcement Learning policy conditioned on abstract topological actions rather than goal states. This change allows more delicate topological state transitions while avoiding costly and ineffective data collection protocols, thus enabling better generalization across diverse knot configurations. Experimental results demonstrate that TWISTED-RL manages to solve previously unattainable knots of higher complexity, including commonly used knots such as the Figure-8 and the Overhand. Furthermore, the increase in success rates and drop in planning time establishes TWISTED-RL as the new state-of-the-art in robotic knot-tying without human demonstrations.

Guy Freund, Tom Jurgenson, Matan Sudry +1

Reichman University · Technion – Israel Institute of Technology · The University of Texas at Austin

Robotic ManipulationRobot Skill LearningDeformable Object ManipulationRobotic RLHierarchical RL
AskPDF
Feb 2, 2026

HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos

Enabling humanoid robots to perform agile and adaptive interactions has long been a core challenge in robotics. Current approaches are bottlenecked by either the scarcity of realistic interaction data or the need for meticulous, task-specific reward engineering, which limits their scalability. To narrow this gap, we present HumanX, a unified framework that compiles human video into generalizable, real-world interaction skills for humanoids, without per-task design. HumanX integrates two co-designed components: XGen, an anchor-centric data generation pipeline that yields physically plausible robot interaction data from monocular video and augments it to create a diverse, scalable training dataset; and XMimic, an interaction-prioritized learning framework to achieve generalizable interaction skills from seconds of demonstration. Evaluated across five distinct domains--basketball, football, badminton, cargo pickup, and reactive fighting--HumanX successfully acquires 9 different skills and transfers them zero-shot to a physical Unitree G1 humanoid. The learned capabilities include complex maneuvers such as pump-fake turnaround fadeaway jumpshots without any external perception, as well as interactive tasks like sustained human-robot passing sequences over 10 consecutive cycles--learned from a single video demonstration. Our experiments show that HumanX achieves higher generalization success than prior methods, demonstrating a scalable pathway within a class of anchorable interactions for learning versatile, real-world robot interactive skills.

Yinhuai Wang, Qihan Zhao, Yuen Fui Lau +6

The Hong Kong University of Science and Technology · Shanghai AI Laboratory

Robot Skill LearningHumanoid RoboticsRobot Imitation LearningHuman-Robot InteractionSynthetic Data Generation for Robotics
AskPDF
Jan 27, 2026

Sim-and-Human Co-training for Data-Efficient and Scene-Generalizable Bimanual Manipulation

Real-robot demonstrations are prohibitively expensive, while simulation data and real-world human demonstrations are both scalable but each leaves a distinct gap: simulation suffers from a sim-to-real visual gap, and human data suffers from a human-to-robot embodiment gap. In this work, we identify a natural yet underexplored complementarity between these sources: simulation contributes robot-valid actions absent in human data, while human data provides real-world observations that simulation struggles to render. Building on this insight, we present SimHum, a co-training recipe that extracts kinematic priors from simulation and visual priors from human observations, then fine-tunes on a small real-robot dataset. SimHum exhibits strong scene-generalizable and data-efficient capabilities. With only 80 real-robot episodes per task, it achieves 62.5% success on held-out OOD scenes across four bimanual tabletop tasks, 53.7% higher than Real only in absolute success rate. Moreover, in a controlled data-collection study with matched collection time, SimHum improves over the best single-source pre-training baseline by 35.0% in absolute success rate. Project page: https://kaipengfang.github.io/sim-and-human/

Kaipeng Fang, Weiqing Liang, Yuyang Li +5

University of Electronic Science and Technology of China · Southwest Jiaotong University · Tongji University +1

Robot Skill LearningBimanual Robotic ManipulationSim-to-Real TransferGeneralization in Robotic ManipulationCross-Embodiment Robot LearningData-Efficient Robot Learning+1
AskPDF
Jan 16, 2026

Skill-Aware Diffusion for Generalizable Robotic Manipulation

Robust generalization in robotic manipulation is crucial for robots to adapt flexibly to diverse environments. Existing methods usually improve generalization by scaling data and networks, but model tasks independently and overlook skill-level information. Observing that tasks within the same skill share similar motion patterns, we propose Skill-Aware Diffusion (SADiff), which explicitly incorporates skill-level information to improve generalization. SADiff learns skill-specific representations through a skill-aware encoding module with learnable skill tokens, and conditions a skill-constrained diffusion model to generate object-centric motion flow. A skill-retrieval transformation strategy further exploits skill-specific trajectory priors to refine the mapping from 2D motion flow to executable 3D actions. Furthermore, we introduce IsaacSkill, a high-fidelity dataset containing fundamental robotic skills for comprehensive evaluation and sim-to-real transfer. Experiments in simulation and real-world settings show that SADiff achieves good performance and generalization across various manipulation tasks. Dataset and videos are available at https://sites.google.com/view/sa-diff.

Aoshen Huang, Jiaming Chen, Jiyu Cheng +3

School of Control Science and Engineering, Shandong University, Jinan 250061, China. · Department of Computer Science, The University of Manchester, M13 9PL Manchester, U.K.

Robotic ManipulationGeneralization in Robotic ManipulationRobot Skill Learning
AskPDF
Dec 30, 2025

World In Your Hands: A Large-Scale and Open-Source Ecosystem for Learning Human-Centric Manipulation in the Wild

We introduce World In Your Hands (WIYH), a large-scale open-source ecosystem comprising over 1,000 hours of human manipulation data collected in-the-wild with millimeter-scale motion accuracy. Specifically, WIYH includes (1) the Oracle Suite, a wearable data collection kit with an auto-labeling pipeline for accurate motion capture; (2) the WIYH Dataset, featuring over 1,000 hours of multimodal manipulation data across hundreds of skills in diverse real-world scenarios; and (3) extensive annotations and benchmarks supporting tasks from perception to action. Furthermore, experiments based on the WIYH ecosystem show that integrating WIYH's human-centric data improves robotic manipulation success rates from 8% to 60% in cluttered scenes. World In Your Hands provides a foundation for advancing human-centric data collection and cross-embodiment policy learning. All data and hardware design will be open-source.

Yupeng Zheng, Jichao Peng, Weize Li +22

TARS Robotics

Human-Object InteractionRobot Skill LearningRobotic Data CollectionCross-Embodiment Robot LearningRobotic ManipulationRobot Manipulation Benchmarks+2
AskPDF
Dec 20, 2025

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation

Scaling imitation learning to diverse multi-task robot manipulation remains challenging due to suboptimal demonstrations, behavioral multi-modality, and destructive interference across tasks. While skill-based methods offer a promising direction by decomposing behaviors into reusable abstractions, existing approaches often learn skills that are either biased toward linguistic structure or lack semantic alignment across tasks, limiting generalization. In this work, we propose AtomSkill, a novel framework that learns a semantically aligned Atomic Skill Space from demonstrations and enables robust long-horizon execution through keypose imagination. Our method introduces: (1) semantic contrastive skill alignment, which partitions demonstrations into variable-length atomic skills and employs a contrastive objective to jointly enforce semantic consistency and temporal coherence, yielding a compact and reusable skill library; and (2) action decoding with keypose imagining, where the policy predicts both a skill's terminal keypose and immediate actions, thereby supporting progress-aware skill transitions. During inference, an atomic skill diffusion sampler generates plausible skill sequences, while predicted keyposes autonomously trigger smooth skill chaining. Extensive experiments in simulation and real-world settings show that AtomSkill consistently outperforms state-of-the-art imitation learning and skill-based baselines. Project page: https://atom-skill.github.io.

Yihang Zhu, Weiqing Wang, Shijie Wu +2

ShanghaiTech University, Shanghai, China · InstAdapt

Long-Horizon Robotic ManipulationRobot Skill LearningGeneralization in Robotic ManipulationMulti-Task Robotic ManipulationRobotic ManipulationRobot Imitation Learning+1
AskPDF
Dec 17, 2025

ISS Policy : Scalable Diffusion Policy with Implicit Scene Supervision

Vision-based imitation learning has enabled impressive robotic manipulation skills, but action imitation alone provides limited supervision of the geometric consequences of robot behavior. To address this limitation, we introduce Implicit Scene Supervision (ISS) Policy, a 3D visuomotor diffusion policy with a DiT backbone that predicts continuous action sequences from point-cloud observations. ISS augments action diffusion with a supervised robot motion predictor that maps generated actions and robot-state context to end-effector motion, and then uses the predicted motion together with gripper intent to forecast future point-cloud representations. By explicitly modeling the intermediate transition from action to robot motion, ISS encourages the policy to capture how its actions affect the surrounding 3D scene. We further introduce asymmetric gradient routing to separate direct motion regression from scene-level policy supervision, together with a change-balanced objective that accounts for variations in scene-change magnitude. These auxiliary objectives provide dynamics-aware geometric supervision using only expert demonstrations, without requiring additional annotations or auxiliary modules at inference time. ISS Policy achieves state-of-the-art performance on single-arm manipulation tasks in MetaWorld and dexterous manipulation tasks in Adroit, while real-world dual-arm experiments further demonstrate its effectiveness on physical robotic manipulation. The resulting framework preserves the scalable DiT backbone and standard diffusion-policy control interface. Code and videos will be released.

Wenlong Xia, Jinhao Zhang, Ce Zhang +6

[cs.RO]

Robot Skill LearningVisuomotor Policy LearningDiffusion PolicyRobot Motion PredictionRobotic ManipulationRobot Imitation Learning
AskPDF
Dec 11, 2025

Learning Category-level Last-meter Navigation from RGB Demonstrations of a Single-instance

Achieving precise positioning of the mobile manipulator's base is essential for successful manipulation actions that follow. Most of the RGB-based navigation systems only guarantee coarse, meter-level accuracy, making them less suitable for the precise positioning phase of mobile manipulation. This gap prevents manipulation policies from operating within the distribution of their training demonstrations, resulting in frequent execution failures. We address this gap by introducing an object-centric imitation learning framework for last-meter navigation, enabling a quadruped mobile manipulator robot to achieve manipulation-ready positioning using only RGB observations from its onboard cameras. Our method conditions the navigation policy on three inputs: goal images, multi-view RGB observations from the onboard cameras, and a text prompt specifying the target object. A language-driven segmentation module and a spatial score-matrix decoder then supply explicit object grounding and relative pose reasoning. Using real-world data from a single object instance within a category, the system generalizes to unseen object instances across diverse environments with challenging lighting and background conditions. To comprehensively evaluate this, we introduce two metrics: an edge-alignment metric, which uses ground truth orientation, and an object-alignment metric, which evaluates how well the robot visually faces the target. Under these metrics, our policy achieves 74.58% success in edge-alignment and 89.42% success in object-alignment when positioning relative to unseen target objects. These results show that precise last-meter navigation can be achieved at a category-level without depth, LiDAR, or map priors, enabling a scalable pathway toward unified mobile manipulation. Project page: https://rpm-lab-umn.github.io/category-level-last-meter-nav/

Tzu-Hsien Lee, Fidan Mahmudova, Karthik Desingh

University of Minnesota, Twin Cities

Robot Skill LearningRobot NavigationRobot Imitation LearningObject-Goal Navigation
AskPDF
Nov 22, 2025

Continually Evolving Skill Knowledge in Vision Language Action Model

Vision-language-action (VLA) models show promising knowledge accumulation ability through pretraining, yet continual learning remains challenging. Existing continual imitation learning (CIL) methods often require additional parameters or modules, limiting scalability for large VLA models. We propose Stellar VLA, a knowledge-driven CIL framework without parameter growth, featuring two variants: T-Stellar for task-centric modeling and TS-Stellar for hierarchical modeling of reusable skills across tasks. Stellar VLA jointly optimizes task representations and a self-evolving knowledge space, with knowledge-posterior-guided expert routing using knowledge relations and Top-K semantic embeddings for task specialization. Experiments on LIBERO demonstrate strong performance against VLA and CIL baselines with only 1% replay. Dual-arm real-world experiments validate knowledge transfer across embodiments and scenes. TS-Stellar excels in hierarchical manipulation, while visualizations demonstrate knowledge retention and task discovery. Project Website: https://stellarvla.github.io/

Yuxuan Wu, Guangming Wang, Zhiheng Yang +4

Shanghai Jiao Tong University · Shanghai Innovation Institute · University of Cambridge +4

Continual Learning for VLA ModelsContinual Robot LearningVision-Language-Action ModelsRobot Skill LearningInteractive Imitation Learning
AskPDF
Nov 4, 2025

XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations

Recent progress in large-scale robotic datasets and vision-language models (VLMs) has advanced research on vision-language-action (VLA) models. However, existing VLA models still face two fundamental challenges: (i) producing precise low-level actions from high-dimensional observations, (ii) bridging domain gaps across heterogeneous data sources, including diverse robot embodiments and human demonstrations. Existing methods often encode latent variables from either visual dynamics or robotic actions to guide policy learning, but they fail to fully exploit the complementary multi-modal knowledge present in large-scale, heterogeneous datasets. In this work, we present X Robotic Model 1 (XR-1), a novel framework for versatile and scalable VLA learning across diverse robots, tasks, and environments. XR-1 introduces the Unified Vision-Motion Codes (UVMC), a discrete latent representation learned via a dual-branch VQ-VAE that jointly encodes visual dynamics and robotic motion. UVMC addresses these challenges by (i) serving as an intermediate representation between the observations and actions, and (ii) aligning multimodal dynamic information from heterogeneous data sources to capture complementary knowledge. To effectively exploit UVMC, we propose a three-stage training paradigm: (i) self-supervised UVMC learning, (ii) UVMC-guided pretraining on large-scale cross-embodiment robotic datasets, and (iii) task-specific post-training. We validate XR-1 through extensive real-world experiments with more than 14,000 rollouts on six different robot embodiments, spanning over 120 diverse manipulation tasks. XR-1 consistently outperforms state-of-the-art baselines such as π0.5π_{0.5}, π0π_0, RDT, UniVLA, and GR00T-N1.5 while demonstrating strong generalization to novel objects, background variations, distractors, and illumination changes. Our project is at https://xr-1-vla.github.io/.

Shichao Fan, Kun Wu, Zhengping Che +12

Beijing Innovation Center of Humanoid Robotics, Beijing, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China · State Key Laboratory of Virtual Reality Technology and Systems, SCSE, Beihang University, Beijing, China +1

Representation LearningRobot Skill LearningCross-Embodiment Robot LearningRobotic ManipulationVision-Language-Action Models
AskPDF
Sep 23, 2025

Robotic Ultra-Long-Horizon Manipulation Skills via Human-guided Lifelong Code Generation

Large language models (LLMs) can translate natural-language instructions for robotic manipulation into executable code, but ambiguity, noisy generations, and limited context windows make ultra-long-horizon tasks unreliable. Closed-loop approaches that rely only on LLM feedback also struggle because LLMs have limited robotic reasoning, even when task errors are obvious to humans. Feedback is often stored in representations that generalize poorly to unseen tasks and can cause catastrophic forgetting as new corrections accumulate. We propose LYRA, a human-guided lifelong skill learning and code generation framework that distills human feedback into modular, reusable skills and incrementally extends their functionality across successive interactions while preserving previously learned behavior. External memory stores learned skills and execution examples; retrieval-augmented generation selects relevant knowledge, while user hints guide reuse when retrieval is insufficient, supporting ultra-long-horizon execution. Experiments on Ravens, Franka Kitchen, LIBERO-long, MetaWorld, and real-world tasks show a 0.93 success rate, up to 27% higher than baselines, and a 42% improvement in correction efficiency. LYRA also robustly solves ``build a house'', which requires planning over 20 primitives.

Yuan Meng, Zhenguo Sun, Max Fest +4

School of Computation, Information and Technology Technical University of Munich Munich, Germany · Beijing Academy of Artificial Intelligence (BAAI) Beijing, China · The State Key Laboratory for Novel Software Technology Nanjing University Suzhou, China

Code Generation for RoboticsLong-Horizon Robotic ManipulationRobot Skill LearningRobotic ManipulationContinual Robot LearningHuman-in-the-Loop AI
AskPDF
Sep 22, 2025

GeCCo -- a Generalist Contact-Conditioned Policy for Loco-Manipulation Skills on Legged Robots

Most modern approaches to quadruped locomotion focus on using Deep Reinforcement Learning (DRL) to learn policies from scratch, in an end-to-end manner. Such methods often fail to scale, as every new problem or application requires time-consuming and iterative reward definition and tuning. We present Generalist Contact-Conditioned Policy (GeCCo) --- a low-level policy trained with Deep Reinforcement Learning that is capable of tracking arbitrary contact points with a quadruped robot. We shift from task-specific end-to-end learning to a modular hierarchy in which a single planner-agnostic, contact-conditioned tracking policy serves as a robust interface between planning and control. By maintaining stable contact execution under dynamic uncertainty and planner mismatch, the policy enables high-level planners (e.g., handcrafted, learned, or optimization-based) to be composed and swapped without retraining. We demonstrate the scalability and robustness of our method by evaluating on a wide range of locomotion and manipulation tasks in a common framework and under a single generalist policy. These include a variety of gaits, traversing complex terrains (e.g., stairs and slopes) as well as previously unseen stepping-stones and narrow beams, and interacting with objects (e.g., pressing a button, pushing a barrel). Our framework acquires new behaviors more efficiently, simply by combining a task-specific high-level contact planner and the pre-trained generalist policy. Project website: https://vassil-atn.github.io/gecco.github.io

Vassil Atanassov, Wanming Yu, Siddhant Gangapurwala +2

Dynamic Robot Systems Group, Oxford Robotics Institute, University of Oxford · Sony AI · Dyson Institute of Engineering and Technology

Robotic ControlLegged RoboticsRobot Policy LearningRobot Skill LearningRobotic ManipulationRL for Legged Locomotion+1
AskPDF
Sep 21, 2025

Imagine2Act: Leveraging Object-Action Motion Consistency from Imagined Goals for Robotic Manipulation

Relational object rearrangement (ROR) tasks (e.g., insert flower to vase) require a robot to manipulate objects with precise semantic and geometric reasoning. Existing approaches either rely on pre-collected demonstrations that struggle to capture complex geometric constraints or generate goal-state observations to capture semantic and geometric knowledge, but fail to explicitly couple object transformation with action prediction, resulting in errors due to generative noise. To address these limitations, we propose Imagine2Act, a 3D imitation-learning framework that incorporates semantic and geometric constraints of objects into policy learning to tackle high-precision manipulation tasks. We first generate imagined goal images conditioned on language instructions and reconstruct corresponding 3D point clouds to provide robust semantic and geometric priors. These imagined goal point clouds serve as additional inputs to the policy model, while an object-action consistency strategy with soft pose supervision explicitly aligns predicted end-effector motion with generated object transformation. This design enables Imagine2Act to reason about semantic and geometric relationships between objects and predict accurate actions across diverse tasks. Experiments in both simulation and the real world demonstrate that Imagine2Act outperforms previous state-of-the-art policies. More visualizations can be found at https://sites.google.com/view/imagine2act.

Liang Heng, Jiadong Xu, Yiwen Wang +6

CFCS, School of Computer Science, Peking University · PrimeBot · AGIBOT

Robot Skill LearningLanguage-Conditioned Robot ManipulationGoal-Conditioned Robotic ManipulationRobotic ManipulationRobot Imitation Learning
AskPDF
Aug 28, 2025

Physics-Guided Residual Reinforcement Learning for Humanoid Narrow-Path Traversal

Traversing narrow paths is challenging for humanoid robots due to the sparse and safety-critical footholds required. Purely template-based or end-to-end reinforcement learning-based methods suffer from such harsh terrains. This paper proposes a two stage training framework for such narrow path traversing tasks, coupling a template-based foothold planner with a low-level foothold tracker from Stage-I training and a lightweight perception aided foothold modifier from Stage-II training. With the curriculum setup from flat ground to narrow paths across stages, the resulted controller in turn learns to robustly track and safely modify foothold targets to ensure precise foot placement over narrow paths. This framework preserves the interpretability from the physics-based template and takes advantage of the generalization capability from reinforcement learning, resulting in easy sim-to-real transfer. The learned policies outperform purely template-based or reinforcement learning-based baselines in terms of success rate, centerline adherence and safety margins. Validation on a Unitree G1 humanoid robot yields successful traversal of a 0.2m wide and 3m long beam for 20 trials without any failure.

Tianchen Huang, Sisheng Chen, Wei Zhou +8

Institute of Humanoid Robots, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei, Anhui 230026, China

Humanoid Robot LocomotionRobot Skill LearningRobot LocomotionRL for Legged LocomotionResidual Policy Learning
AskPDF
Aug 12, 2025

Rational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planning

Humans can learn a new manipulation task from one or two demonstrations and then perform it in a new room, with new objects, under new constraints. Modern robot imitation learning, in contrast, typically needs hundreds to thousands of demonstrations and still degrades under modest shifts in layout, geometry, object set or task constraints. We argue this gap is not just about data, but also about the level of abstraction at which learning occurs; generalization requires inferring the latent intent underlying why a demonstrator behaved in a certain way, rather than reproducing how they moved. We present Rational Inverse Reasoning (RIR), which casts few-shot imitation as inference over latent explanation programs: compact, executable descriptions of intent that map an object-centric scene to a structured task-and-motion-planning (TAMP) specification of goals, subgoals and constraints. A vision-language model proposes candidate programs, and a hierarchical planner supplies a bounded-rational likelihood. By combining VLM program proposals, and planner-grounded feedback, RIR iteratively refines the candidate set to approximate a posterior over concise, executable programs. On a 2D reasoning benchmark and a real Franka FR3, RIR recovers transferable task structure from as little as one demonstration. Generalizing to substantially new layouts and object sets, RIR outperforms VLM-planning baselines that lack explicit rationality and planning-grounded inference, increasing downstream success rate by 3434 and 2828 percentage points in the one- and three-shot settings.

Ben Zandonati, Tomás Lozano-Pérez, Leslie Pack Kaelbling

MIT CSAIL

Robot Skill LearningGeneralization in Robotic ManipulationProgram SynthesisFew-Shot LearningRobot Imitation LearningRobot Policy Generalization
AskPDF
Jul 15, 2025

Robot Drummer: Learning Rhythmic Skills for Humanoid Drumming

Humanoid robots have seen remarkable advances in dexterity, balance, and locomotion, yet their role in expressive domains such as music performance remains largely unexplored. Musical tasks, like drumming, present unique challenges such as split-second timing, rapid contacts, and multi-limb coordination over performances lasting minutes. In this paper, we introduce Robot Drummer, a simulation framework for humanoid drumming across a diverse repertoire of songs. We formulate humanoid drumming as the realization of timed contact events encoded as a Rhythmic Contact Chain. To handle the long-horizon nature of musical performance, we decompose each track into fixed-length segments and train a single policy across all segments in parallel using reinforcement learning. Through extensive experiments on over thirty popular tracks, our results demonstrate that Robot Drummer consistently achieves high F1 scores and enables efficient learning of long-horizon musical performances. The learned behaviors exhibit emergent human-like drumming strategies, such as cross-arm strikes, and adaptive stick assignments, demonstrating the potential of reinforcement learning to bring humanoid robots into the domain of creative musical performance. Project page: robotdrummer.github.io

Asad Ali Shahid, Francesco Braghin, Loris Roveda

IDSIA, SUPSI, Lugano, Switzerland · Department of Mechanical Engineering, Politecnico di Milano, Italy

Long-Horizon Robotic ManipulationRobot Skill LearningRobotics SimulationHumanoid Robot ControlRobotic RL
AskPDF
May 28, 2025

Prior Reinforce: Goal-Conditioned Dynamic Manipulation with Limited Trials

Embodied robots have achieved strong performance in many real-world manipulation tasks, yet agile dynamic manipulation remains challenging due to high sensitivity to motion parameters and sparse outcome-level feedback. Tasks such as shooting a basketball into a hoop require precise control of fast open-loop motions, where small trajectory variations can lead to large outcome deviations, making data-efficient adaptation difficult for existing methods that rely on large-scale interaction, reward engineering, or accurate dynamic modeling. We propose Prior Reinforce (P.R.), a simple and practical framework for goal-conditioned dynamic manipulation. The method first learns a structured motion manifold from a small set of demonstrations using a conditional diffusion model, and then adapts motions toward new goals through feedback-driven optimization in a low-dimensional condition space. By separating motion generation from outcome-driven adaptation, the framework enables efficient refinement using only a small number of real-world trials under noisy perception. Experiments on multiple real-world dynamic manipulation tasks demonstrate that P.R. reliably achieves new goals within as few as ten total trials while remaining robust to perception noise and hardware uncertainty, suggesting a practical approach for low-trial real-world robot adaptation. Project website: https://adap-robotics.github.io/.

Yihang Hu, Pingyue Sheng, Yuyang Liu +2

IIIS, Tsinghua University. · Shanghai Qi Zhi Institute. · Spirit AI.

Robot Policy AdaptationRobot Skill LearningGoal-Conditioned Robotic ManipulationRobot Motion GenerationRobotic ManipulationDynamic Robotic Manipulation+1
AskPDF