Long-Horizon Robotic Manipulation

Ask about this topic

Overview

Long-horizon robotic manipulation concerns tasks that require many dependent actions, such as assembling furniture, preparing food, or moving objects through changing environments. Common approaches combine hierarchical planning and reusable skills with vision-language-action policies, world models, symbolic task representations, and memory of past actions or object states. Researchers train these systems with demonstrations, reinforcement learning, simulated experience, and human corrections, and evaluate them in multi-stage benchmarks and real-robot settings. A central challenge is that strong performance on individual skills does not guarantee reliable task completion: errors can accumulate at skill transitions, while memory, recovery, and safety remain difficult to evaluate and generalize.

Latest papers

CardsList

435 papers88 papers in the last four weeks, up 49% on the four weeks before. 0.5% of all new papers.

Feb 18, 2026

EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data

Human behavior is among the most scalable sources of data for learning physical intelligence, yet how to effectively leverage it for dexterous manipulation remains unclear. While prior work demonstrates human to robot transfer in constrained settings, it is unclear whether large scale human data can support fine grained, high degree of freedom dexterous manipulation. We present EgoScale, a human to dexterous manipulation transfer framework built on large scale egocentric human data. We train a Vision Language Action (VLA) model on over 20,854 hours of action labeled egocentric human video, more than 20 times larger than prior efforts, and uncover a log linear scaling law between human data scale and validation loss. This validation loss strongly correlates with downstream real robot performance, establishing large scale human data as a predictable supervision source. Beyond scale, we introduce a simple two stage transfer recipe: large scale human pretraining followed by lightweight aligned human robot mid training. This enables strong long horizon dexterous manipulation and one shot task adaptation with minimal robot supervision. Our final policy improves average success rate by 54% over a no pretraining baseline using a 22 DoF dexterous robotic hand, and transfers effectively to robots with lower DoF hands, indicating that large scale human motion provides a reusable, embodiment agnostic motor prior.

Ruijie Zheng, Dantong Niu, Yuqi Xie +12

NVIDIA · University of California, Berkeley · University of Maryland

Robotic ManipulationRobot LearningTransfer LearningLong-Horizon Robotic ManipulationCross-Embodiment Robot LearningData-Efficient Robot Learning
AskPDF
Feb 18, 2026

Articulated 3D Scene Graphs for Open-World Mobile Manipulation

Semantic scene understanding has enabled robots to reason and perform affordance-driven object interaction. Despite this, real-world manipulation remains fundamentally limited by the inability to predict how objects move and are articulated. Long-horizon mobile manipulation requires closing the gap between semantics, geometry, and kinematics. In this work, we present MoMa-SG, a novel framework for building semantic-kinematic 3D scene graphs of articulated scenes containing a myriad of interactable objects. Given RGB-D sequences containing multiple object articulations, we temporally segment object interactions and infer object motion using occlusion-robust point tracking. We lift point trajectories to 3D and estimate articulation models using a novel unified twist estimation formulation that robustly estimates revolute and prismatic joint parameters in a single optimization pass. Next, we associate objects with articulations and detect contained objects by reasoning over parent-child relations at identified opening states. We also introduce the Arti4D-Semantic dataset, which provides parent-child relation annotations between container and contained objects together with object axis annotations across 62 in-the-wild RGB-D sequences containing 600 object interactions and three distinct observation paradigms. We extensively evaluate the performance of MoMa-SG on two datasets and ablate key design choices of our approach. In addition, real-world experiments on both a quadruped and a mobile manipulator demonstrate robust manipulation of articulated objects in everyday home environments. We provide code and data at: https://momasg.cs.uni-freiburg.de.

Martin Büchner, Adrian Röfer, Tim Engelbracht +5

University of Freiburg · ETH Zürich · University of Bonn

3D SGGArticulated Object ManipulationMobile ManipulationArticulated Object ReconstructionLong-Horizon Robotic Manipulation
AskPDF
Feb 14, 2026

Humanoid Hanoi: Investigating Shared Whole-Body Control for Skill-Based Box Rearrangement

We introduce Humanoid Hanoi, a challenge problem for long-horizon humanoid skill composition that requires a robot to repeatedly sequence pickup, locomotion, and placement skills to rearrange stacked boxes in simulation and on hardware. This symbolically simple task exposes a weakness in a common modular humanoid architecture, where reusable high-level skills are learned on top of a shared, task-agnostic whole-body controller (WBC). While such a WBC provides a unified low-level interface for skill composition, it may not be sufficiently hardened to execute the full collection of learned skills under domain randomization and long-horizon closed-loop execution. We compare strategies for addressing this issue, including skill-specific WBC fine-tuning, residual low-level adaptation, and task-driven data aggregation that expands the shared-WBC training distribution with closed-loop trajectories induced by learned skills while preserving its original tracking objective. In our evaluated system, the shared-WBC approach achieves the strongest long-horizon robustness. A direct transition-smoothing intervention further shows that low-level controller handoffs account for a substantial part of the composition loss observed with non-shared controllers. Project page: https://osudrl.github.io/Humanoid_Hanoi/

Minku Kim, Kuan-Chia Chen, Aayam Shrestha +3

The authors are with the Collaborative Robotics and Intelligent Systems Institute, Oregon State University, Corvallis, Oregon, 97331, USA

Humanoid Robot ControlWhole-Body ControlLong-Horizon Robotic ManipulationRobot Manipulation Benchmarks
AskPDF
Feb 13, 2026

UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs

Achieving general-purpose robotic manipulation requires robots to bridge high-level semantic intent with low-level physical interaction in unstructured environments. However, zero-shot generalization remains challenging: end-to-end Vision-Language-Action (VLA) models often lack the precision required for long-horizon tasks, while traditional hierarchical planners suffer from semantic rigidity under open-world variations. We present UniManip, a unified framework centered on an Agentic Operational Graph (AOG), an explicit cognitive and operational backbone for task orchestration, state tracking, execution verification, and closed-loop failure recovery. High-level semantic intents are grounded into actionable primitives via a physically grounded scene graph that provides geometric affordances, contact-aware interaction parameters, and collision-free trajectory synthesis. By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, UniManip continuously aligns abstract planning with geometric constraints, enabling robust execution. Unlike static pipelines, UniManip operates as a dynamic agentic loop: it instantiates object-centric scene graphs from unstructured perception, parameterizes these representations into collision-free trajectories via a safety-aware local planner, and exploits structured memory to autonomously diagnose and recover from execution failures. UniManip achieves state-of-the-art performance on LIBERO-Pro, while extensive real-world experiments demonstrate robust zero-shot generalization across diverse tasks, embodiments, and dexterous hands, outperforming leading VLA models and hierarchical baselines by 22.5% and 25.0% in task success rate, respectively. Open-source project page: https://henryhcliu.github.io/unimanip.

Haichao Liu, Yuanjiang Xue, Yuheng Zhou +5

School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Automation, Tsinghua University, China

Robotic ManipulationLong-Horizon Robotic ManipulationZero-Shot Robotic Manipulation
AskPDF
Feb 11, 2026

H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model

World models are becoming central to robotic planning and control by predicting future state transitions. Existing approaches mainly rely on visual, latent, or language prediction, which can be difficult to ground in executable robot actions and prone to compounding errors over long horizons. In contrast, traditional robotic task and motion planning enables structured long-horizon reasoning through compact symbolic representations of world transitions, but typically lacks synchronized visual prediction. We propose Hierarchical World Model (H-WM), which jointly predicts logical and visual state transitions by combining a high-level logical world model with a low-level visual world model. The predicted logical actions and latent visual state transitions are jointly incorporated into Vision-Language-Action (VLA) models as intermediate state guidance for long-horizon task execution. Experiments on three long-horizon benchmarks and real robots show that H-WM consistently improves VLA's performance by stabilizing long-horizon execution and mitigating error accumulation. We also construct LIBERO-Logic, a frame-level aligned dataset that pairs visual observations and continuous robot states with logical actions and predicate-based logical states.

Jinbang Huang, Wenyuan Chen, Zhiyuan Li +9

Huawei Noah’s Ark Lab · University of Toronto · University of British Columbia +1

Long-Horizon Robotic ManipulationWorld Models for RoboticsWorld ModelsWorld Model-Based PlanningVision-Language-Action ModelsHierarchical World Models+1
AskPDF
Jan 30, 2026

CoLA-Flow Policy: Temporally Coherent Imitation Learning via Continuous Latent Action Flow Matching for Robotic Manipulation

Learning long-horizon robotic manipulation requires jointly achieving expressive behavior modeling, real-time inference, and stable execution, which remains challenging for existing generative policies. Diffusion-based approaches offer strong modeling capacity but incur high inference latency, while flow matching enables fast, near-single-step generation yet often suffers from unstable execution when operating directly in the raw action space. We propose Continuous Latent Action Flow Policy (CoLA-Flow Policy), a trajectory-level imitation learning framework that performs flow matching in a continuous latent action space. By encoding action sequences into temporally coherent latent trajectories and learning an explicit latent-space flow, CoLA-Flow Policy decouples global motion structure from low-level control noise, enabling smooth and reliable long-horizon execution. The framework further integrates geometry-aware point cloud conditioning and execution-time multimodal modulation, using visual cues as a representative modality to enhance real-world robustness. Experiments in simulation and on real robots show that CoLA-Flow Policy achieves near-single-step inference, improves trajectory smoothness by up to 93.7% and task success by up to 25 percentage points over raw action-space flow baselines, while remaining significantly faster than diffusion-based policies.

Wu Songwei, Jiang Zhiduo, Sun Wandong +4

Harbin Institute of Technology, Harbin 150001, China · Honor Device Co., Ltd., Shenzhen, China

Flow MatchingLatent Action LearningLong-Horizon Robotic ManipulationRobot Imitation Learning
AskPDF
Dec 20, 2025

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation

Scaling imitation learning to diverse multi-task robot manipulation remains challenging due to suboptimal demonstrations, behavioral multi-modality, and destructive interference across tasks. While skill-based methods offer a promising direction by decomposing behaviors into reusable abstractions, existing approaches often learn skills that are either biased toward linguistic structure or lack semantic alignment across tasks, limiting generalization. In this work, we propose AtomSkill, a novel framework that learns a semantically aligned Atomic Skill Space from demonstrations and enables robust long-horizon execution through keypose imagination. Our method introduces: (1) semantic contrastive skill alignment, which partitions demonstrations into variable-length atomic skills and employs a contrastive objective to jointly enforce semantic consistency and temporal coherence, yielding a compact and reusable skill library; and (2) action decoding with keypose imagining, where the policy predicts both a skill's terminal keypose and immediate actions, thereby supporting progress-aware skill transitions. During inference, an atomic skill diffusion sampler generates plausible skill sequences, while predicted keyposes autonomously trigger smooth skill chaining. Extensive experiments in simulation and real-world settings show that AtomSkill consistently outperforms state-of-the-art imitation learning and skill-based baselines. Project page: https://atom-skill.github.io.

Yihang Zhu, Weiqing Wang, Shijie Wu +2

ShanghaiTech University, Shanghai, China · InstAdapt

Long-Horizon Robotic ManipulationRobot Skill LearningGeneralization in Robotic ManipulationMulti-Task Robotic ManipulationRobotic ManipulationRobot Imitation Learning+1
AskPDF
Dec 7, 2025

MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment

Scalable embodied intelligence is constrained by the scarcity of diverse, long-horizon robotic manipulation data. Existing video world models in this domain are limited to synthesizing short clips of simple actions and often rely on manually defined trajectories. To this end, we introduce MIND-V, a cognitive hierarchical world model designed to synthesize physically plausible and logically coherent videos of long-horizon robotic manipulation. Inspired by cognitive science, MIND-V bridges high-level reasoning with pixel-level synthesis through three core components: a Semantic Reasoning Hub (SRH) that leverages a pre-trained vision-language model for task planning; a Behavioral Semantic Bridge (BSB) that translates abstract instructions into domain-invariant representations; and a Motor Video Generator (MVG) for conditional video rendering. MIND-V employs Staged Visual Future Rollouts, a test-time optimization strategy to enhance long-horizon robustness. To enforce adherence to physical laws, we introduce a GRPO reinforcement learning post-training phase guided by a novel Physical Foresight Coherence (PFC) reward. PFC leverages the V-JEPA2 world model as a physics referee to penalize implausible dynamics in the latent feature space. Experiments confirm MIND-V's SOTA performance in long-horizon simulation and its significant value for policy learning, introducing a scalable and fully autonomous framework for embodied data synthesis.

Ruicheng Zhang, Mingyang Zhang, Jun Zhou +6

Tsinghua University · Sun Yat-sen University · X Square Robot +1

Physical Consistency in Video GenerationLong-Horizon Robotic ManipulationRL for Video GenerationSynthetic Data Generation for RoboticsRL for Generative ModelsEmbodied World Models
AskPDF
Oct 20, 2025

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems often follow a dual-system paradigm, where System 2 performs high-level reasoning and System 1 handles low-level control. We refer to System 2 as the embodied brain, the cognitive core for decision-making in manipulation. Although evaluating this embodied brain is crucial, existing benchmarks mainly measure execution success or cover only limited aspects of high-level cognition and task realism. We introduce RoboBench, a benchmark for evaluating multimodal large language models (MLLMs) as embodied brains. RoboBench covers five dimensions: Instruction Comprehension, Perception Reasoning, Generalized Planning, Affordance Prediction, and Failure Analysis. It spans 14 capabilities, 25 tasks, and 6,092 QA pairs. To improve realism, it draws from large-scale real robotic data and in-house collection across diverse embodiments, attribute-rich objects, multi-view scenes, and memory-driven navigation. For planning, RoboBench introduces an MLLM-as-world-simulator framework that assesses whether predicted plans can achieve critical object-state changes under physical and visual constraints, enabling more faithful evaluation of long-horizon reasoning than symbolic matching. Experiments on 18 state-of-the-art MLLMs reveal persistent limitations in implicit instruction understanding, spatiotemporal reasoning, cross-scenario planning, fine-grained affordance understanding, and failure diagnosis. We further analyze how embodied cognitive abilities relate to downstream robotic control. RoboBench offers a comprehensive scaffold for quantifying high-level cognition and guiding next-generation MLLMs toward more robust robotic intelligence.

Yulin Luo, Chun-Kai Fan, Menghang Dong +19

State Key Laboratory of Multimedia Information Processing, School of Computer2026 Science, Peking University · Institute for Brain and Intelligence, Fudan University · University of Science andJul Technology Beijing +2

Long-Horizon Robotic ManipulationMultimodal Large Language ModelsRobot Manipulation BenchmarksEmbodied AIMultimodal Model EvaluationEmbodied Reasoning
AskPDF
Sep 30, 2025

RoboPilot: Generalizable Dynamic Robotic Manipulation with Dual-thinking Modes

Despite rapid progress in robotics, complex or long-horizon tasks remain a fundamental challenge. Most current approaches follow an open-loop paradigm with limited reasoning and no feedback, resulting in poor robustness to environmental changes and severe error accumulation. We present RoboPilot, a dual-thinking closed-loop agentic framework for robotic manipulation that supports adaptive reasoning for complex tasks in real-world dynamic environments. RoboPilot leverages primitive actions for structured task planning and flexible action generation as a agentic system, while introducing feedback to enable replanning from dynamic changes and execution errors. Chain-of-Thought reasoning further enhances high-level task planning and guides low-level action generation. The agentic system dynamically switches between fast and slow thinking to balance efficiency and accuracy. To systematically evaluate the robustness of RoboPilot in diverse robot manipulation scenarios, we introduce RoboPilot-Bench, a benchmark spanning 21 tasks across 10 categories, including infeasible-task recognition and dynamic recovery. Experiments show that RoboPilot outperforms state-of-the-art baselines by 11% in task success rate, and the real-world deployment on an industrial robot further demonstrates its robustness.

Xinyi Liu, Mohammadreza Fani Sani, Zewei Zhou +3

Microsoft, Denmark · Technical University of Denmark · University of California, Los Angeles

Robotic ManipulationDynamic Robotic ManipulationLong-Horizon Robotic ManipulationRobot Task PlanningGeneralization in Robotic ManipulationRobot Manipulation Benchmarks
AskPDF
Sep 23, 2025

Robotic Ultra-Long-Horizon Manipulation Skills via Human-guided Lifelong Code Generation

Large language models (LLMs) can translate natural-language instructions for robotic manipulation into executable code, but ambiguity, noisy generations, and limited context windows make ultra-long-horizon tasks unreliable. Closed-loop approaches that rely only on LLM feedback also struggle because LLMs have limited robotic reasoning, even when task errors are obvious to humans. Feedback is often stored in representations that generalize poorly to unseen tasks and can cause catastrophic forgetting as new corrections accumulate. We propose LYRA, a human-guided lifelong skill learning and code generation framework that distills human feedback into modular, reusable skills and incrementally extends their functionality across successive interactions while preserving previously learned behavior. External memory stores learned skills and execution examples; retrieval-augmented generation selects relevant knowledge, while user hints guide reuse when retrieval is insufficient, supporting ultra-long-horizon execution. Experiments on Ravens, Franka Kitchen, LIBERO-long, MetaWorld, and real-world tasks show a 0.93 success rate, up to 27% higher than baselines, and a 42% improvement in correction efficiency. LYRA also robustly solves ``build a house'', which requires planning over 20 primitives.

Yuan Meng, Zhenguo Sun, Max Fest +4

School of Computation, Information and Technology Technical University of Munich Munich, Germany · Beijing Academy of Artificial Intelligence (BAAI) Beijing, China · The State Key Laboratory for Novel Software Technology Nanjing University Suzhou, China

Code Generation for RoboticsLong-Horizon Robotic ManipulationRobot Skill LearningRobotic ManipulationContinual Robot LearningHuman-in-the-Loop AI
AskPDF
Jul 15, 2025

Robot Drummer: Learning Rhythmic Skills for Humanoid Drumming

Humanoid robots have seen remarkable advances in dexterity, balance, and locomotion, yet their role in expressive domains such as music performance remains largely unexplored. Musical tasks, like drumming, present unique challenges such as split-second timing, rapid contacts, and multi-limb coordination over performances lasting minutes. In this paper, we introduce Robot Drummer, a simulation framework for humanoid drumming across a diverse repertoire of songs. We formulate humanoid drumming as the realization of timed contact events encoded as a Rhythmic Contact Chain. To handle the long-horizon nature of musical performance, we decompose each track into fixed-length segments and train a single policy across all segments in parallel using reinforcement learning. Through extensive experiments on over thirty popular tracks, our results demonstrate that Robot Drummer consistently achieves high F1 scores and enables efficient learning of long-horizon musical performances. The learned behaviors exhibit emergent human-like drumming strategies, such as cross-arm strikes, and adaptive stick assignments, demonstrating the potential of reinforcement learning to bring humanoid robots into the domain of creative musical performance. Project page: robotdrummer.github.io

Asad Ali Shahid, Francesco Braghin, Loris Roveda

IDSIA, SUPSI, Lugano, Switzerland · Department of Mechanical Engineering, Politecnico di Milano, Italy

Long-Horizon Robotic ManipulationRobot Skill LearningRobotics SimulationHumanoid Robot ControlRobotic RL
AskPDF
Jun 12, 2025

Gondola: Grounded Vision Language Planning for Robotic Manipulation

Vision-language-action (VLA) models have shown promising progress in robotic manipulation. However, directly mapping visual observations and language instructions to low-level actions often results in limited interpretability and weak robustness in complex, long-horizon tasks. To address these challenges, we employ a modular manipulation framework that separates high-level planning from low-level control. At its core is Gondola, a grounded vision-language planning model that generates structured plans with explicit pixel-level object grounding before action execution. Given multi-view observations and planning history, Gondola predicts the next-step plan as interleaved textual instructions and multi-view segmentation masks corresponding to target objects and goal locations. To train Gondola, we construct synthetic datasets that provide explicit supervision for short-horizon grounded planning, multi-view referring expression, and long-horizon compositional reasoning. By coupling grounded plan generation with a 3D-based execution policy, our framework achieves state-of-the-art performance on the challenging GemBench benchmark. The system further demonstrates promising transfer to real robots. Ablation studies confirm that pixel-level grounding and the proposed planning-oriented supervision are critical for effective high-level reasoning. Project webpage: https://cshizhe.github.io/projects/robot_gondola.html

Shizhe Chen, Ricardo Garcia, Paul Pacaud +1

Inria, École normale supérieure, CNRS, PSL Research University

Long-Horizon Robotic ManipulationRobot Task PlanningVisually Grounded ReasoningVisual GroundingVLMs for RoboticsVisual Grounding for Robotics
AskPDF
Apr 23, 2025

MOSAIC: Skill-Centric Manipulation Planning with Physics Simulation

Planning long-horizon manipulation motions using a set of predefined skills is a central challenge in robotics; solving it efficiently could enable general-purpose robots to tackle novel tasks by flexibly composing generic skills. Solutions to this problem lie in an infinitely vast space of parameterized skill sequences -- a space where common incremental methods struggle to find sequences that have non-obvious intermediate steps. Some approaches reason over lower-dimensional, symbolic spaces, which are more tractable to explore but may be brittle and are laborious to construct. In this work, we introduce MOSAIC, a skill-centric, multi-directional planning approach that targets these challenges by reasoning about which skills to employ and where they are most likely to succeed, by utilizing physics simulation to estimate skill execution outcomes. Specifically, MOSAIC employs two complementary skill families: Generators, which identify ``islands of competence'' where skills are demonstrably effective, and Connectors, which link these skill-trajectories by solving boundary value problems. By focusing planning efforts on regions of high competence, MOSAIC efficiently discovers physically-grounded solutions. We demonstrate its efficacy on complex long-horizon problems in both simulation and the real world, using a diverse set of skills including generative diffusion models, motion planning algorithms, and manipulation-specific models. Visit skill-mosaic.github.io for demonstrations and examples.

Itamar Mishani, Yorai Shaoul, Maxim Likhachev

Robotics Institute, School of Computer Science, Carnegie Mellon University, United States

Long-Horizon Robotic ManipulationRobot Task Planning
AskPDF
Mar 9, 2025

AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems

We explore how scalable robot data can address real-world challenges for generalized robotic manipulation. Introducing AgiBot World, a large-scale platform comprising over 1 million trajectories across 217 tasks in five deployment scenarios, we achieve an order-of-magnitude increase in data scale compared to existing datasets. Accelerated by a standardized collection pipeline with human-in-the-loop verification, AgiBot World guarantees high-quality and diverse data distribution. It is extensible from grippers to dexterous hands and visuo-tactile sensors for fine-grained skill acquisition. Building on top of data, we introduce Genie Operator-1 (GO-1), a novel generalist policy that leverages latent action representations to maximize data utilization, demonstrating predictable performance scaling with increased data volume. Policies pre-trained on our dataset achieve an average performance improvement of 30% over those trained on Open X-Embodiment, both in in-domain and out-of-distribution scenarios. GO-1 exhibits exceptional capability in real-world dexterous and long-horizon tasks, achieving over 60% success rate on complex tasks and outperforming prior RDT approach by 32%. By open-sourcing the dataset, tools, and models, we aim to democratize access to large-scale, high-quality robot data, advancing the pursuit of scalable and general-purpose intelligence.

AgiBot World Contributors, Qingwen Bu, Jisong Cai +49

Robot Manipulation DatasetsGeneralization in Robotic ManipulationRobotic Data CollectionLong-Horizon Robotic Manipulation
AskPDF