Visuomotor Policy Learning

Ask about this topic

Overview

Visuomotor policies map what a robot sees to actions, helping it manipulate objects, navigate, and interact when the scene or task changes. Common approaches learn from demonstrations through action-chunking transformers, diffusion or flow-matching policies, and vision-language-action models; reinforcement learning and policy distillation are also used. Researchers improve these policies with geometric and object-centered visual features, memory, contact sensing, and predictions of how actions will change the scene, evaluating them on benchmarks such as LIBERO and RoboTwin as well as physical robots. A central tension is achieving reliable generalization and safe, precise control in real settings while keeping data needs, model size, and inference latency low.

Latest papers

CardsList

286 papers54 papers in the last four weeks, up 29% on the four weeks before. 0.3% of all new papers.

Apr 23, 2026

FingerViP: Learning Real-World Dexterous Manipulation with Fingertip Visual Perception

Current dexterous manipulation systems commonly rely on external or wrist-mounted cameras for visual feedback. However, such sensing setups are prone to occlusion and can limit performance on tasks requiring close-range, multi-view perception. In this work, we design a vision-enhanced fingertip module that brings clear visual perception directly to the fingertips and present FingerViP, a learning system that leverages fingertip visual perception for dexterous manipulation. Specifically, the fingertip module integrates miniature-camera sensing within compact-footprint hardware and is installed on each finger of a multi-fingered hand, which substantially improves visual perception by providing comprehensive, multi-view observations of objects and surrounding environment. Building on the integrated fingertip modules, we develop a diffusion-based whole-body visuomotor policy conditioned on a third-view camera and multi-view fingertip vision, enabling the robot to effectively learn complex manipulation skills directly from human demonstrations. To improve view-proprioception alignment and contact awareness in the policy observation representation, each fingertip visual feature is augmented with its corresponding camera-pose encoding and per-finger joint-current encoding. We validate the effectiveness of multi-view fingertip vision and demonstrate the robustness and adaptability of FingerViP across a range of challenging real-world tasks, including retrieving objects behind an occluding curtain, achieving a 56.6%56.6\% relative improvement in average success rate over the baseline using an external third-view camera and a wrist-mounted camera. Videos, code, and hardware designs are available on our project page: https://fingervip.github.io/.

Zhen Zhang, Weinan Wang, Hejia Sun +4

The Chinese University of Hong Kong · The Hong Kong Polytechnic University

Dexterous ManipulationVisuomotor Policy LearningRobot Imitation LearningRobotic Perception
AskPDF
Apr 21, 2026

Gated Memory Policy

Robotic manipulation tasks exhibit varying memory requirements, ranging from Markovian tasks that require no memory to non-Markovian tasks that depend on historical information spanning single or multiple interaction trials. Surprisingly, simply extending observation histories of a visuomotor policy often leads to a significant performance drop due to distribution shift and overfitting. To address these issues, we propose Gated Memory Policy (GMP), a visuomotor policy that learns both when to recall memory and what to recall. To learn when to recall memory, GMP employs a learned memory gate mechanism that selectively activates history context only when necessary, improving robustness and reactivity. To learn what to recall efficiently, GMP introduces a lightweight cross-attention module that constructs effective latent memory representations. To further enhance robustness, GMP injects diffusion noise into historical actions, mitigating sensitivity to noisy or inaccurate histories during both training and inference. On our proposed non-Markovian benchmark MemMimic, GMP achieves a 30.1% average success rate improvement over long-history baselines, while maintaining competitive performance on Markovian tasks in RoboMimic. All code, data and in-the-wild deployment instructions are available on our project website https://gated-memory-policy.github.io/.

Yihuai Gao, Jinyun Liu, Shuang Li +1

Stanford University

Sequential Decision MakingVisuomotor Policy LearningRobot Imitation LearningRobot Manipulation BenchmarksMemory-Augmented Agents
AskPDF
Apr 16, 2026

R3D: Revisiting 3D Policy Learning

3D policy learning promises superior generalization and cross-embodiment transfer, but progress has been hindered by training instabilities and severe overfitting, precluding the adoption of powerful 3D perception models. In this work, we systematically diagnose these failures, identifying the omission of 3D data augmentation and the adverse effects of Batch Normalization as primary causes. We propose a new architecture coupling a scalable transformer-based 3D encoder with a diffusion decoder, engineered specifically for stability at scale and designed to leverage large-scale pre-training. Our approach significantly outperforms state-of-the-art 3D baselines on challenging manipulation benchmarks, establishing a new and robust foundation for scalable 3D imitation learning. Project Page: https://r3d-policy.github.io/

Zhengdong Hong, Shenrui Wu, Haozhe Cui +8

Zhejiang University · ShanghaiTech University

3D ViTsVisuomotor Policy LearningDiffusion PolicyImitation LearningRobot Imitation Learning
AskPDF
Apr 16, 2026

DockAnywhere: Data-Efficient Visuomotor Policy Learning for Mobile Manipulation via Novel Demonstration Generation

Mobile manipulation is a fundamental capability that enables robots to interact in expansive environments such as homes and factories. Most existing approaches follow a two-stage paradigm, where the robot first navigates to a docking point and then performs fixed-base manipulation using powerful visuomotor policies. However, real-world mobile manipulation often suffers from the view generalization problem due to shifts of docking points. To address this issue, we propose a novel low-cost demonstration generation framework named DockAnywhere, which improves viewpoint generalization under docking variability by lifting a single demonstration to diverse feasible docking configurations. Specifically, DockAnywhere lifts a trajectory to any feasible docking points by decoupling docking-dependent base motions from contact-rich manipulation skills that remain invariant across viewpoints. Feasible docking proposals are sampled under feasibility constraints, and corresponding trajectories are generated via structure-preserving augmentation. Visual observations are synthesized in 3D space by representing the robot and objects as point clouds and applying point-level spatial editing to ensure the consistency of observation and action across viewpoints. Extensive experiments on ManiSkill and real-world platforms demonstrate that DockAnywhere substantially improves policy success rates and easily generalizes to novel viewpoints from unseen docking points during training, significantly enhancing the generalization capability of mobile manipulation policy in real-world deployment.

Ziyu Shan, Yuheng Zhou, Gaoyuan Wu +3

Beijing University of Posts and Telecommunications, Beijing, China · Nanyang Technological University, Singapore

Mobile ManipulationGeneralization in Robotic ManipulationVisuomotor Policy LearningSynthetic Data Generation for RoboticsRobotic ManipulationRobot Policy Generalization
AskPDF
Apr 15, 2026

Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection

Recent vision-language-action (VLA) systems have demonstrated strong capabilities in embodied manipulation. However, most existing VLA policies rely on limited observation windows and end-to-end action prediction, which makes them brittle in long-horizon, memory-dependent tasks with partial observability, occlusions, and multi-stage dependencies. Such tasks require not only precise visuomotor control, but also persistent memory, adaptive task decomposition, and explicit recovery from execution failures. To address these limitations, we propose a dual-system framework for long-horizon embodied manipulation. Our framework explicitly separates high-level semantic reasoning from low-level motor execution. A high-level planner, implemented as a VLM-based agentic module, maintains structured task memory and performs goal decomposition, outcome verification, and error-driven correction. A low-level executor, instantiated as a VLA-based visuomotor controller, carries out each sub-task through diffusion-based action generation conditioned on geometry-preserving filtered observations. Together, the two systems form a closed loop between planning and execution, enabling memory-aware reasoning, adaptive replanning, and robust online recovery. Experiments on representative RMBench tasks show that the proposed framework substantially outperforms representative baselines, achieving a 32.4% average success rate compared with 9.8% for the strongest baseline. Ablation studies further confirm the importance of structured memory and closed-loop recovery for long-horizon manipulation.

Zhen Liu, Xinyu Ning, Zhe Hu +8

Beijing University of Posts and Telecommunications · InspireOmni AI · Tsinghua University

Long-Horizon Robotic ManipulationRobot Task PlanningVLMs for RoboticsVisuomotor Policy Learning
AskPDF
Apr 12, 2026

WARPED: Wrist-Aligned Rendering for Robot Policy Learning from Egocentric Human Demonstrations

Recent advancements in learning from human demonstration have shown promising results in addressing the scalability and high cost of data collection required to train robust visuomotor policies. However, existing approaches are often constrained by a reliance on multiview camera setups, depth sensors, or custom hardware, and are typically limited to policy execution from third-person or egocentric cameras. In this paper, we present WARPED, a framework designed to synthesize realistic wrist-view observations and actions from human demonstration videos to facilitate the training of visuomotor policies using only monocular RGB data. With data collected from an egocentric RGB camera, our system leverages vision foundation models to initialize the interactive scene. A hand-object interaction pipeline is then employed to track the hand and manipulated object and retarget the trajectories to a robotic end-effector. Lastly, photorealistic wrist-view observations are synthesized via Gaussian Splatting to directly train a robotic policy. We demonstrate that WARPED achieves success rates comparable to teleoperation on four of five rigid-object tabletop tasks and one articulated-object task, while requiring 4-9x less data collection time.

Harry Freeman, Chung Hee Kim, Jenny Wang +1

Carnegie Mellon University Robotics Institute, PA, USA

Visuomotor Policy LearningRobot Imitation LearningSynthetic Data Generation for Robotics
AskPDF
Apr 7, 2026

Action Images: End-to-End Policy Learning via Multiview Video Generation

World action models (WAMs) have emerged as a promising direction for robot policy learning, as they can leverage powerful video backbones to model the future states. However, existing approaches often rely on separate action modules, or use action representations that are not pixel-grounded, making it difficult to fully exploit the pretrained knowledge of video models and limiting transfer across viewpoints and environments. In this work, we present Action Images, a unified world action model that formulates policy learning as multiview video generation. Instead of encoding control as low-dimensional tokens, we translate 7-DoF robot actions into interpretable action images: multi-view action videos that are grounded in 2D pixels and explicitly track robot-arm motion. This pixel-grounded action representation allows the video backbone itself to act as a zero-shot policy, without a separate policy head or action module. Beyond control, the same unified model supports video-action joint generation, action-conditioned video generation, and action labeling under a shared representation. On RLBench and real-world evaluations, our model achieves the strongest zero-shot success rates and improves video-action joint generation quality over prior video-space world models, suggesting that interpretable action images are a promising route to policy learning.

Haoyu Zhen, Zixian Gao, Qiao Sun +7

UMass Amherst · NVIDIA · Harvard University +1

Action Representation LearningWorld Action ModelsVisuomotor Policy LearningAction-Conditioned Video GenerationMulti-View Video Generation
AskPDF
Apr 7, 2026

Precise Aggressive Aerial Maneuvers with Sensorimotor Policies

Precise aggressive maneuvers with lightweight onboard sensors remains a key bottleneck in fully exploiting the maneuverability of drones. Such maneuvers are critical for expanding the systems' accessible area by navigating through narrow openings in the environment. Among the most relevant problems, a representative one is aggressive traversal through narrow gaps with quadrotors under SE(3) constraints, which require the quadrotors to leverage a momentary tilted attitude and the asymmetry of the airframe to navigate through gaps. In this paper, we achieve such maneuvers by developing sensorimotor policies directly mapping onboard vision and proprioception into low-level control commands. The policies are trained using reinforcement learning (RL) with end-to-end policy distillation in simulation. We mitigate the fundamental hardness of model-free RL's exploration on the restricted solution space with an initialization strategy leveraging trajectories generated by a model-based planner. Careful sim-to-real design allows the policy to control a quadrotor through narrow gaps with low clearances and high repeatability. For instance, the proposed method enables a quadrotor to navigate a rectangular gap at a 5 cm clearance, tilted at up to 90-degree orientation, without knowledge of the gap's position or orientation. Without training on dynamic gaps, the policy can reactively servo the quadrotor to traverse through a moving gap. The proposed method is also validated by training and deploying policies on challenging tracks of narrow gaps placed closely. The flexibility of the policy learning method is demonstrated by developing policies for geometrically diverse gaps, without relying on manually defined traversal poses and visual features.

Tianyue Wu, Guangtong Xu, Zihan Wang +6

Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University, Hangzhou, China. · Huzhou Institute of Zhejiang University, Huzhou, China. · Differential Robotics Technology Co., Ltd., Hangzhou, China.

UAV NavigationQuadrotor ControlVisuomotor Policy LearningRL for Robotics
AskPDF
Apr 7, 2026

Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation

This paper addresses a fundamental problem of visuomotor policy learning for robotic manipulation: how to enhance robustness in out-of-distribution execution errors or dynamically re-routing trajectories, where the model relies solely on the original expert demonstrations for training. We introduce the Referring-Aware Visuomotor Policy (ReV), a closed-loop framework that can adapt to unforeseen circumstances by instantly incorporating sparse referring points provided by a human or a high-level reasoning planner. Specifically, ReV leverages the coupled diffusion heads to preserve standard task execution patterns while seamlessly integrating sparse referring via a trajectory-steering strategy. Upon receiving a specific referring point, the global diffusion head firstly generates a sequence of globally consistent yet temporally sparse action anchors, while identifies the precise temporal position for the referring point within this sequence. Subsequently, the local diffusion head adaptively interpolates adjacent anchors based on the current temporal position for specific tasks. This closed-loop process repeats at every execution step, enabling real-time trajectory replanning in response to dynamic changes in the scene. In practice, rather than relying on elaborate annotations, ReV is trained only by applying targeted perturbations to expert demonstrations. Without any additional data or fine-tuning scheme, ReV achieve higher success rates across challenging simulated and real-world tasks.

Jiahua Ma, Yiran Qin, Xin Wen +5

Sun Yat-sen University · Oxford University.

Visuomotor Policy LearningRobot Policy AdaptationRobotic ManipulationRobot Policy GeneralizationDiffusion Policy
AskPDF
Apr 6, 2026

GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields

Learning visuomotor policies for Autonomous Aerial Vehicles (AAVs) relying solely on monocular vision is an attractive yet highly challenging paradigm. Existing end-to-end learning approaches directly map high-dimensional RGB observations to action commands, which frequently suffer from low sample efficiency and severe sim-to-real gaps due to the visual discrepancy between simulation and physical domains. To address these long-standing challenges, we propose GaussFly, a novel framework that explicitly decouples representation learning from policy optimization through a cohesive real-to-sim-to-real paradigm. First, to achieve a high-fidelity real-to-sim transition, we reconstruct training scenes using 3D Gaussian Splatting (3DGS) augmented with explicit geometric constraints. Second, to ensure robust sim-to-real transfer, we leverage these photorealistic simulated environments and employ contrastive representation learning to extract compact, noise-resilient latent features from the rendered RGB images. By utilizing this pre-trained encoder to provide low-dimensional feature inputs, the computational burden on the visuomotor policy is significantly reduced while its resistance against visual noise is inherently enhanced. Extensive experiments in simulated and real-world environments demonstrate that GaussFly achieves superior sample efficiency and asymptotic performance compared to baselines. Crucially, it enables robust and zero-shot policy transfer to unseen real-world environments with complex textures, effectively bridging the sim-to-real gap.

Yuhang Zhang, Mingsheng Li, Yujing Shang +4

School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798, Singapore · College of Design and Engineering, National University of Singapore, Singapore 119077, Singapore +1

Visuomotor Policy LearningVisual RLSim-to-Real TransferAerial Robotics
AskPDF
Mar 31, 2026

Efficient Camera Pose Augmentation for View Generalization in Robotic Policy Learning

Prevailing 2D-centric visuomotor policies exhibit a pronounced deficiency in novel view generalization, as their reliance on static observations hinders consistent action mapping across unseen views. In response, we introduce GenSplat, a feed-forward 3D Gaussian Splatting framework that facilitates view-generalized policy learning through novel view rendering. GenSplat employs a permutation-equivariant architecture to reconstruct high-fidelity 3D scenes from sparse, uncalibrated inputs in a single forward pass. To ensure structural integrity, we design a 3D-prior distillation strategy that regularizes the 3DGS optimization, preventing the geometric collapse typical of purely photometric supervision. By rendering diverse synthetic views from these stable 3D representations, we systematically augment the observational manifold during training. This augmentation forces the policy to ground its decisions in underlying 3D structures, thereby ensuring robust execution under severe spatial perturbations where baselines severely degrade.

Sen Wang, Huaiyi Dong, Jingyi Tian +7

Xi’an Jiaotong University, Noah’s Ark Lab

Robot Policy GeneralizationRobot Policy LearningVisuomotor Policy Learning3D Gaussian Splatting
AskPDF
Mar 28, 2026

HiFlow: Tokenization-Free Scale-Wise Autoregressive Policy Learning via Flow Matching

Coarse-to-fine autoregressive modeling has recently shown strong promise for visuomotor policy learning, combining the inference efficiency of autoregressive methods with the global trajectory coherence of diffusion-based policies. However, existing approaches rely on discrete action tokenizers that map continuous action sequences to codebook indices, a design inherited from image generation where learned compression is necessary for high-dimensional pixel data. We observe that robot actions are inherently low-dimensional continuous vectors, for which such tokenization introduces unnecessary quantization error and a multi-stage training pipeline. In this work, we propose Hierarchical Flow Policy (HiFlow), a tokenization-free coarse-to-fine autoregressive policy that operates directly on raw continuous actions. HiFlow constructs multi-scale continuous action targets from each action chunk via simple temporal pooling. Specifically, it averages contiguous action windows to produce coarse summaries that are refined at finer temporal resolutions. The entire model is trained end-to-end in a single stage, eliminating the need for a separate tokenizer. Experiments on MimicGen, RoboTwin 2.0, and real-world environments demonstrate that HiFlow consistently outperforms existing methods including diffusion-based and tokenization-based autoregressive policies.

Daichi Yashima, Koki Seno, Shuhei Kurita +2

Keio University · National Institute of Informatics Research and Development Center for Large Language Models · National Institute of Informatics

Visuomotor Policy LearningFlow-Based Policy LearningReinforcement LearningFlow-Based Generative Modeling
AskPDF
Mar 26, 2026

Emergent Neural Automaton Policies: Learning Symbolic Structure from Visuomotor Trajectories

Scaling robot learning to long-horizon tasks remains a formidable challenge. While end-to-end policies often lack the structural priors needed for effective long-term reasoning, traditional neuro-symbolic methods rely heavily on hand-crafted symbolic priors. We introduce ENAP (Emergent Neural Automaton Policy), a neuro-symbolic imitation learning framework in which a bi-level policy adaptively emerges from visuomotor demonstrations. Specifically, we employ adaptive clustering and an extension of the L* algorithm to learn the task structure from demonstrations as a Mealy state machine, which serves as an interpretable high-level planner capturing latent task modes. This discrete structure then guides a low-level reactive residual network to learn precise continuous control via behavior cloning (BC). By explicitly modeling discrete transitions and continuous residuals, ENAP achieves sample efficiency and interpretability without task-specific labels. Experiments on complex manipulation and long-horizon tasks show that ENAP outperforms state-of-the-art end-to-end vision-language-action (VLA) policies by up to 27% in low-data regimes. ENAP can also distill task structure from the latent space of a pre-trained VLA, surpassing it on CALVIN benchmark with 39% fewer parameters (Fig. 1).

Yiyuan Pan, Xusheng Luo, Hanjiang Hu +2

Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA

Automata LearningVisuomotor Policy LearningNeurosymbolic LearningLong-Horizon Robotic ManipulationRobot Imitation Learning
AskPDF
Mar 25, 2026

Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulation

Robots often observe information that determines a future action long before that action is executed. In a shell game, for example, a robot first sees which cup hides the ball, watches the cups move, and only later needs to choose the correct cup. The final observation alone is not enough for a decision: the correct action depends on an earlier event. We refer to this temporal gap as observation-action delay. It makes memory a policy-facing problem: a policy must keep similar histories distinct, retrieve the past event relevant to the current decision, and convert that recall into an action-ready state. We call these requirements separability, addressability, and prospectiveness. We introduce Chameleon, a ~60M visuomotor policy for control-indexed prospective memory. Chameleon writes embodied event memory, preserves separable histories, retrieves control-relevant traces, and trains the resulting working state to be prospective. We also introduce Camo-Dataset, a real-robot benchmark that isolates observation-action delay by making the decision scene visually ambiguous, so the correct action must be inferred from earlier observations. Chameleon improves decision/end-to-end success on Camo-Dataset from 22.5%/21.3% to 80.8%/71.3%. On public long-horizon memory benchmarks, it achieves 87.1% +/- 0.8% on LIBERO-10, 97.3% +/- 4.5% on MemoryBench, and 75.1% +/- 1.4% on MIKASA-Robo, setting the state of the art for same-size models and exceeding multiple larger VLA baselines under the reported protocols. Probes and ablations show that Chameleon learns separable, addressable, and prospective memory, and that these properties drive its performance gains.

Xinying Guo, Chenxi Jiang, Hyun Bin Kim +4

MARS Lab, Nanyang Technological University · Institute for Infocomm Research, A*STAR · National University of Singapore

Memory-Augmented AgentsLong-Horizon Robotic ManipulationVisuomotor Policy LearningRobot Manipulation Benchmarks
AskPDF
Mar 24, 2026

Efficient Hybrid SE(3)-Equivariant Visuomotor Flow Policy via Spherical Harmonics for Robot Manipulation

While existing equivariant methods enhance data efficiency, they suffer from high computational intensity, reliance on single-modality inputs, and instability when combined with fast-sampling methods. In this work, we propose E3Flow, a novel framework that addresses the critical limitations of equivariant diffusion policies. E3Flow overcomes these challenges, successfully unifying efficient rectified flow with stable, multi-modal equivariant learning for the first time. Our framework is built upon spherical harmonic representations to ensure rigorous SO(3) equivariance. We introduce a novel invariant Feature Enhancement Module (FEM) that dynamically fuses hybrid visual modalities (point clouds and images), injecting rich visual cues into the spherical harmonic features. We evaluate E3Flow on 8 manipulation tasks from the MimicGen and further conduct 4 real-world experiments to validate its effectiveness in physical environments. Simulation results show that E3Flow achieves a 3.12% improvement in average success rate over the state-of-the-art Spherical Diffusion Policy (SDP) while simultaneously delivering a 7x inference speedup. E3Flow thus demonstrates a new and highly effective trade-off between performance, efficiency, and data efficiency for robotic policy learning. Code: https://github.com/zql-kk/E3Flow.

Qinglun Zhang, Shen Cheng, Tian Dan +3

University of Electronic Science and Technology of China · Dexmal

Visuomotor Policy LearningRobot LearningRobotic ManipulationData-Efficient Robot Learning
AskPDF
Mar 22, 2026

Dreaming the Unseen: World Model-regularized Diffusion Policy for Out-of-Distribution Robustness

Diffusion policies excel at visuomotor control but often fail catastrophically under severe out-of-distribution (OOD) disturbances, such as unexpected object displacements or visual corruptions. To address this vulnerability, we introduce the Dream Diffusion Policy (DDP), a framework that deeply integrates a diffusion world model into the policy's training objective via a shared 3D visual encoder. This co-optimization endows the policy with robust state-prediction capabilities. When encountering sudden OOD anomalies during inference, DDP detects the real-imagination discrepancy and actively abandons the corrupted visual stream. Instead, it relies on its internal "imagination" (autoregressively forecasted latent dynamics) to safely bypass the disruption, generating imagined trajectories before smoothly realigning with physical reality. Extensive evaluations demonstrate DDP's exceptional resilience. Notably, DDP achieves a 73.8% OOD success rate on MetaWorld (vs. 23.9% without predictive imagination) and an 83.3% success rate under severe real-world spatial shifts (vs. 3.3% without predictive imagination). Furthermore, as a stress test, DDP maintains a 76.7% real-world success rate even when relying entirely on open-loop imagination post-initialization.

Ziou Hu, Xiangtong Yao, Yuan Meng +2

School of Computation, Information and Technology, Technical University of Munich, Garching, Germany · State Key Laboratory for Novel Software Technology and the School of Science and Technology, Nanjing University (Suzhou Campus), China

Diffusion PolicyOOD GeneralizationVisuomotor Policy LearningWorld Models for Robotics
AskPDF
Mar 18, 2026

VolumeDP: Modeling Volumetric Representation for Manipulation Policy Learning

Imitation learning is a prominent paradigm for robotic manipulation. However, existing visual imitation methods map 2D image observations directly to 3D action outputs, imposing a 2D-3D mismatch that hinders spatial reasoning and degrades robustness. We present VolumeDP, a policy architecture that restores spatial alignment by explicitly reasoning in 3D. VolumeDP first lifts image features into a Volumetric Representation via cross-attention. It then selects task-relevant voxels with a learnable module and converts them into a compact set of spatial tokens, markedly reducing computation while preserving action-critical geometry. Finally, a multi-token decoder conditions on the entire token set to predict actions, thereby avoiding lossy aggregation that collapses multiple spatial tokens into a single descriptor. VolumeDP achieves a state-of-the-art average success rate of 88.8% on the LIBERO simulation benchmark, outperforming the strongest baseline by a substantial 14.8% improvement. It also delivers large performance gains over prior methods on the ManiSkill and LIBERO-Plus benchmarks. Real-world experiments further demonstrate higher success rates and robust generalization to novel spatial layouts, camera viewpoints, and environment backgrounds. Code and videos are available on the project page: https://yzc0731.github.io/VolumeDP/

Tianxing Zhou, Feiyang Xue, Zhangchen Ye +3

1IIIS, Tsinghua University · 2Galaxea AI

Visuomotor Policy LearningRobotic ManipulationRobot Imitation Learning3D Representation Learning
AskPDF
Mar 18, 2026

REAL: Robust Extreme Agility via Spatio-Temporal Policy Learning and Physics-Guided Filtering

Extreme legged parkour demands rapid terrain assessment and precise foot placement under highly dynamic conditions. While recent learning-based systems achieve impressive agility, they remain fundamentally fragile to perceptual degradation, where even brief visual noise or latency can cause catastrophic failure. To overcome this, we propose Robust Extreme Agility Learning (REAL), an end-to-end framework for reliable parkour under sensory corruption. Instead of relying on perfectly clean perception, REAL tightly couples vision, proprioceptive history, and temporal memory. We distill a cross-modal teacher policy into a deployable student equipped with a FiLM-modulated Mamba backbone to actively filter visual noise and build short-term terrain memory actively. Furthermore, a physics-guided Bayesian state estimator enforces rigid-body consistency during high-impact maneuvers. Validated on a Unitree Go2 quadruped, REAL successfully traverses extreme obstacles even with a 1-meter visual blind zone, while strictly satisfying real-time control constraints with a bounded 13.1 ms inference time.

Jialong Liu, Dehan Shen, Yanbo Wen +2

PEAK-Lab, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China.

Robot Policy LearningRobot LocomotionLegged RoboticsVisuomotor Policy LearningTerrain-Aware Robot Locomotion
AskPDF
Mar 18, 2026

GigaWorld-Policy: An Efficient Action-Centered World--Action Model

World-Action Models (WAM) initialized from pre-trained video generation backbones have demonstrated remarkable potential for robot policy learning. However, existing approaches face two critical bottlenecks that hinder performance and deployment. First, jointly reasoning over future visual dynamics and corresponding actions incurs substantial inference overhead. Second, joint modeling often entangles visual and motion representations, making motion prediction accuracy heavily dependent on the quality of future video forecasts. To address these issues, we introduce GigaWorld-Policy, an action-centered WAM that learns 2D pixel-action dynamics while enabling efficient action decoding, with optional video generation. Specifically, we formulate policy training into two coupled components: the model predicts future action sequences conditioned on the current observation, and simultaneously generates future videos conditioned on the predicted actions and the same observation. The policy is supervised by both action prediction and video generation, providing richer learning signals and encouraging physically plausible actions through visual-dynamics constraints. With a causal design that prevents future-video tokens from influencing action tokens, explicit future-video generation is optional at inference time, allowing faster action prediction during deployment. To support this paradigm, we curate a diverse, large-scale robot dataset to pre-train an action-centered video generation model, which is then adapted as the backbone for robot policy learning. Experimental results on real-world robotic platforms show that GigaWorld-Policy runs 9x faster than the leading WAM baseline, Motus, while improving task success rates by 7%. Moreover, compared with pi-0.5, GigaWorld-Policy improves performance by 95% on RoboTwin 2.0.

Angen Ye, Boyuan Wang, Chaojun Ni +21

World Action ModelsVisuomotor Policy LearningEfficient Inference for World Action ModelsAction-Conditioned Video Generation
AskPDF
Mar 16, 2026

MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Task-Conditioned Mixture-of-Experts Transformers

Developing a unified policy for multi-task robotic manipulation remains challenging due to policy degradation from task interference and negative transfer. In this work, we propose Mixture-of-Experts-Enhanced Action Chunking Transformer (MoE-ACT), a parameter-efficient multi-task visuomotor framework tailored for bimanual manipulation. MoE-ACT incorporates sparse MoE layers into the ACT encoder, dynamically routing image tokens to selected experts based on task context, visual observations, and proprioceptive states. Furthermore, the framework incorporates task-conditioned Feature-wise Linear Modulation (FiLM) in the action decoder alongside multi-scale cross-attention, ensuring precise task grounding and capturing fine-grained spatial cues. Extensive evaluations on the RoboTwin 2.0 benchmark across 16 challenging bimanual tasks demonstrate that MoE-ACT achieves an average success rate of 62.0%, outperforming standard ACT by 17.4 percentage points. Crucially, with only 195M activated parameters, MoE-ACT surpasses the 16-fold larger foundation model ππ0 (3.24B) by 5.1 percentage points, exhibiting substantial gains in parameter efficiency and deployment feasibility. Real-robot dual-arm experiments consistently confirm its superior multi-task execution capabilities. Our open-source project page can be found at https://j3k7.github.io/MoE-ACT/.

Kangjun Guo, Haichao Liu, Yanji Sun +3

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China

Bimanual Robotic ManipulationVisuomotor Policy LearningMulti-Task LearningMulti-Task Robotic ManipulationAction ChunkingMixture-of-Experts Models
AskPDF
Mar 12, 2026

One-Step Flow Policy: Self-Distillation for Fast Visuomotor Policies

Generative flow and diffusion models provide the continuous, multimodal action distributions needed for high-precision robotic policies. However, their reliance on iterative sampling introduces severe inference latency, degrading control frequency and harming performance in time-sensitive manipulation. To address this problem, we propose the One-Step Flow Policy (OFP), a from-scratch self-distillation framework for high-fidelity, single-step action generation without a pre-trained teacher. OFP unifies a self-consistency loss to enforce coherent transport across time intervals, and a self-guided regularization to sharpen predictions toward high-density expert modes. In addition, a warm-start mechanism leverages temporal action correlations to minimize the generative transport distance. Evaluations across 56 diverse simulated manipulation tasks demonstrate that a one-step OFP achieves state-of-the-art results, outperforming 100-step diffusion and flow policies while accelerating action generation by over 100×100\times. We further integrate OFP into the π0.5π_{0.5} model on RoboTwin 2.0, where one-step OFP surpasses the original 10-step policy. These results establish OFP as a practical, scalable solution for highly accurate and low-latency robot control.

Shaolong Li, Lichao Sun, Yongchao Chen

CSE Department, University of Michigan · Lehigh University · College of AI, Tsinghua University

Visuomotor Policy LearningOne-Step Generative ModelingFlow-Based Generative ModelingRobotic ManipulationSelf-Distillation
AskPDF
Mar 12, 2026

Learning Visuomotor Policy for Multi-Robot Laser Tag Game

In this paper, we study multi robot laser tag, a simplified yet practical shooting-game-style task. Classic modular approaches on these tasks face challenges such as limited observability and reliance on depth mapping and inter robot communication. To overcome these issues, we present an end-to-end visuomotor policy that maps images directly to robot actions. We train a high performing teacher policy with multi agent reinforcement learning and distill its knowledge into a vision-based student policy. Technical designs, including a permutation-invariant feature extractor and depth heatmap input, improve performance over standard architectures. Our policy outperforms classic methods by 16.7% in hitting accuracy and 6% in collision avoidance, and is successfully deployed on real robots. Code will be released publicly.

Kai Li, Shiyu Zhao

College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.

Visuomotor Policy LearningMulti-Robot SystemsVision-Based Robot Control
AskPDF
Mar 12, 2026

Noise-Space Attribution and Control of Chunk-Boundary Artifact

Action chunking is widely used in generative visuomotor policies, yet the recurring execution discontinuities at chunk boundaries still lack a mechanistic explanation. This paper treats chunk-boundary artifact as an analyzable mechanism variable. We first show that successful and failed episodes separate stably on artifact metrics. We then show that, in stochastic action-chunked policies, fixing the observation context and changing only latent noise is sufficient to modulate artifact systematically. On the same Diffusion Policy checkpoint, comparisons among DDPM, zero-variance DDPM, and DDIM further show that this local controllability depends on whether the information path from initial noise to action output remains intact. Finally, from controlled interventions at fixed local execution states, we find that artifact changes can carry through to final outcome, and that the preferred direction can reverse even within the same task: some contexts achieve higher success under lower artifact, whereas others achieve higher success under higher artifact. In a representative high-artifact-favoring key context selected by held-out matched-continuation validation, success rate increases from 0.033 to 0.717. These results show that chunk-boundary artifact is not a mere execution-side by-product, but a variable in noise space that can be attributed, controlled, and mechanistically linked to task outcome.

Rui Wang

Nanjing University

Visuomotor Policy LearningAction ChunkingDiffusion Policy
AskPDF
Mar 11, 2026

FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model

Predictive foresight is important to intelligent embodied agents. Since the motor execution of a robot is intrinsically constrained by its visual perception of environmental geometry, effectively anticipating the future requires capturing this tightly coupled visuomotor interplay. While recent vision-language-action models attempt to incorporate future guidance, they struggle with this joint modeling. Existing explicit methods divert capacity to task-irrelevant visual details, whereas implicit methods relying on sparse frame pairs disrupt temporal continuity. By heavily relying on visual reconstruction, these methods become visually dominated, entangling static scene context with dynamic action intent. We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling. To this end, we propose FutureVLA, featuring a novel Joint Visuomotor Predictive Architecture. FutureVLA is designed to extract joint visuomotor embeddings by first decoupling visual and motor information, and then jointly encoding generalized physical priors. Specifically, in the pretraining stage, we leverage heterogeneous manipulation datasets and introduce a Joint Visuomotor Gating mechanism to structurally separate visual state preservation from temporal action modeling. It allows the motor stream to focus on continuous physical dynamics while explicitly querying visual tokens for environmental constraints, yielding highly generalizable joint visuomotor embeddings. Subsequently, in the post-training stage, we employ a latent embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors without modifying their inference architectures. Extensive experiments demonstrate that FutureVLA consistently improves VLA frameworks.

Xiaoxu Xu, Hao Li, Jinhui Ye +7

Beihang University. · Shanghai Artificial Intelligence Laboratory. · University of Science and Technology of China. +1

Vision-Language-Action ModelsVisuomotor Policy LearningRobotic ManipulationMotion Representation Learning
AskPDF
Mar 9, 2026

PhaForce: Phase-Scheduled Visual-Force Policy Learning with Slow Planning and Fast Correction for Contact-Rich Manipulation

Contact-rich manipulation requires not only vision-dominant task semantics but also closed-loop reactions to force/torque (F/T) transients. Yet, generative visuomotor policies are typically constrained to low-frequency updates due to inference latency and action chunking, underutilizing F/T for control-rate feedback. Furthermore, existing force-aware methods often inject force continuously and indiscriminately, lacking an explicit mechanism to schedule when / how much / where to apply force across different task phases. We propose PhaForce, a phase-scheduled visual--force policy that coordinates low-rate chunk-level planning and high-rate residual correction via a unified contact/phase schedule. PhaForce comprises (i) a contact-aware phase predictor (CAP) that estimates contact probability and phase belief, (ii) a Slow diffusion planner that performs dual-gated visual--force fusion with orthogonal residual injection to preserve vision semantics while conditioning on force, and (iii) a Fast corrector that applies control-rate phase-routed residuals in interpretable corrective subspaces for within-chunk micro-adjustments. Across multiple real-robot contact-rich tasks, PhaForce achieves an average success rate of 86% (+40 pp over baselines), while also substantially improving contact quality by regulating interaction forces and exhibiting robust adaptability to OOD geometric shifts.

Mingxin Wang, Zhirun Yue, Renhao Lu +7

Robotic ControlContact-Rich Robotic ManipulationVisuomotor Policy LearningForce-Controlled Robotic Manipulation
AskPDF
Mar 9, 2026

Vector Field Augmented Differentiable Policy Learning for Vision-Based Drone Racing

Autonomous drone racing in complex environments requires agile, high-speed flight while maintaining reliable obstacle avoidance. Differentiable-physics-based policy learning has recently demonstrated high sample efficiency and remarkable performance across various tasks, including agile drone flight and quadruped locomotion. However, applying such methods to drone racing remains difficult, as key objective like gate traversal are inherently hard to express as smooth, differentiable losses. To address these challenges, we propose DiffRacing, a novel vector field-augmented differentiable policy learning framework. DiffRacing integrates differentiable losses and vector fields into the training process to provide continuous and stable gradient signals, balancing obstacle avoidance and high-speed gate traversal. In addition, a differentiable Delta Action Model compensates for dynamics mismatch, enabling efficient sim-to-real transfer without explicit system identification. Extensive simulation and real-world experiments demonstrate that DiffRacing achieves superior sample efficiency, faster convergence, and robust flight performance, thereby demonstrating that vector fields can augment traditional gradient-based policy learning with a task-specific geometric prior.

Yang Su, Feng Yu, Yu Hu +4

The authors are with the Shanghai Jiao Tong University, Shanghai 200240, China

Autonomous Drone RacingPolicy LearningVisuomotor Policy Learning
AskPDF
Mar 9, 2026

Foundation and Small Models Coordination for Visuomotor Policy Learning

Visuomotor policy learning enables robots to perform a wide range of tasks, but small policy models often remain sensitive to changes in object and background appearance. In this work, we investigate the coordination of pretrained vision foundation models with small policy models to improve appearance generalization. We propose a framework in which a small policy model operates on task-relevant visual observations constructed through semantic repainting. A segmentation foundation model identifies the robot and target object, which are rendered with fixed role colors on a constant background. An alternative representation replaces the target's role color with normalized monocular depth predicted by a depth foundation model, providing additional geometric cues. The perception models are adapted using in-distribution data where needed and held fixed during policy training. This design combines the perceptual capabilities of foundation models with a small policy model trained on the resulting observations for action prediction. Evaluations with flow matching policies on simulation benchmarks, together with experiments on two real-world robotic tasks, demonstrate substantial improvements in task success under the evaluated appearance shifts.

Haoran Ding, Liang Ma, Yaxun Yang +7

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE

Visuomotor Policy LearningRobot Policy GeneralizationRobot Learning
AskPDF
Mar 7, 2026

DexKnot: Generalizable Visuomotor Policy Learning for Dexterous Bag-Knotting Manipulation

Knotting plastic bags is a common task in daily life, yet it is challenging for robots due to the bags' infinite degrees of freedom and complex physical dynamics. Existing methods often struggle in generalization to unseen bag instances or deformations. To address this, we present DexKnot, a framework that combines keypoint affordance with diffusion policy to learn a generalizable bag-knotting policy. Our approach learns a shape-agnostic representation of bags from keypoint correspondence data collected through real-world manual deformation. For an unseen bag configuration, the keypoints can be identified by matching the representation to a reference. These keypoints are then provided to a diffusion transformer, which generates robot action based on a small number of human demonstrations. DexKnot enables effective policy generalization by reducing the dimensionality of observation space into a sparse set of keypoints. Experiments show that DexKnot achieves reliable and consistent knotting performance across a variety of previously unseen instances and deformations.

Jiayuan Zhang, Ruihai Wu, Haojun Chen +5

Peking University

Robotic ManipulationGeneralization in Robotic ManipulationVisuomotor Policy Learning
AskPDF
Mar 6, 2026

HarvestFlex: Strawberry Harvesting via Vision-Language-Action Policy Adaptation in the Wild

This work presents the first study on transferring vision-language-action (VLA) policies to real greenhouse tabletop strawberry harvesting, a long-horizon, unstructured task challenged by occlusion and specular reflections. We built an end-to-end closed-loop system on the HarvestFlex platform using three-view RGB sensing (two fixed scene views plus a wrist-mounted view) and intentionally avoided depth clouds and explicit geometric calibration. We collected 3.71 h of VR teleoperated demonstrations (227 episodes) and fine-tuned pi_0, pi_0.5, and WALL-OSS with full fine-tuning and LoRA. Under a unified 50 trials real-greenhouse protocol and metrics spanning completion, pi_0.5 with full fine-tuning achieved success rate of 74.0% with 32.6 s/pick and damage rate of 4.1%. Asynchronous inference-control decoupling further improved performance over synchronous deployment. Results showed non-trivial closed-loop picking with fewer than four hours of real data, while remaining limited by close-range observability loss and contact-dynamics mismatch. A demonstration video is available at: https://youtu.be/bN8ZowZKPMI.

Ziyang Zhao, Shuheng Wang, Zhonghua Miao +1

The Intelligent Equipment Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing 100097, China. · The School of Mechanical Electrical Engineering and Automation, Shanghai University, Shanghai 20044, China.

Robotic HarvestingVisuomotor Policy LearningAgricultural RoboticsLong-Horizon Robotic ManipulationVision-Language-Action Models
AskPDF
Mar 5, 2026

VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory

Imitation learning from human demonstrations has achieved significant success in robotic control, yet most visuomotor policies still condition on single-step observations or short-context histories, making them struggle with non-Markovian tasks that require long-term memory. Simply enlarging the context window incurs substantial computational and memory costs and encourages overfitting to spurious correlations, leading to catastrophic failures under distribution shift and violating real-time constraints in robotic systems. By contrast, humans can compress important past experiences into long-term memories and exploit them to solve tasks throughout their lifetime. In this paper, we propose VPWEM, a non-Markovian visuomotor policy equipped with working and episodic memories. VPWEM retains a sliding window of recent observation embeddings as short-term working memory, and introduces a Transformer-based contextual memory compressor that recursively converts out-of-window observations into a fixed number of episodic memory embeddings. The compressor uses self-attention over a cache of past summary embeddings and cross-attention over a cache of historical observations, and is trained jointly with the policy. We instantiate VPWEM on diffusion policies to exploit both short-term and episode-wide information for action generation with nearly constant memory and computation per step. Experiments demonstrate that VPWEM outperforms state-of-the-art baselines including diffusion policies and vision-language-action (VLA) models by more than 20% on the memory-intensive manipulation tasks in MIKASA and achieves an average 5% improvement on the mobile manipulation benchmark MoMaRT. Code is available at https://github.com/HarryLui98/code_vpwem.

Yuheng Lei, Zhixuan Liang, Hongyuan Zhang +1

School of Computing and Data Science, University of Hong Kong

Robot Policy LearningVisuomotor Policy LearningRobot Imitation LearningEpisodic MemoryDiffusion Policy
AskPDF