Robotic Manipulation

Ask about this topic

Overview

Robotic manipulation studies how machines use sensing and motion to change objects and environments, from picking and assembly to tool use and handling deformable materials. Approaches combine geometric planning and force or impedance control with imitation learning, reinforcement learning, diffusion policies, and vision-language-action models; world models, memory, tactile feedback, and human-video demonstrations are also used. Methods are tested in simulation and on real robots using suites such as LIBERO, RoboTwin, ManiSkill, and RLBench, across clutter, bimanual coordination, mobile manipulation, and precision contact tasks. A central challenge is turning benchmark performance into safe, reliable long-horizon behavior across new objects, cameras, robot bodies, and real-world dynamics, while limiting data collection and inference costs.

Latest papers

CardsList

1,699 papers341 papers in the last four weeks, up 29% on the four weeks before. 2.0% of all new papers.

Jul 31, 2026

Balancing of Humanoid with Object Mass: Trade-off Analyses and Lifting Control

The demand for humanoid loco-manipulation tasks with an object has recently increased, and most existing control approaches for stability in such tasks rely on heuristics or machine-learning techniques. This study rigorously analyzes and exploits the dynamic effects of the object mass on balance stability. By formulating the object mass parameters in the whole-body dynamics with distributed contact wrenches and centers of pressure at the stance contacts, their nonlinear effects on the system momenta and constraints are quantified. The dynamic models and constraints are incorporated into the construction of the balanced state basin/boundary (BSB), a partition of the center-of-mass state space for a biped system to maintain balance in its desired contacts. The implications of the BSB for prediction and control are highlighted using a humanoid robot and an analytically tractable reduced-order mechanism. The BSBs under different conditions of base of support, actuation capacity, and pose provide systematic analyses of the effects of object mass on the balancing capability of a system. In particular, the trade-off relationships between momentum regulation and limiting factors in balancing are characterized, introducing two key quantities of the object: the critical mass, at which the system's balancing capability is maximum, and the transition mass, which activates different limiting factors. In addition, sufficient conditions for imposing balanced states on a trajectory are established and implemented with BSBs as explicit threshold constraints in the whole-body trajectory optimization for stable object-lifting control of the humanoid, demonstrating the lift-and-hold and lift-and-release tasks with distinct mass properties in simulations and experiments.

Hyunjong Song, William Z. Peng, Joo H. Kim

Department of Mechanical and Aerospace Engineering, New York University (NYU), New York, USA · NYU Center for Robotics and Embodied Intelligence (CREO), New York, USA · NYU Center for Urban Science and Progress (CUSP), New York, USA

Humanoid Loco-ManipulationHumanoid Robot ControlWhole-Body ControlRobotic Manipulation
AskPDF
Jul 31, 2026

Automated Straight-line Sewing of Stretchable Fabrics with Different Lengths

Different Length Alignment Sewing (DLAS), which involves stretching the shorter fabric to match the longer one and sewing them together in a straight line, is a challenging task that needs to satisfy several requirements when automating the sewing process. To address the challenges, this research proposes a novel robotic sewing system, Different Length Robotic Sewing System (DLRoSS), which consists of a roller type end-effector, attached to a 6-DoF manipulator. The end-effector composed of active shorter and longer fabric rollers, and a passive press-roller attached to the shorter-fabric roller. Assuming that one end of the two fabric layers are initially positioned under the sewing machine's presser foot, the system automates DLAS by operating in four distinct phases. (P1) Fabric wrapping: Individual fabric layers are picked, held, and wrapped from the other end onto the feed rollers. (P2) Sewing: During the sewing, the shorter fabric is stretched and aligned with the longer fabric in real-time using roller velocity control based on the sewing speed and apriori known length ratio. (P3) Sewing completion: In the final sewing round on the fabric rollers, the press roller is engaged to prevent the stretched fabric from slipping off due to internal tension. (P4) Sewing fabric release: At the end of sewing, the fabric edge moves past the press roller, and the fabric releases from the rollers. Experimental results demonstrate that DLRoSS achieves consistent, high-quality sewing of stretchable fabrics of different materials and lengths.

Bingchen Jin, Akinari Kobayashi, Dipankar Bhattacharya +4

JC STEM Lab of Robotics for Soft Materials, Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China · Centre for Transformative Garment Production, Hong Kong Science Park, Hong Kong SAR, China · Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China

Deformable Object ManipulationRoboticsRobotic ManipulationIndustrial Robotics
AskPDF
Jul 31, 2026

ActFovea: Runtime Safeguarding for VLA Policies via Spatiotemporal Visual-Action Consistency

Vision-language-action (VLA) policies achieve strong performance in robotic manipulation but remain vulnerable to runtime disturbances that break the temporal alignment among visual observations, robot states, and executed actions. We introduce ActFovea, a plug-and-play safeguarding framework that detects and mitigates such failures without retraining or modifying the underlying VLA policy. ActFovea uses robot kinematics, proprioceptive states, and recent actions to construct action-conditioned foveated regions that retain contact-relevant areas and predicted motion corridors while suppressing task-irrelevant visual content. It detects runtime risks by evaluating whether visual motion and observation freshness remain consistent with geometric, proprioceptive, and action transitions. For recoverable disturbances, ActFovea constructs disturbance-specific candidate observations and accepts a recovery only after verifying the resulting action chunk. When stale or replayed observations make reliable recovery impossible, it invokes a bounded safe-failure procedure. In closed-loop evaluations of π0π_0 across multiple LIBERO suites, ActFovea increases success under localized visual overlays from 49.3% to 90.3%, closing 93.7% of the gap to clean performance. It further improves success under action drift and visual delay by 7.0 and 9.8 percentage points, respectively, while preserving clean-task performance. Under frozen-observation replay, ActFovea triggers timely safe failure in all trials, with no unprotected failures. These results demonstrate that spatiotemporal visual-action consistency provides an effective basis for runtime safeguarding of VLA policies.

Wenda Yu, Tianshi Wang, Fengling Li +3

Tongji University · Mohamed bin Zayed University of Artificial Intelligence · Shanghai Artificial Intelligence Laboratory +1

Robot SafetyRobotic ManipulationVision-Language-Action ModelsRobustness of VLA Models
AskPDF
Jul 30, 2026

SemAnCorr: Semantic Anchored Correspondence for Zero-Shot Manipulation Skill Transfer

Transferring manipulation skills across object instances that share functionality but differ in geometry remains a fundamental challenge in robot learning. While recent correspondence methods leverage dense visual descriptors and 3D feature fields, nearest-neighbor feature matching often produces spatially incoherent correspondences that fail to recover the local geometric frames required for reliable skill transfer. We introduce SemAnCorr, a training-free framework that establishes dense correspondence by selecting semantically consistent anchor regions through joint pose-correspondence optimization and propagating these constraints over the object surface using functional maps. The resulting correspondences preserve both semantic consistency and geometric coherence, enabling object-centric manipulation skills to transfer across geometrically diverse instances. We evaluate SemAnCorr on a dense correspondence benchmark built on PartNet-Mobility, achieving 90.8% semantic accuracy in our benchmark evaluation while improving geometric coherence over recent state-of-the-art baselines. Finally, we show that these improvements translate directly into real-world manipulation performance: using a single demonstration, SemAnCorr enables substantially more reliable zero-shot manipulation skill transfer to previously unseen objects than existing correspondence methods. Videos and additional visualizations are available at https://semancorr.github.io .

Xiaoxiang Dong, William Baron, Hongyi Chen +3

Robotics Institute, Carnegie Mellon University, USA. · College of Connected Computing, Vanderbilt University, USA. · College of Engineering, University of California, Berkeley, USA. +1

Semantic CorrespondenceZero-Shot Robotic ManipulationGeneralization in Robotic ManipulationRobotic ManipulationTransfer Learning
AskPDF
Jul 29, 2026

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM

Vision-language-action (VLA) models commonly adopt an LLM-centric V→L→AV \to L \to A pathway, processing visual observations and language instructions through a large language model before predicting robot actions. Although effective, this design incurs substantial computation and memory overhead. In this work, we introduce TurboVLA, a compact VLA architecture built on a direct V+L→AV + L \to A mapping. Instead of using a large language model as the central interface between perception and action, TurboVLA independently encodes visual observations and language instructions, directly exchanges information between them through lightweight bidirectional vision-language interaction, and predicts continuous action chunks with a compact decoder. This simple design directly constructs task-conditioned representations while avoiding the overhead of an LLM-centered execution pathway. On LIBERO, TurboVLA achieves 97.6% average success with only 0.2B parameters, 31.2 ms inference latency, and 0.9 GiB inference VRAM on a consumer-grade RTX 4090. Notably, a 0.4B TurboVLA achieves 88.06% success on RoboTwin 2.0, even matching or outperforming substantially larger VLA policies. These results demonstrate that the simple V+L→AV + L \to A design of TurboVLA can achieve high performance without requiring an LLM-centric execution pathway, offering a new perspective on how vision, language, and action can be connected for efficient robotic manipulation. Code is available at https://github.com/H-EmbodVis/TurboVLA.

Hengyi Xie, Chenfei Yao, Xianjin Wu +4

Huazhong University of Science and Technology

Efficient VLA Model InferenceEfficient VLA ModelsRobotic ManipulationVision-Language-Action Models
AskPDF
Jul 29, 2026

SymmGrid: Super-Scaling On-Robot Learning with Parallelized Symmetries and Egocentric-Exocentric Visual Perception

Deep reinforcement policy learning directly in physical robots (on-robot learning) remains bottlenecked by slow wall-clock training times. We present SymmGrid, a trajectory level augmentation framework inspired by parallelized symmetries that super-scales group transformations to significantly accelerate on-robot learning in both egocentric and exocentric visual setups. We model a Markov Decision Process (MDP) under a symmetry tree, in which state-action pairs have admissible parallelized invariant transformations that yield a geometric grid structure. The state is modelled with ego- or exocentric images and proprioception information. The latter require special treatment, in the form of homographies, to warp visual scenes in line with their corresponding spatial transformations. These parallelized transformations produce a large set of unique symmetric equivalences that populate the replay buffer with diverse and consistent experiences that speed up learning and improve performance. We present extensive training and evaluations performed directly on real robot manipulation contact tasks including peg-insertions, cable routing, and object relocations. Relative to SOTA, SymmGrid achieved wall-clock training convergence speed-ups of 1.37-2.17x, evaluation success rate improvements of 1.09x-1.27x, fastest training convergence times of 16.6, 10.9, and 79.3 minutes respectively. For trajectory wide assessments, we used normalized area under the curve (nAUC) ratios. SymmGrid achieved improvements of up to 2.59x. These results confirm that simple branch symmetries can have an outsized result due to super-scaling and bring us closer to sub-10 minute on-robot learning training in manipulation tasks suitable for arms and humanoids. The project page is available at symmgrid-robot.github.io

Gabe Everett, Brice Gunter, Ryan Vander Stelt +3

Data AugmentationVisuomotor Policy LearningRobotic RLRobotic Manipulation
AskPDF
Jul 29, 2026

From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence

The key bottleneck in embodied AI is not model architecture but data. Although billions of human manipulation videos exist online, robots cannot directly learn from them due to the embodiment gap between human morphology and robot hardware. We introduce Pegasus, a low-resource framework that bridges this gap by translating human demonstrations into robot-learnable data through structured knowledge transfer. Instead of relying on raw video prompts, Pegasus constructs a graph-based intermediate representation: a Task Graph extracted from human videos is transformed through Affordance and Constraint Graphs into a Robot Planning Graph for robot-conditioned video generation. A hierarchical affordance latent space models the relationship between object states, affordances, and tasks, enabling generalization beyond object identities. A closed-loop physics verifier further filters invalid generations using kinematic feasibility, collision constraints, and joint limits. We evaluate Pegasus across a range of egocentric manipulation benchmarks, including GTEA Gaze+ and EPIC-KITCHENS-100, and diverse robot embodiments, assessing Task Correctness, Executability, State Consistency, and Learnability. Results demonstrate reliable cross-embodiment translation and show that robot data generation can be reframed from a hardware collection problem into a scalable, low-resource knowledge transfer problem.

Jia Luo

Huazhong University of Science and Technology

Synthetic Data Generation for RoboticsRobotic ManipulationAgent Skill LearningControllable Video GenerationCross-Embodiment Transfer
AskPDF
Jul 29, 2026

Practice Makes Policies: Bootstrapping and Consolidating Robotic Capabilities from Zero Human Demonstrations

General-purpose robotic manipulation requires robots to perform diverse tasks in open-world environments while improving their skills over time. Despite recent progress in robotic manipulation, existing systems still primarily acquire manipulation skills in a static manner, where capabilities are learned for specific tasks or settings rather than adaptively evolving through physical interaction. Resembling how repeated practice enables humans to develop muscle memory, advanced manipulation proficiency requires an autonomous capability evolution mechanism that allows robots to progressively transform interaction experiences into increasingly effective manipulation abilities. To this end, we propose HERO, a self-improving hierarchical embodied agent that enables autonomous capability evolution from zero human demonstrations. HERO organizes heuristic reasoning, exemplar reuse, and reflexive execution into a unified orchestration framework, allowing robots to autonomously bootstrap manipulation experience, rapidly accumulate reusable behaviors through experience transfer, and progressively consolidate recurring interactions into efficient closed-loop visuomotor policies. By tightly coupling autonomous data collection with task execution, HERO continuously expands and dynamically schedules manipulation capabilities according to different stages of experience accumulation and execution requirements. Extensive experiments demonstrate that HERO substantially reduces human intervention during robotic data collection while achieving robust manipulation across diverse tasks, providing a promising path toward self-improving robotic systems.

Jialiang Li, Yuhan Wang, Haojun Li +5

Shanghai Jiao Tong University · University of Sussex

Continual Robot LearningRobotic Data CollectionRobotic ManipulationRobot Policy GeneralizationRobot Learning
AskPDF
Jul 29, 2026

Explicit Kinematic Guidance from Analytic Concepts for Vision-Language-Action Models

Current Vision-Language-Action (VLA) models rely mainly on 2D inputs, neglecting the rich object structural information and commonsense knowledge inherent in the 3D physical world. This deficiency restricts their spatial awareness and adaptability for complex, high-precision manipulation. To bridge this crucial gap, we construct a Concept Expert module for VLA to build executable Analytic Concepts that represent objects as explicit, programmatic blueprints. Our mechanism operates in two synergistic phases: First, prior to VLA inference, the Concept Expert leverages 3D information from Vision Foundation Models (VFMs) to estimate the initial kinematic and structural parameters. Second, throughout the manipulation process, the VLA model utilizes its inherent capability to dynamically track the dynamic concept parameters, continuously aligning them with observational changes to ensure persistent accuracy. Once established, the Analytic Concepts provide explicit, high-quality guidance for VLA fine-tuning through (1) dense, programmatic manipulation rewards and (2) precise spatial guidance. This formulation allows VLA models to learn physically grounded interaction behaviors while maintaining end-to-end learning flexibility. Our experimental results show consistent improvements in success rate and learning efficiency across supervised and reinforcement learning settings, demonstrating the effectiveness of structured, concept-based guidance for VLA post-training.

Mingyang Sun, Jiude Wei, Xiujian Liang +4

Zhejiang University, Hangzhou, China · Shanghai Innovation Institute, Shanghai, China · Westlake University, Hangzhou, China +2

Robotic ManipulationVision-Language-Action Models
AskPDF
Jul 29, 2026

It's Not Just More Demos: Counterfactual Action Sensitivity Coverage for Data-Efficient Robust Robot Imitation

Visuomotor imitation learning has demonstrated success for manipulation tasks. However, the trained policies remain brittle to visual nuisances', with even minor task-preserving variations such as lighting, distractions or changes in colour result in heavy degradation of the trained policy's performance. While increasing data diversity can improve robustness, it is unclear which additional demonstrations are informative for a particular trained policy. We propose Counterfactual Nuisance Behaviour Cloning (CFNBC), an offline data-selection framework for targeted robustness repair. Starting from a nominal policy trained on clean' demonstrations, CFNBC generates paired clean and nuisance observations that preserve the expert action, then measures action drift: the change in the policy's predicted action under a nuisance that should not alter the desired behaviour. This provides a policy-specific sensitivity signal for selecting a compact, response-diverse repair set from a larger candidate pool, without requiring rollout success labels or online policy execution. We show in MuJoCo bimanual cube transfer and SimplerEnv cube stacking that action drift correlates with nuisance-induced failure, and that response-guided repair with only 2020--3030 selected candidates substantially outperforms matched-budget random selection while approaching the performance of much larger random repair budgets. These results support a data-centric view of robustness repair: the most useful data are not necessarily the most numerous, visually diverse, or obviously difficult, but the examples that cover fragile response modes of the current policy.

Giovanni D'urso, Kaushik Roy, Nicholas Lawrance +1

CSIRO Robotics, CSIRO, Australia

Visuomotor Policy LearningImitation LearningRobotic ManipulationRobot Imitation Learning
AskPDF
Jul 28, 2026

Reeling It In: Flexible Needle Pick Up via Thread Manipulation for Autonomous Suturing

Suture-needle pickup is necessary for autonomous suturing, as a needle can be unexpectedly dropped or strategically released to adjust the grasping configuration. Current methods for autonomous needle pickup typically guide a robot to move straight toward the needle and grasp it, limited to conditions where the needle is observable and directly approachable. In addition, grasping the needle lying on tissue can lead to the robot pinching nearby tissue or the needle jumping around due to its slippery surface, posing potential safety issues. This work proposes an autonomous framework that uses a suture thread as an assistive tool for indirect needle pickup, avoiding unnecessary tool-tissue contact and enabling pickup even when the needle is occluded or inaccessible. The framework spans the entire workflow, including thread and tissue reconstruction, safe grasp-point selection, stable thread lifting, and bimanual thread-following until securing needle grasping. The robot policies account for visual uncertainty to maximize robustness in real-world environments. We evaluate the proposed framework on a da Vinci Research Kit under various real-world conditions. The results demonstrate robust performance even with a challenging thread configuration or a non-approachable needle, closing the gap in applying autonomous robot policies to unstructured suturing environments.

Emma Huang, Zih-Yun Chiu, Neelay Joglekar +2

Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093 USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218 USA. · CMU Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213 USA. +1

Robotic GraspingRobotic ManipulationSurgical Robotics
AskPDF
Jul 28, 2026

MoMo: Dial Motion Mode in Robot Manipulation with Spatiotemporal Action Tokenization

To operate effectively across diverse contexts, robots must not only perform manipulation tasks accurately but also adapt how their actions unfold to the task, object, and interaction setting. We ask whether this execution-level variation can be learned as a reusable behavioral factor shared across tasks. We present MoMo, a two-stage imitation-learning framework consisting of a spatiotemporal action tokenizer and a behavior-cloning transformer that takes task and a continuous motion-mode condition as inputs. Across six real-robot manipulation tasks, varying this condition produces steady, dynamic, and intermediate behaviors that human raters can distinguish and that differ in joint speed, acceleration, and end-effector approach pitch. On tasks demonstrated in only one mode, MoMo transfers the unseen requested mode while largely preserving task success. Together, these results provide evidence of compositional generalization to unseen task--mode combinations and show that motion mode can be reused across tasks to control how a manipulation skill is performed.

Yuhan Hu, Hugues Thomas, Peide Huang +3

Apple

Robot Skill LearningGeneralization in Robotic ManipulationGoal-Conditioned Robotic ManipulationRobotic ManipulationRobot Imitation Learning
AskPDF
Jul 28, 2026

πR2π\mathbf{R}^2: Reactive Real-time Flow Policies

Generalist manipulation policies increasingly take the form of action-chunking flow policies built on large pretrained backbones. Such chunks run open-loop, so the policy cannot react to sensory input arriving mid-execution, sacrificing reactivity. Replanning more often would restore it, but the perception-to-action pipeline (a large backbone plus multiple denoising steps) is too slow: this latency forbids frequent replanning and leaves committed actions stale, making such policies ill-suited for dynamic, closed-loop control. We present πR2π\mathbf{R}^2, which makes these policies reactive and real-time while retaining large backbones, expressive multi-modal policies, and multi-action prediction. Built on the per-position noise schedule of diffusion forcing, πR2π\mathbf{R}^2 contributes two ideas. First, it splits conditioning into a fast channel (proprioception, fresh every tick) and an asynchronously updated slow channel (vision-language features), so the policy reacts to proprioception within a chunk while tolerating stale vision. Second, a latency-adaptive flow schedule treats in-flight actions as inpainting conditioning and emits actions in one denoising step per call, letting one trained model adapt to varying hardware latency. Requiring minimal modification to existing architectures, πR2π\mathbf{R}^2 can be finetuned from a pretrained policy: applied to GR00T-N1.7 on a real xArm6+XHand platform, it replans closed-loop roughly 4×4\times faster than the base policy (~2525Hz on an A5000 GPU), acting on a fresh observation every 4040ms. Across simulation and real-world manipulation tasks, πR2π\mathbf{R}^2 improves the success rate by up to 23%23\% in simulation and 30%30\% in the real world over the strongest baseline. Project page: https://pi-r2-flow.github.io/

Sungjae Park, Shubham Tulsiani

Carnegie Mellon University

Robotic ControlReal-Time Robot ControlFlow MatchingRobotic ManipulationPolicy LearningVisuomotor Policy Learning+2
AskPDF
Jul 28, 2026

S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Information

Acoustic information provides rich cues about object location, material properties, and changes caused by contact or motion. This paper introduces a new set of acoustic-aware manipulation tasks for imitation learning, in which robots must use auditory cues to determine manipulation targets. These tasks require sound source localization and identification for active exploration in robotic manipulation. Also, we propose a multimodal imitation learning framework, Spatial-Spectral Audio Action (S2A2), that integrates visual features with acoustic spatial and acoustic signal information for the acoustic-aware manipulation tasks. We implemented S2A2 models that integrates policies such as ACT, Diffusion Policy, VQ-BeT, and π0π_0, into our framework. Simulation experiments showed that the proposed method is the most effective for tasks requiring both position and timbre. Furthermore, real-robot experiments confirm the applicability of the proposed tasks and framework to real-world manipulation.

Kaneyoshi Hiratsuka, Benjamin Yen, Ryosuke Kojima

Kyoto University · RIKEN, Institute of Science Tokyo · Kyoto University, RIKEN

Imitation LearningRobotic ManipulationRobot Imitation LearningSpatial Audio
AskPDF
Jul 28, 2026

DC-WAM: Dynamic-Centric Visual Supervision and Reasoning for World-Action Models

World-Action Models (WAMs) augment robot policies with future visual prediction, but it remains unclear what the visual modality should learn for control. While photorealistic future prediction provides dense supervision, it also incurs substantial computation and can allocate capacity to texture, illumination, and background variations that are only weakly related to action selection. Recent efficient WAM variants suggest that the main benefit of the video branch may not lie in the rendered future itself, but in the control-relevant visual representations induced during training. In this work, we revisit future video prediction from a dynamic-centric perspective and ask whether an existing RGB-based WAM can be redirected from appearance-dominated reconstruction toward interaction-induced visual dynamics without introducing additional modality-specific predictions or online inputs at deployment. We propose DC-WAM, a dynamic-centric WAM framework that redistributes supervision and computation in the RGB video branch. At the supervision level, DC-WAM combines temporal-difference flow matching with trajectory-guided weighting, emphasizing dense temporal changes and localized regions where the gripper, manipulated objects, and contact areas move. At the reasoning level, DynaRoute predicts token-wise dynamic relevance and converts it into an attention bias, guiding the model toward control-relevant future tokens. Experiments in simulation and on real-world manipulation tasks show that DC-WAM consistently improves policy performance, especially under out-of-distribution perturbations in lighting, object appearance, and background texture.

Haoyuan Ji, Lingxiang Fan, Shang Su +4

Tsinghua University · Dense-AI · University of Michigan

World Models for RoboticsVisuomotor Policy LearningRobotic ManipulationWorld Action Models
AskPDF
Jul 28, 2026

Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design

An often overlooked factor of robot manipulation performance is the embodiment of the robot itself. Motivated by this problem, we study motion-conditioned robot co-design, where the goal is to generate complete robot designs that track target end-effector trajectories (from human demonstrations) while optimizing user-defined rewards. We introduce Transformer Transformer, a diffusion transformer trained on RoboTokens, a unified tokenization of robot embodiments, states, and actions. The same architecture can be used across embodiment spaces (e.g., wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross embodiment controller). Rather than overfitting to one reward function, Transformer Transformer is a dynamics model, whose reward-agnostic state and action predictions can be converted into reward-specific value predictions. These value predictions are used to steer embodiment diffusion towards high value robot designs, through a procedure we call Dynamics Self-Guidance. Experiments across multiple design spaces show zero-shot optimization of unseen rewards and trajectories, improving performance and runtime over the evolutionary baseline. Finally, we fabricated an optimized ALOHA design, which reduced tracking error by over 70% compared to the original design.

Huy Ha, C. Karen Liu, Shuran Song

Stanford University · Columbia University

Robot Morphology-Control Co-DesignDiffusion TransformerCross-Embodiment Robot LearningRoboticsRobotic ManipulationRobotic Optimization
AskPDF
Jul 28, 2026

Tri-Manual Visuomotor Imitation Learning of Robot Policies

Bimanual teleoperation provides an effective way to collect robot demonstrations, but it assumes that the operator and robot have matching numbers of simultaneous control channels. This assumption breaks for tri-manual systems: the robot can coordinate three arms concurrently, whereas a single operator can continuously control only two. Pairwise mode switching may therefore record otherwise independent motions sequentially, causing behaviour cloning to reproduce delays imposed by the interface rather than required by the task. We present TriManPolicy, a tri-manual imitation learning system that allows one operator to demonstrate behaviours for three arms. Its central component is Dependency-Aware Tri-Arm Scheduling (DATS). The key idea is to preserve the demonstrated arm motions while reconsidering when they occur. DATS retimes demonstrations offline by preserving local sensorimotor segments of fixed duration and repositioning them according to constraints on task order and arm usage that are reviewed by a human. The resulting data train a single synchronous policy for all three arms, while deployment requires neither the dependency graph nor the scheduler. Across six challenging tasks performed in the real world, policies trained on demonstrations retimed by DATS exhibit more efficient coordination while maintaining comparable observed task success. Offline analysis further shows that DATS changes the supervision across arms rather than merely removing idle periods. Project videos and additional material are available at https://aus.bot/trimanpolicy/.

James Zhao, Mingyuan Ba, Weiming Zhi

School of Computer Science and 2Australian Center For Robotics, The University of Sydney, Australia · College of Connected Computing, Vanderbilt University, TN, USA

Robot TeleoperationVisuomotor Policy LearningRobot Motion GenerationRobotic ManipulationRobot Imitation Learning
AskPDF
Jul 28, 2026

When Does Legacy Data Start to Help? Emergent Transfer in Cross-Configuration Robot Learning

Robotic hardware evolves over time, but demonstration data is often tied to a specific sensor and actuator configuration. This raises a practical and underexplored question: when does legacy data begin to benefit an upgraded robot? We study this question on a wheeled humanoid platform across two hardware generations, where both the camera and gripper are changed while the overall morphology remains fixed. Contrary to the common assumption that more cross-configuration data is always helpful, we observe a grokking-like transition: legacy data remains ineffective until the upgraded configuration acquires a minimum level of task competence, after which co-training gains rise sharply before diminishing near saturation. We hypothesize that this task-dependent transition is governed by a transfer threshold and characterize the resulting three-phase pattern. Across real-robot manipulation tasks, we observe all three phases: no measurable benefit at low competence (10.0%→10.0%10.0\% \rightarrow 10.0\%), a sharp gain after crossing the threshold (23.3%→86.7%23.3\% \rightarrow 86.7\% on flower insertion), and diminishing returns at high competence (85.0%→93.3%85.0\% \rightarrow 93.3\% on pen insertion). We provide a theoretical account based on gradient alignment and residual policy uncertainty, and derive a phase-aware rule for deciding when to collect more new-hardware data and when to reuse legacy demonstrations. We further validate this three-phase pattern on a mobile dual-arm watering task, with results consistent with our predictions.

Tao Wang, Hudson Hou, Yingdong Hu +7

Huazhong University of Science and Technology · Spirit AI · Peking University +3

Robot Skill LearningRobotic Data CollectionCross-Embodiment Robot LearningRobotic ManipulationLearning from DemonstrationTransfer Learning
AskPDF
Jul 28, 2026

A Causality-aware Infer-diagnose-refine Framework for Test-time Modality Adaptation in VLA Models

Vision-language-action (VLA) models predict sequential actions to execute tasks specified by language instructions, conditioned on visual observations and proprioceptive states. However, how to fuse modalities in VLA models remains an open problem, since robot manipulation involves dynamic phases, such as long-distance movements and close-range interactions, in which the importance of visual observations may vary over time. In this paper, we propose an infer-diagnose-refine (IDR) framework, a model-agnostic framework that can be integrated with diverse VLA architectures for refining action predictions at test time. IDR first infers actions under factual and counterfactual scenarios of visual observations, and then diagnoses the causal effects of visual observations as the estimated dynamic importance, which is finally used to refine the action predictions in a training-free manner. We further design a causality-aware action refiner to realize the IDR framework, including zero-padding interventions for inferring counterfactual actions, norm-based quantification for diagnosing causal effects, and gated residual fusion for refining actions. Extensive experiments on both simulation benchmarks and real-world tasks show improvements in overall performance across multiple VLA backbones, demonstrating the efficacy of dynamically adjusting visual importance at test time.

Haoyu Zhang, Yuwei Wu, Jin Chen +6

1Beijing Institute of Technology · 2AInnovation Co. Ltd. · 3Beijing Innovation Center of Humanoid Robotics

Robotic ManipulationVision-Language-Action ModelsTest-Time Adaptation of VLA ModelsVLMs for Robotics
AskPDF
Jul 28, 2026

Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations

Successfully automating dexterous, long-horizon robotic manipulation requires frameworks capable of both high-level reasoning and fine-grained execution. Traditional task and motion planning (TAMP), while excellent at symbolic planning, is often brittle in contact-rich operations. Simultaneously, imitation learning (IL), while effective in manipulation tasks with visual feedback, is limited by its low capability in spatial generalization and multi-stage operation. To reconcile their complementary strengths and limitations, we propose DR-LfD (Decomposed and Reorganized Skills Learned from Demonstrations), a framework that seamlessly integrates visuomotor policies into a TAMP-gated decision-making system. Based on contact relationships, DR-LfD decomposes human demonstrations into atomic skills, which are reproduced as visuomotor policies or object-centric primitives. The initiation, termination, and constraints of the visuomotor policies are carefully modeled and implemented in a TAMP-compatible form, enabling reorganization of skills learned from different sources. DR-LfD transforms the learning problem from one requiring exponential demonstration data over possible skill sequences to one whose demonstration burden scales with the number of distinct skill types, with limited data for each skill. Through comprehensive real-world and simulation benchmarking across diverse scenarios, we demonstrate the strong performance of DR-LfD on tasks involving multiple steps, unseen setups, and physical constraints. Project website: https://dr-lfd.github.io/DR-LfD-website.

Yizhou Chen, Hang Xu, Dongjie Yu +7

The University of Hong Kong, Hong Kong · JD.com (JingDong) · Nanyang Technological University (NTU), Singapore +3

Long-Horizon Robotic ManipulationRobot Skill LearningGeneralization in Robotic ManipulationVisuomotor Policy LearningRobotic ManipulationLearning from Demonstration
AskPDF
Jul 27, 2026

Data Pyramid for Embodied Manipulation: A Survey

Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of their data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work paves the foundation for the design of next-generation embodied systems.

Yifan Ye, Yankai Fu, Yaoxu Lv +26

PKU · NTU · HKUST +8

Robotic Data CollectionRobot Foundation ModelsMultimodal PretrainingRobotic ManipulationRobot Learning
AskPDF
Jul 27, 2026

NEO: NeRF It Once, Edit It Many Times for Continuous Object Manipulation

In this paper, we present NEO, a unified framework providing language-guided NeRF editing for robotic manipulation. Our paper introduces (i) a language-guided object removal that combines neural field resampling with multiview-consistent progressive inpainting, (ii) a direct NeRF weight editing method utilizing knowledge distillation, composing original and edited NeRFs via a teacher-student model, enabling coherent modeling of future scene states before a robot executes an action, and (iii) the first benchmark (NEO-Dataset) for quantitatively evaluating NeRF scene editing methods suitable for robot manipulation. We show that our approach outperforms state-of-the-art baselines in scene editing tasks, including object removal and pick-and-place robotic experiments, yielding visually coherent and geometrically consistent edits that reduce artifacts commonly introduced by prior methods.

Mikołaj Zieliński, David Hall, Dominik Belter +1

Institute of Robotics and Machine Intelligence, Poznan University of Technology, 61-131 Pozna´n, Poland · CSIRO Robotics, CSIRO, Australia

3D Scene EditingRobotic ManipulationRobot Manipulation BenchmarksNeural Radiance Fields
AskPDF
Jul 27, 2026

PAC-DP: PAC-Bayesian Diffusion Policy Learning

Diffusion Policies (DPs) are able to perform complex manipulation tasks. However, DPs are typically trained by minimizing a denoising objective, which provides limited control over generalization in the finite-data regimes common in robotics. In this letter, we propose PAC-DP, an approach that increases the performance of DPs in robotic manipulation tasks. By modeling the DP as a Bayesian neural network, and defining a PAC-Bayes generalization bound, we derive a novel training objective that augments the standard denoising loss with a Kullback-Leibler divergence regularizer between the posterior and prior parameter distributions. From the theoretical perspective, our approach provides a principled approach to regularize the training of DPs without significantly increasing the training time. From the practical point of view, experimental results demonstrate improved denoising performance, lower variational negative log-likelihood, and higher success rates across multiple robotic manipulation benchmarks. Crucially, the largest improvements are observed in low-data training regimes and complex tasks, establishing PAC-DP as a theoretically grounded framework for robot policy learning.

Mohammad Hasan Yeganegi, Dian Yu, Andrea Del Prete +2

Department of Industrial Engineering, University of Trento, Italy. · Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM), Germany.

PAC-Bayesian Generalization BoundsRobot Policy LearningDiffusion PolicyRobotic ManipulationPolicy Learning
AskPDF
Jul 27, 2026

WorldDiT: A Unified Diffusion Architecture for World and Action Modeling

Many recent robot policies pursue stronger control by using large pretrained vision-language models (VLMs) as the action backbone. We introduce WorldDiT, a unified diffusion transformer architecture that couples action generation with visual world modeling and achieves strong performance without a large pretrained VLM action backbone. During training, a single diffusion transformer generates continuous action chunks and predicts normalized RGB patch targets from future camera frames. Across four LIBERO simulation suites, WorldDiT lies on the reported Pareto frontier for total model parameters and mean success among methods reporting all four suites. These results provide a strong sub-billion-parameter baseline for future scaling studies.

Sen Wang, R. Gnana Praveen, Bidhan Roy +1

Bagel Labs (bagel.com)

Diffusion TransformerDiffusion PolicyRobotic ManipulationEmbodied World Models
AskPDF
Jul 26, 2026

Try Once, Then Optimal: De-Redundified Procedure Memory for Cross-Episode Exploration Amortization

Manipulating objects with hidden internal state, such as a latched microwave, forces a robot to probe before it can act. Yet a robot that has solved an instance once re-runs the same probes whenever it encounters that instance again, because existing cross-episode memories target task success and organize reuse around states, not the object or the cost of re-exploring it. We present Instance-Oriented Memory (IOM), an object-centric framework that amortizes this exploration: from a single encounter that uncovers the hidden state, whether or not it succeeds, IOM records a short procedure for manipulating that instance, keys it on the object's identifiable features, and injects it as a soft bias on a procedure-conditioned policy. A later encounter recognizes the object and recalls its procedure instead of re-exploring. We instantiate this distillation with an off-the-shelf vision-language model (VLM) that parses each encounter into the procedure without task-specific training. Across four articulated-object tasks, two in simulation (microwave, door) and two on a real robot (bottle, cabinet), an oracle procedure memory cuts manipulation operations by 16-30% over re-exploration at non-regressing success, and the VLM instantiation recovers 69-88% of that saving out of the box. Because the procedure is a soft bias on a feedback-driven policy, an incorrect memory is recovered from rather than obeyed: success holds even when a retrieved procedure is wrong, as for ≈\approx12% of door instances. Across all tasks the benefit is purely one of efficiency: success never regresses, and on the real robot even improves. Code will be released upon acceptance.

Haizhou Ge, Haochen Ouyang, Zhixing Chen +6

1Tsinghua University. · 3Northeast Agricultural University. · 2DISCOVER Robotics.

RL ExplorationRobotic ManipulationArticulated Object ManipulationMemory-Augmented Agents
AskPDF
Jul 26, 2026

LEACL: LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks

Long-horizon manipulation tasks pose significant challenges for reinforcement learning due to sparse reward signals and long horizons. Automatic curriculum learning (ACL) has been proposed to tackle these challenges by progressively training agents on a sequence of tasks, from easier to more difficult. However, the success of ACL depends heavily on task-dependent specifications-such as well-defined task parameter spaces and difficulty measures-which are often manually crafted and difficult to generalize across diverse tasks. Recent advances in large language models (LLMs) offer a promising alternative by enabling the decomposition of complex tasks into meaningful subtasks using the LLMs' web-scale common-sense knowledge. This decomposition can provide a natural curriculum structure for efficient learning of long-horizon tasks. However, existing LLM-based methods typically rely on hand-designed dense reward functions to learn each subtask, which can introduce bias and still requires significant human supervision. In this work, we propose LLM-enhanced automatic curriculum learning (LEACL), a framework that integrates LLMs and ACL to address these limitations. Specifically, LLMs are used to both decompose tasks into subtasks and to generate task-dependent specifications for each subtask. These specifications are then used by ACL algorithms to guide learning using only sparse reward signals, eliminating the need for dense reward design. We evaluate LEACL on five long-horizon manipulation tasks from the LIBERO benchmark. LEACL achieves better asymptotic performance in terms of the success rates compared to human-designed dense rewards.

Faraz Heravi, James Ouyang, Zifan Xu +3

The University of Texas at Austin · Nanyang Technological University · Sony AI

LLM-Guided RLRobotic ManipulationSparse-Reward RL
AskPDF
Jul 25, 2026

The Curse of Precision: A Data Scaling Law for High-Precision Robotic Manipulation

While scaling laws for imitation learning have primarily focused on generalization in open-world settings, the relationship between data and precision in closed-world tasks like robotic assembly remains largely unexplored. This paper systematically investigates this relationship and introduces a novel scaling law. We find that to achieve a fixed success rate, the required number of demonstrations NN grows super-exponentially as the target precision PP approaches a limit cc. This relationship is accurately captured by the model log⁡(N)∝1/(P−c)\log(N) \propto 1/(P-c). Crucially, we reveal that the limit precision cc is not a static physical constant of the task but an emergent property of the entire agent system, including its sensors and expert policy. Through experiments on canonical manipulation tasks, we validate this law and demonstrate that improving system components, such as adding a wrist camera or using a more effective expert, measurably lowers cc, thus expanding the system's achievable precision. Our work provides a new theoretical framework for precision in robotics and a quantitative metric to evaluate system capabilities. Furthermore, these findings provide a practical methodology for guiding the development and debugging of high-precision manipulation systems.

Cuijie Xu, Yuanfan Xu, Min Xue +5

Department of Electronic Engineering, Tsinghua University, Beijing, China. · OpenMind (WuHu) Robotics Co., Ltd., Wuhu, China. · Institute for Embodied Intelligence and Robotics, Tsinghua University.

Robotic ManipulationRobot Imitation LearningRobotic Assembly
AskPDF
Jul 24, 2026

MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution

Long-horizon embodied tasks require policies that execute many dependent actions before task success can be observed. Representing policies as executable control pro- grams (code-as-policy) enables their decision logic to be inspected and revised after rollout evaluation. Revised programs can then be executed and compared by rollout performance, framing policy improvement as execution-guided program search. Evo- lutionary methods driven by large language models (LLMs) provide a natural mecha- nism for this search by generating variants and selecting high-performing candidates. However, existing approaches primarily select among independently generated vari- ants and lack a sequential local improvement phase. We introduce MEMENTO, a memory-guided single-elite memetic framework for code-as-policy evolution. ME- MENTO first evolves a rollout evaluator that maps policy rollouts to scalar fitness and structured feedback metrics. Fitness selects accepted candidates and the next elite, while feedback metrics condition policy proposals generated by memory-guided hill-climbing, macro-mutation, and crossover. We evaluate MEMENTO on two long- horizon embodied domains: Robosuite Franka Tower-of-Hanoi manipulation and AI2- THOR household interaction. MEMENTO outperforms Eureka and REvolve, adapted as code-as-policy evolutionary baselines, in task success and generalization to held- out Robosuite object configurations and unseen AI2-THOR scenes. Ablations show that zero-shot generation and unevolved evaluators fail to solve either domain, and that removing policy-search branches reduces performance. Finally, we deploy the best-evolved Robosuite policy on a physical Franka robot, demonstrating the feasibil- ity of sim-to-real transfer of the evolved code-as-policy. Code, prompts, and videos are available at: https://github.com/sygkounas/MEMENTO.

Alkis Sygkounas, Victor Aregbede, Amy Loutfi +1

Center for Applied Autonomous Sensor Systems (AASS), Örebro University, Sweden · Örebro University, 70281 Örebro, Sweden · Linköing University, 58183 Linköping, Sweden

Metaheuristic OptimizationLarge Language Model-Guided OptimizationLong-Horizon Agent EvaluationRobotic Manipulation
AskPDF
Jul 24, 2026

Robot-Factored World Models via Robot Rendering

Action-conditioned video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes: they are first realized into robot motion by the robot body and controller, and the scene then responds through contact and object motion. Conditioning directly on action commands asks the world model to learn the realization process itself, while conditioning on logged future states leaks the interaction outcomes it is meant to predict. We propose robot-factored world models, which move two robot-specific factors outside the world model. First, action realization: each command is rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory, a middle signal that avoids both action-realization learning and future-state leakage. Second, robot rendering: this nominal trajectory is rendered through the robot URDF, factoring the robot's geometry, kinematics, and appearance out of the model and into explicit rendered robot geometry. To resolve depth ambiguity, we pair end-effector depth with scene depth, giving geometric cues for contact and occlusion beyond image-plane overlap. Together, camera-aware static RGB/depth context and rendered robot geometry form a shared visual world-model interface that stays consistent across viewpoints and robot embodiments, so the model sees the action only as visible robot geometry and learns how objects respond to it. Our experiments show that the rendered interface outperforms vector-conditioned baselines and generalizes to unseen robot embodiments at inference. We further demonstrate that our model generates robot manipulation videos from human demonstrations by retargeting and rendering the hand motion as robot geometry.

Byungjun Kim, Taeksoo Kim, Hyunsoo Cha +1

Seoul National University · RLWRLD

Action-Conditioned World ModelsCross-Embodiment Robot LearningRobotic ManipulationEmbodied World Models
AskPDF
Jul 24, 2026

Impedance Control of Ship-Borne Manipulators via Optimization-based Task-Space Inverse Dynamics

Ship-borne manipulators operating in maritime environments are subject to stochastic wave-induced base motions that introduce kinematic disturbances and dynamic coupling, degrading trajectory tracking accuracy and complicating safe, contact-rich manipulation. This paper proposes a torque-level optimization-based control framework that integrates high-precision trajectory tracking with task-space impedance for ship-borne manipulators. The controller is formulated using task-space inverse dynamics (TSID) and solved via quadratic programming to explicitly compensate for the dynamic coupling introduced by base motion. To enable accurate feedforward compensation, an error-state Kalman filter (ESKF) is developed to estimate the base state by fusing inertial measurements with end-effector pose feedback. The framework is validated in simulation and real-world experiments using a 7-DOF manipulator mounted on a 6-DOF Stewart platform. The proposed method reduces real-world end-effector position tracking error by over 25.7% compared with the best baseline. Furthermore, the controller enables dynamic peg-in-hole insertion with 1~mm clearance under base motion, increasing the success rate while reducing average contact forces by 45%, demonstrating precise and compliant manipulation in contact-rich environments.

Lingxiao Meng, Bi-Ke Zhu, Xuheng Gao +5

Robot Trajectory TrackingContact-Rich Robotic ManipulationImpedance ControlRobotic Manipulation
AskPDF