Robotic Manipulation

Ask about this topic

Overview

Robotic manipulation studies how machines use sensing and motion to change objects and environments, from picking and assembly to tool use and handling deformable materials. Approaches combine geometric planning and force or impedance control with imitation learning, reinforcement learning, diffusion policies, and vision-language-action models; world models, memory, tactile feedback, and human-video demonstrations are also used. Methods are tested in simulation and on real robots using suites such as LIBERO, RoboTwin, ManiSkill, and RLBench, across clutter, bimanual coordination, mobile manipulation, and precision contact tasks. A central challenge is turning benchmark performance into safe, reliable long-horizon behavior across new objects, cameras, robot bodies, and real-world dynamics, while limiting data collection and inference costs.

Latest papers

CardsList

1,704 papers341 papers in the last four weeks, up 29% on the four weeks before. 2.0% of all new papers.

Sep 20, 2026

Manipulation Feasible Navigation Among Movable Obstacles with Discrete Contact Pushing

In environments with large movable obstacles, detour-only navigation can be inefficient or even infeasible, while obstacle interaction requires reasoning about navigation benefit, feasible placement, and executable manipulation. We present a hierarchical navigation among movable obstacles (NAMO) framework for mobile manipulators. At the high level, the planner identifies key blocking obstacles from reference paths and searches for relocation plans that jointly satisfy geometric, manipulation, and downstream navigation constraints. When direct relocation is hindered by other movable objects, a large language model (LLM) is selectively invoked to infer auxiliary manipulation dependencies, which are then verified by deterministic geometric planning. To execute the resulting relocation goals, we define discrete contact modes on the surfaces of box-shaped obstacles and select contact faces and regions online based on position and orientation errors, enabling straight, side, and corner pushing through contact switching. A recurrent reinforcement-learning policy coordinates the mobile base and manipulator to track tool center point (TCP) targets while preserving end-effector reachability during sustained pushing. Simulation and real-robot experiments demonstrate feasible navigation-manipulation in detour, single- and multi-obstacle relocation, and dependency-constrained scenarios, validating the framework for interactive navigation with large non-graspable obstacles. The open-source project is available at https://cloudytosunny.github.io/NAMO_DCPushing/.

Shaohu Wang, Aiguo Song, Yulong Yuan +3

State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Robot Perception and Control Technology, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China

Robot Motion PlanningMotion PlanningRobotic ManipulationContact-Rich Robotic Manipulation
AskPDF
Sep 20, 2026

SCULPT-VLA: Learning Structured Control through Staged Action Grounding

Vision-language-action (VLA) policies increasingly incorporate structured intermediate supervision beyond action labels. Yet specifying what an intermediate representation should encode leaves open how action prediction learns to depend on it. We introduce SCULPT-VLA, a policy that learns structured control through staged action grounding. Its action-conditioning state comprises complementary factors for task progression, scene dynamics, and spatial grounding. Training first forms these factors with teacher scaffolds, then grounds coarse action prediction through their composition as scaffold inputs are withdrawn. Direct perceptual access is subsequently restored for continuous refinement, combining the learned state with perceptual detail. The curriculum separates learning to condition actions on structure from refining continuous control. Deployment requires neither teachers nor discrete-action autoregression. SCULPT-VLA achieves higher average success than shared-backbone baselines on LIBERO, SimplerEnv-WidowX, and RoboTwin 2.0 Full. On SimplerEnv-WidowX, final success is 83.5%, versus 71.3% when Stage-II action learning directly accesses vision and language. Across four physical robot tasks, average success under the tested distribution shifts reaches 58.1%, compared with 45.6% for π0.5π_{0.5}. Training ablations and factor-wise interventions support the staged design and show that the learned state continues to contribute to control after direct perceptual access is restored.

Wenbo Li, Yiteng Chen, Wei Zhang +3

School of Software Engineering, South China University of Technology Guangzhou, China · Yuanwu Technology, Shenzhen, China

Visuomotor Policy LearningVision-Language-Action ModelsRobotic Manipulation
AskPDF
Sep 19, 2026

Task aware Dynamic Movement Primitives for failure detection and recovery in contact rich manipulation

Assembly remains a challenging robotic manipulation task in presence of tight tolerances and complex contact interactions. While Learning from Demonstration (LfD) frameworks like Dynamic Movement Primitives (DMPs) can effectively encode trajectories from a single demonstration, they are highly sensitive to variations in initial grasp configurations and external contact forces. Such variations often lead to task failures during the contact rich phases. This paper presents a task aware failure detection and recovery framework that integrates DMP based trajectory generation with real time stage classification. Utilizing Quadratic Discriminant Analysis (QDA) trained on multimodal sensor data, the framework segments execution into approach, alignment, and insertion stages for a Peg in Hole (PiH) assembly operation. By using goal relative position data as features, this classification generalizes to unseen goal positions without requiring retraining, matching the inherent generalization capability of DMPs. Anomaly detection is performed online using a Mahalanobis distance metric computed over force features, isolating contact induced failures from nominal trajectory execution. Upon failure detection, a spiral search recovery policy is triggered to actively realign the peg under contact before resuming the learned DMP insertion. The proposed approach is evaluated on an experimental setup achieving 95% stage classification accuracy, and demonstrates reliable failure recovery under lateral misalignments of up to 3 mm using only a single demonstration.

Bhavnashri A, Sobia Shafi, Krishnapuram Himavarshini +1

Atomberg Technologies Limited, India · Indian Institute of Technology Madras (IITM), India · Appian Corporation, India

Robotic ManipulationContact-Rich Robotic ManipulationRobotic AssemblyRobot Motion Generation
AskPDF
Sep 19, 2026

Tip Manipulation in Soft Everting Robots via Wall Retraction and Deployable Fingers

Soft everting robots can traverse long, confined paths by continuously growing, yet active interaction remains limited to a single tool fixed at or near the tip, unable to be repositioned on demand and difficult to reconcile with the robot's soft body. We introduce a tip-manipulation and multi-tool deployment strategy for soft everting robots based on wall retraction, implemented with a base roller assembly that independently meters membrane flow in the outer wall while a tail spool regulates growth in the internal tail section. Coordinated wall and tail actuation decouples robot length from membrane-material position, enabling membrane-mounted devices to be transported, exposed, and repositioned at selected locations near the distal tip. We pair this capability with ultralight pleated inflatable fingers integrated into the membrane, fabricated from TPU-coated nylon with an internal airtight bladder. The fingers achieve large bending at low pressures (approximately 100 degrees at 50 kPa in high-pleat designs) and generate blocking forces up to 1.9 N, while remaining limp during transport. The system demonstrates adaptive grasping across diverse household objects (21 to 550 g; 16 to 200 mm), three-dimensional object manipulation and stacking, environmentally braced extension, distal camera panning for confined-space inspection, and controlled sequential payload delivery. These results enable embodied and reversible tip manipulation for soft growing robots in cluttered and tortuous environments.

Nelson Badillo Perez, Niccolo Pagliarani, Matteo Cianchetti +1

John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · The BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy

Soft RoboticsRobotic ManipulationRobotic Grasping
AskPDF
Sep 19, 2026

Search, Ground, Plan: Functional Sufficiency for Task and Motion Planning under Incomplete Scene Knowledge

Foundation models (FMs) have expanded task and motion planning (TAMP) to manipulation problems specified through language and visual observations. However, incomplete scene knowledge leaves a critical gap between understanding what the task requires and knowing whether the physical scene can actually realize it. We introduce GRAB-TAMP, an FM-based TAMP framework that searches for scene entities required for task completion, grounds functional roles to valid physical objects, and plans only after a complete joint assignment establishes functional sufficiency. We represent the task through functional roles, relations, and assignment constraints, and incrementally inspect the scene while requirements remain unresolved, verifying candidate objects through semantic, geometric, and relational checks. We evaluate GRAB-TAMP across 32 scene variants spanning Kitchen, Living Room, and Workshop domains. Across 200 feasible trials, our approach achieves 54.0% end-to-end success with 67.3% plan goal coverage. Compared with three FM-based TAMP frameworks under the same execution setting, GRAB-TAMP improves end-to-end success by 25.7 percentage points over the mean baseline. Implementation and evaluation code: https://github.com/Narendhiranv04/GRAB-TAMP

Narendhiran Vijayakumar, Nav Singhal, Girish Varma +1

Robotics Research Center, IIIT Hyderabad · Center for Security, Theory and Algorithmic Research, IIIT Hyderabad

Motion PlanningRobotic ManipulationLarge Language Model-Based Robot PlanningDecision-Making under Uncertainty
AskPDF
Sep 19, 2026

Anatomy of a Closed-Loop Collapse: A Causal Case Study of a Compressed VLA Policy

Compressed manipulation policies can pass offline evaluation while failing in closed-loop execution; this dissociation is established in prior work and is not our claim. We contribute a causal anatomy of one naturally occurring case. An 8-layer distillation of Octo-Base retains 86% of parameters, passes every offline check we applied (0.996 and 1.000 teacher-ratios on the family's own validation metrics), and collapses in closed loop: 0/72 vs. the teacher's 40/72 on a simulated WidowX pick-and-place task. The collapse is structured, not diffuse: early task stages degrade gradually (the student moves the object at 90% of the teacher's rate and grasps at 55%), while transport-to-target fails categorically, at 0% in every training variant. Paired action-trace forensics isolate the signature: a negative, late-heavy zz residual, roughly 10x its post-repair magnitude, and persistent across the base distillation and both continuation branches. Four standard therapies fail under matched controls: continued training and in-domain offline data leave success at zero, even though the latter measurably improves marginal action statistics; command-level compensation recovers nothing at any offset, although the same perturbations degrade healthy policies; clamping the symptom in the command channel preserves grasping, yet success stays at floor. A minimal-pair intervention that substitutes half of the training stream with deployment-distribution teacher rollouts, with every other setting held fixed, restores parity with the teacher (18/36 vs. 17/36 held-out), eliminates that signature, and recovers a teacher-like perturbation-response profile. We claim existence, not universality. Operationally, offline gates, including a family's own validation metrics, are insufficient acceptance tests for compressed policies; a few dozen closed-loop trials sufficed to find what they missed.

Fengze Jia

The Ohio State University · The Ohio State University Columbus, OH, USA

Robotic Policy EvaluationVision-Language-Action ModelsRobotic ManipulationKnowledge DistillationOn-Policy Distillation
AskPDF
Sep 19, 2026

Physical-Touch Observability from Wrist Wrench in Granular Scooping

Mining and earthmoving are important real-world deployment settings for embodied intelligence. Autonomous transport and driving systems have improved substantially, but loading and scooping still often depend on skilled human operators, exposing personnel and equipment to operational risk. For robotic scooping, pre-contact RGB-D sensing reveals surface geometry but not the resistance, compaction, tool engagement, or load transfer that emerge during interaction. We test whether the current scoop's six-axis wrist force/torque (F/T), or wrist wrench, contains information about final collected volume, and whether that information depends on the correctly paired action-terrain interaction. We call this property physical-touch observability. Using 6,700 real-robot scoops across 67 terrains, we evaluate correctly paired current-scoop F/T against pre-contact prediction and correspondence-breaking controls under terrain-held-out testing. At the retrospective 60% sequence boundary, correctly paired F/T reduces mean absolute error by 14.2% relative to Action-only and by 21.9% relative to cross-terrain mismatched F/T. Engineered signal summaries reproduce the result across model architectures. Together, these findings position wrist wrench not merely as a low-level feedback signal, but as a task-level perceptual modality through which embodied robots can infer hidden physical states during interaction, providing a foundation for response-aware autonomy in mining and other contact-rich tasks.

Hongyi Lin, Song Zhang, Xubo Liu +1

School of Vehicle and Mobility, Tsinghua University · Tsing-AI(Shanghai) Technology Co., Ltd

Robotic PerceptionRobotic ManipulationRobotic Tactile Sensing
AskPDF
Sep 19, 2026

ForceRFT: Refining VLA Actions through Force-Guided Residual Reinforcement Learning

Force-conditioned vision-language-action (VLA) policies can respond to contact, but when trained solely on demonstrations, their recovery behavior may be limited by demonstration coverage, and they do not learn from deployment outcomes. Human corrective imitation provides additional recovery examples, but its objective matches local action targets without explicitly optimizing task return. We present ForceRFT, a force-guided residual reinforcement learning framework that learns contact-dependent corrections from human supervision and autonomous task outcomes. A frozen, demonstration-trained SmolVLA-based prior generates force-conditioned action chunks, while a lightweight residual actor refines individual end-effector pose commands using wrist feedback acquired during chunk execution. The decision-time wrist wrench, its temporal change, and the selected base motion condition both residual correction and value estimation. Human corrections supervise the residual actor, while verified autonomous transitions train the twin critics and support value-guided updates to the same actor. Bootstrapping is restricted to autonomous segments, preventing TD credit from crossing human-intervention boundaries. Real-robot experiments on plug insertion, ring-on-peg assembly, and whiteboard wiping show higher autonomous success rates than the evaluated demonstration-trained and residual-imitation baselines. Comparisons with residual imitation support value-guided residual optimization, while plug-insertion ablations indicate the benefit of direct execution-time wrist feedback.

Yichen Wang, Chaoyang Zhang, Xuqi Su +3

College of Information Science and Technology, Eastern Institute of Technology, Ningbo, Ningbo 315200, China. · Jiangsu Key Laboratory of Advanced Food Manufacturing Equipment and Technology, Jiangnan University, Wuxi, Jiangsu, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China. +3

Robotic ManipulationVision-Language-Action ModelsRL for VLMsResidual Policy LearningInteractive Imitation Learning
AskPDF
Sep 19, 2026

Whole-Body UMI: Transferring UMI Manipulation Skills to Humanoid Whole-Body Manipulation via Real-Time Motion Generation

Collecting whole-body demonstrations for humanoid manipulation mostly relies on teleoperation, which is costly and hard to scale up. The Universal Manipulation Interface (UMI) provides a scalable data collection paradigm, but end-effector trajectories alone underdetermine humanoid whole-body coordination, which is insufficient for whole-body demonstration collection. Therefore, we introduce Whole-Body UMI (WB-UMI), a task-agnostic, real-time and end-effector conditioned motion generator that decouples whole-body coordination learning from task semantics learning through a shared end-effector interface. A diffusion policy learns from native UMI demonstrations, while WB-UMI learns independently from retargeted motion capture, requiring no body trackers or paired image--whole-body demonstrations during task-specific data collection. In real deployment, an asynchronous hierarchy integrates the diffusion policy, motion generator, and a whole-body controller with latency compensation and measured-state feedback. Real-robot experiments on G1 support real-time closed-loop transfer across four tasks, achieving 90% success in drawer closing, 80% in shelf pick-and-place, 30% in ball toss, and 40% in Loco-PnP, which shows the effectiveness of this hierarchy in transferring native UMI skills to humanoid whole-body manipulation.

Yuxuan Nai, Leixin Chang, Liangjing Yang +2

Zhejiang University · Hong Kong Embodied AI Lab · Mondo Robotics +1

Whole-Body Robotic ManipulationRobot Motion GenerationRobotic ManipulationRobot Imitation LearningCross-Embodiment Transfer
AskPDF
Sep 19, 2026

Expert-Play Contouring Control: Faster-than-Demonstration Planning from Slow Expert and Fast Play

Expert demonstrations often specify what a robot should do, but not how fast it can do it. Imitation Learning (IL) inherits demonstration timing, while directly accelerating the learned motion can fail when faster execution changes the robot-object dynamics. We study faster-than-demonstration execution as a dynamics-aware control problem and introduce Expert-Play Contouring Control (EPCC), which combines slow expert demonstrations with fast, non-expert play. Expert demonstrations train a latent trajectory generator whose predictions are reparameterized into a time-independent contour of successful task progression, while play trains a world model (WM) of fast-action outcomes. At deployment, our proposed planner uses the WM to optimize actions that makes maximize progress along the expert-derived contour while penalizing deviation from the intended task evolution. Averaged across three visuomotor manipulation tasks, EPCC achieves a 2.0×2.0\times the throughput of the IL baseline, including 2.2×2.2\times that of the throughput of the strongest acceleration baseline on a task with interaction-sensitive object dynamics. Our analysis shows that the gains concentrate where faster execution changes robot-object evolution. Together, our results highlight a simple yet effective principle: demonstrations provide task intent, while play data provides the dynamic coverage needed to execute that intent faster.

Seunghoon Cho, Wonsuhk Jung, Sundhar Vinodh Sangeetha +1

Seoul National University, Seoul, South Korea · Georgia Institute of Technology, Atlanta, GA, USA

Robot Motion PlanningRobotic ManipulationLearning from DemonstrationRobot Dynamics Learning
AskPDF
Sep 19, 2026

BEACON: Belief-Enabled Adaptive CONtrol for Imitation Learning under Uncertainty

Robot manipulation tasks often involve hidden state information that cannot be directly observed and must be inferred through sequential physical interactions. In such partially observable settings, conditioning an imitation learning policy directly on the recent raw observation history leads to poor performance. This is due to state aliasing, wherein identical observations may arise from different hidden states, and the policy receives conflicting action labels for the same input. To enable history-aware disambiguation capability, we propose conditioning a diffusion policy on a structured representation of the hidden states using Bayesian belief that exposes both the current most likely state estimate and the remaining uncertainty. This representation replaces raw history with a structured, compact input, enabling the policy to implicitly modulate between exploratory and exploitative behaviors based on belief uncertainty, without explicit mode switching or reward shaping. We evaluate across two domains with qualitatively different belief representations: a continuous belief for cornstalk gripper alignment via tactile sensing, and a discrete categorical distribution for latched door opening. In both domains, the belief-conditioned policy substantially outperforms the observation-only baseline and approaches privileged ground-truth performance, with ablations illustrating that the policy adapts its exploration behavior depending on the belief uncertainty at inference time.

Moonyoung Lee, Soumojit Bhattacharya, George Kantor +1

Carnegie Mellon University, Robotics Institute, Pittsburgh PA 15213, USA

Robot Imitation LearningImitation LearningRobotic Manipulation
AskPDF
Sep 19, 2026

A Direct Rigid Transmission 2-DoF Wrist Extension for Tendon-Driven Hand

Dexterous manipulation in confined spaces requires local control of hand orientation. Without a wrist, a dexterous hand must obtain this local orientation through coordinated motion of the robot arm, often involving several joints and a more complex end-effector path. We present CRAFT-Wrist, a concentric 2-DoF wrist extension that mounts between a robot arm and the CRAFT Hand without modifying the hand. Two XC430-T240BB-T servos drive sideways and front-back rotation through short rigid transmissions. Our initial hardware prototype actuates both axes under load, achieving a demonstrated workspace of ±20∘\pm20^{\circ} in radial--ulnar deviation (left--right) and a front-back range of +80∘+80^{\circ} in flexion and −18∘-18^{\circ} in extension. We characterize the resulting finger-motor loading across several wrist postures and demonstrate how the wrist supplies local orientation in grasping, nail hammering, and blackboard wiping. Demonstrations and assembly instructions are available at https://craft-wrist.github.io/.

Yujie Pang, Sadman Sakib, Mohammad Abdullah Al Faruque

The authors are with the Nhu Department of Electrical Engineering and Computer Science, University of California-Irvine, Irvine, CA, USA. · Contributed during participation in the UCInspire program at UCI

Robotic ManipulationRobotic Hand DesignDexterous Manipulation
AskPDF
Sep 19, 2026

SHAFT: A Slack-Compensating, Helical-Buckling-Attenuating Flexible-Shaft Transmission for Lightweight Multi-DoF Manipulation

Lightweight and slim manipulators enable safe operation in human living environments. Proximal actuation using remote transmission mechanisms, such as wire-driven or Bowden cables, effectively reduces inertia and arm size by relocating motors near the base and transmitting torque to distal joints. Existing approaches either increase mass through additional components, such as pulleys for direction changes, or suffer from reduced transmission efficiency due to friction losses. Flexible shaft transmission avoids both mass increase and excessive friction losses, but faces increasing angular transmission error due to helical buckling caused by slack generated at joint bending. To address this problem, we propose SHAFT: a Slack-compensating, Helical-buckling-Attenuating Flexible- shaft Transmission mechanism. This mechanism compensates for slack through a proximal tensioner, improving the angular transmission error and efficiency of flexible shaft transmission without increasing the moving mass of the arm section. In a transmission path containing four 90-degree bends, the proposed mechanism demonstrated approximately 30% higher efficiency and approximately 65% lower angular transmission error compared to a flexible shaft transmission without a tensioner. Using this mechanism, we fabricated a 6-Degree-of-Freedom (DoF) arm with a 1-DoF gripper manipulator consisting of a rotary module housing motors with a tensioner, and a 280 g weight for the arm module. The proposed manipulator represents a novel remote actuation system for achieving lightweight construction with high efficiency, contributing to the acceleration of safe robot deployment in human environments.

Tomoya Takahashi, Moses Gladson Selvamuthu, Richiro Tadakuma +1

OMRON SINIC X Corporation, Hongo 5-24-5, Bunkyo-ku, Tokyo, Japan · Yamagata University, 4 Chome-3-16 Jonan, Yonezawa, Japan · OMRON SINIC X

Mechanism DesignRoboticsRobotic Manipulation
AskPDF
Sep 18, 2026

FRAMES: Failure Recovery And Monitoring of Embodied Skills for Humanoid Loco-Manipulation

Large language model (LLM) planners can decompose natural-language instructions and select reusable robot skills, but choosing the correct skill does not guarantee successful physical execution. This gap is especially important in humanoid loco-manipulation, where errors during approach, grasping, transport, or placement can invalidate the remainder of a long-horizon plan. We present FRAMES, a failure-aware supervisory framework for the Unitree G1 humanoid that operates above the CEER whole-body controller. A Planner Agent selects subtasks through parameterized mid-level skills, while a vision-language-model-based Monitor Agent evaluates each skill using temporal multi-view observations and structured robot and contact evidence. Detected failures stop the active skill and provide grounded feedback to a Recovery Agent. The framework further includes a Memory Module for reusing prior skill experience, and geometric grounding via depth and segmentation. We independently evaluate the monitoring module of the framework in MuJoCo using 100 trials comprising 50 failed and 50 successful executions across five tasks. The monitor detects 48 of 50 failures, correctly accepts 46 of 50 successful executions, and achieves 94.0% overall accuracy. These results provide initial evidence for the monitoring component, while end-to-end evaluation of the complete recovery loop remains ongoing.

Ajay Vikram Periasami, Xinyuan Luo, Haoyu Li +1

Department of Computer Science, Duke University, Durham, NC, USA · Department of Mechanical Engineering and Materials Science, Duke University, Durham, NC, USA

Humanoid Loco-ManipulationLong-Horizon Robotic ManipulationRobotic ManipulationLarge Language Model-Based Robot PlanningVLMs for Robotics
AskPDF
Sep 18, 2026

Gripper-Aware Automatic Dense Packing of Irregular Objects

Automatic dense packing is widely desired in warehouse operations but remains a fundamental challenge in robotic manipulation. Existing work on irregular-object packing largely targets simulation with idealized contact, treating the object as an isolated rigid body. The gripper often enters as a discrete, post-hoc feasibility check, if considered at all, and the perception and contact drift accumulated during execution are not addressed. We present a closed-loop pipeline that integrates perception, gripper-aware placement optimization, and force-guided execution on a real manipulator. The optimizer represents the object together with the gripper as a single composite body of hierarchical sphere trees. It searches over five degrees of freedom on a GPU within a CMA-ES framework, with the vertical coordinate grounded analytically against the current heightmap. During execution, a force-monitored vertical descent stops on first contact. A post-release consolidation push then closes the residual lateral clearance that gripper-aware planning leaves behind. The container is re-perceived between placements so that drift does not accumulate. We validate the system on a Franka Emika Panda robot packing a 3D-printed set of flat, curved, and concave objects, and a YCB object subset. An ablation study isolates the contribution of gripper-aware optimization, the consolidation push, and mesh-derived geometry to end-to-end success, achieved density, and computational cost. We further benchmark against the heightmap-minimization method as a baseline representative of prior irregular-object packing work.

Tianhao Qin, Connor McCann, Berk Calli +1

Robotics Engineering Department Worcester Polytechnic Institute, Worcester, MA 01609, USA

Robotic Manipulation
AskPDF
Sep 18, 2026

MAAP: Multi-Agent Active Perception for Collaborative Manipulation

Multi-agent manipulation naturally produces multiple task-driven viewpoints: every arm carries a wrist camera and moves through the scene while acting. Yet these observations are typically underutilized, and active perception in manipulation is still often treated as requiring a dedicated sensing agent. We introduce MAAP (Multi-Agent Active Perception), in which every arm is dual-purpose: it executes manipulation actions and, through the wrist camera it carries, simultaneously serves as a moving viewpoint for the team. We pair this with RAIL (Role-Aware Imitation Learning), a controller that predicts each arm's current role alongside its action chunk and conditions action generation on it, representing role-dependent actions within one network. Across four simulated tasks, widening the perception regime lifts average success from 56.5% with a fixed camera to 62.5% with one active wrist view and 70.0% with all of them, while MAAP+RAIL reaches 79.2%. RAIL's additional gain is concentrated on the three-arm Microwave task, where success rises from 47% to 82% on identical multi-wrist inputs. On a dual-arm platform, MAAP+RAIL succeeds in 14 of 20 placement trials compared with 0 of 20 for fixed-view ACT. Collaborative manipulation can thus serve as an active perception mechanism in its own right.

Bruno N. Y. Chen, Li Kang, Heng Zhou +4

Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China +3

Active Vision for Robot ManipulationRobotic ManipulationActive VisionMulti-Agent SystemsImitation LearningRobot Imitation Learning+1
AskPDF
Sep 18, 2026

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions. Most state-of-the-art models predict actions in the end-effector pose space as sequences of action chunks. Training and evaluating these models requires large-scale collections of real-world demonstrations, pairing robot actions with the corresponding visual and proprioceptive observations. Collecting such data on real hardware typically relies on human teleoperation, making the process costly, time-consuming, and difficult to scale. We present an open-source sim-to-real experimental protocol that addresses this bottleneck: expert trajectories generated in simulation are replayed open-loop on a real Franka FR3 setup, where the corresponding real visual and proprioceptive observations are recorded and converted into a format compatible with VLA training. The same deployment stack is then reused, in closed-loop, to evaluate a trained policy on that setup, so that data collection and evaluation share an identical hardware configuration. Because each real recording is paired with the simulated trajectory that produced it, the protocol also yields a direct measurement of the sim-to-real gap. We release the collected datasets on Hugging Face together with the pipeline source code https://gitlab.isir.upmc.fr/kappel/sim2real_public_chunk_control.

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani +5

Institute of Intelligent Systems and Robotics (ISIR), CNRS, Sorbonne University, Paris, F-75005, France

Sim-to-Real TransferRobotic ManipulationVision-Language-Action ModelsRobotics SimulationRobot Imitation Learning
AskPDF
Sep 18, 2026

ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction

Manipulating objects requires understanding not only their motion, but also the physical properties that determine it. For articulated objects, these include inertia, friction, and mechanisms such as springs or door closers, whose effects can vary with configuration and velocity. Such properties are not directly observable from appearance: visually identical doors may require very different effort to manipulate. Existing digital-twin pipelines recover primarily kinematics or assign static physical parameters from visual and language priors, which can yield physically implausible estimates. As a result, state-dependent mechanism dynamics remain unidentified and are not represented in standard asset formats. We present ForceTwin, a system for identifying physics-informed digital twins of articulated objects from instrumented human interaction. A person probes an object using a handheld force-sensing gripper, providing synchronized poses and interaction forces from which we estimate the articulation, parametric dynamics including inertia, Coulomb friction, viscous damping, and a structured neural residual capturing state-dependent mechanism forces. ForceTwin nearly halves the inertial-parameter error of a VLM prior. As a feedforward dynamics model for impedance control on a Spot and a Franka FR3, ForceTwin achieves 87% goal completion across nine object-embodiment pairs, compared with 60% using VLM-prior and 57% using kinematics-only twins, with the largest gains on objects whose strong mechanisms cause both baselines to stall. We further use the identified twins to train whole-body door-traversal policies and deploy them in the real world. Project Page: https://timengelbracht.github.io/forcetwin-website/

Tim Engelbracht, René Zurbrügg, Mayank Mittal +4

ETH Zurich · NVIDIA · Microsoft +1

Digital TwinsRobotic ManipulationRobot Dynamics Identification
AskPDF
Sep 18, 2026

Robotic Multiphase Interaction: Manipulating Coupled Liquid and Solid Dynamics with a World Model

This work presents Robotic Multiphase Interaction (RMI), a setting in which liquid enters a porous material and interacts mechanically with its deforming solid skeleton. Manipulation can therefore change pore volume, expel or redistribute retained liquid, and alter grasp stability at the same time. Spilled liquid can also create safety risks in domestic and manufacturing settings. This differs from most manipulation of solid objects and from tasks that involve both liquid and solid while keeping the phases spatially separate. We study a sponge filled with water as the first RMI example. We use implicit incompressible porous flow with smoothed particle hydrodynamics as the dynamics engine and enable robotic manipulation by adding Coulomb contact memory, hybrid velocity and force regulation, and a stability gate for lifting. The resulting environment connects robot commands to changes in the coupled liquid and solid state. A world model conditioned on actions predicts how this state evolves under candidate commands, while a temporal UNet generates actions using either Diffusion Policy or rectified flow matching. Our world model reduces retained water prediction error by more than 60%60\% compared with the baseline. The best action sequence selected by the world model from policy proposals further reduces the predicted terminal water error by about half. These improvements show that modelling the coupled liquid and solid state helps the robot predict how its actions affect both the porous object and the liquid held inside.

Yixuan Feng, Peng Wang

College of Design and Engineering, National University of Singapore, 9 Engineering Dr 1, Singapore,117575. · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford GU2 7XH, United Kingdom.

Robotic ManipulationDynamic Robotic ManipulationWorld Models for Robotics
AskPDF
Sep 18, 2026

A Unified Dynamic Force Guidance Framework for Performance-Optimized Kinesthetic Teaching

Collaborative robots are increasingly deployed in industrial scenarios characterized by frequent product changeovers. As an intuitive programming method, kinesthetic teaching facilitates rapid robot deployment. However, users may overlook the configuration of the robot during kinesthetic teaching, leading to degradation in operational performance. Operational performance refers to the capability of the robot to generate motion and can be quantified by the Minimum Singular Value of the Jacobian matrix. To address this issue, this paper proposes an online dynamic force guidance method that integrates performance constraint and optimization mechanisms. Specifically, variable admittance control maintains the operational performance of the robot above a predefined threshold, while a virtual force actively guides the user to drag the robot towards configurations with improved performance. Experiments are conducted on a 6-DOF collaborative robot, comparing three typical paths in the task space. To evaluate the quality of the taught trajectories, trajectory playback experiments are conducted to analyze the relationship between the operational performance of the robot and the work efficiency. The results demonstrate that the proposed method effectively enhances the operational performance of the robot and consequently improves the work efficiency, holding significant value for reducing production takt time in industrial deployment.

Chunxin Li, Jianhua Wu, Zhenhua Xiong +1

State Key Laboratory of Mechanical System and Vibration, Institute of Robotics, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

Robotic ControlHuman-Robot CollaborationRobotic ManipulationLearning from Demonstration
AskPDF
Sep 18, 2026

ProTracer: Proprioception-Guided Failure Diagnosis in Robot Manipulation

This paper presents a comprehensive framework for robot manipulation failure analysis that includes binary failure detection, failure categorization, explanation generation, and the additional capability of failure onset localization, which aims to identify the earliest moment at which a robot execution deviates from a valid task-completion trajectory and is ultimately followed by task failure. To address these tasks, we propose ProTracer, a training-free framework that leverages existing Vision-Language Models (VLMs) together with proprioceptive signals for failure analysis. Our method uses proprioceptive dynamics to identify temporally informative action boundaries and converts richer robot-state signals into structured natural-language descriptions that can be jointly analyzed together with visual observations by the VLM. This design combines the temporal precision of proprioceptive signals with the multimodal reasoning capabilities of modern VLMs without requiring additional model training. We further introduce FailTime, a benchmark with synchronized visual and proprioceptive observations for evaluating conventional failure diagnosis tasks as well as failure onset localization. Experiments demonstrate that ProTracer achieves strong performance across both conventional failure diagnosis tasks and the newly introduced failure onset localization task, highlighting the importance of proprioceptive reasoning for fine-grained temporal failure analysis.

Chang Dong, Mehdi Hosseinzadeh, King Hang Wong +4

Australian Institute for Machine Learning, Adelaide University, Australia. · School of Allied Health and Human Performance, Adelaide University, Australia.

Robot Failure DetectionRobotic ManipulationFault LocalizationRobot Manipulation Benchmarks
AskPDF
Sep 18, 2026

MicroHookACT: Monocular Microscopic Vision Guided Visuomotor Policy for Flexible Microelectrode Hooking

Automated needle-loop hooking is a critical step in flexible microelectrode (FME) implantation. This paper presents MicroHookACT, an imitation learning-based visuomotor policy for automated 3D hooking under monocular microscopic vision. First, a unidirectional hooking strategy exploits defocus cues and optical-axis guidance to enable palpation-free precise alignment and contact-rich threading. Second, an action-supervised object attention module built on a frozen ViT backbone learns to focus on the micro-needle tip and micro-loop directly from human demonstrations, without requiring manual visual annotations for training. Third, attention-centered global coarse and local fine features are dynamically weighted according to predicted action progress, enabling a single ACT policy to adapt to changing defocus blur and visual requirements throughout the operation. In the experiments, visuomotor policies were trained on 60 human demonstrations and evaluated under five setups with varying difficulties. Our MicroHookACT framework achieved the highest overall success rate of 96.7% with an average execution time of 11.5 s. These results demonstrate the potential of visuomotor policy learning for micron-level control under varying operating conditions.

Yitong Chen, Fangbo Qin, Yang Wang +3

Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China. · State Key Laboratory of Brain Cognition and Brain-Inspired Intelligence Technology, Shanghai 200031, China. +1

Robotic ManipulationVisuomotor Policy LearningSurgical RoboticsRobot Imitation Learning
AskPDF
Sep 18, 2026

KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos

Learning robot manipulation policies typically requires substantial demonstration data, which are costly to collect on real robots. Recent methods generate robot demonstrations from human videos by adapting recovered motion and validating the resulting trajectories in simulation. However, methods centered on motion-reference adaptation can limit behavioral diversity by retaining the demonstrated contact strategies and subtask orders, while insufficient understanding of task requirements and scene relations can reduce demonstration generation efficiency by generating invalid candidates. To address these limitations, we propose KnowDemo, a framework that uses structured manipulation knowledge from human videos to generate diverse robot demonstrations for a target workspace. To distinguish task requirements from demonstration-specific choices, we develop a knowledge extraction and reasoning module based on a vision-language model (VLM) that associates object and action descriptions with inferred task conditions, demonstration references, and permissible execution variations. To translate this knowledge into executable demonstrations, we resolve the descriptions against target-scene entities and geometry to guide candidate generation and screening before motion planning and simulation. The resulting demonstrations exhibit multimodal behavior through alternative contact strategies and valid subtask orders, with structured execution labels. Experiments demonstrate additional verified execution modes beyond a reference-only configuration and improved candidate planning success through task-guided grasp sampling. To validate the generated data for policy learning, we fine-tune the pretrained π0.5π_{0.5} model on simulation data, achieving sim-to-real transfer across three tasks. Project page: https://zhiyuan-gao.github.io/knowdemo/

Zhiyuan Gao, Yanxiang Zhan, Mohammad Khoshnazar +2

University of Bremen · Robotics Institute Germany (RIG)

Robot Demonstration GenerationSynthetic Data Generation for RoboticsRobotic ManipulationLearning from Demonstration
AskPDF
Sep 18, 2026

FOCAL-VLA: Subtask-Guided Geometry Distillation and Implicit World Modeling for Vision-Language-Action Models

Vision-language-action (VLA) models built on pretrained vision-language models have demonstrated strong performance across diverse robotic manipulation tasks. However, VLA models that directly map current 2D observations to actions often lack sufficient spatial and temporal understanding, limiting their performance in precise and long-horizon manipulation. Recent methods enhance VLA models through geometric supervision and future-state prediction across the entire scene. However, these methods can suffer from redundant scene information, distracting the model from learning the geometry and dynamics relevant to the current interaction. To address this issue, we propose FOCAL-VLA, a framework that combines subtask-guided geometry distillation with implicit world modeling to learn representations of current spatial structure and future interaction dynamics. To focus geometric learning on the current subtask, we transfer geometric knowledge from VGGT to the VLA model by aligning geometry latents with features from subtask-relevant image regions. To capture the future 3D evolution of the current interaction, we incorporate implicit world modeling using Track4World features from current and future demonstration frames. The two complementary representations jointly guide action generation without running VGGT or Track4World at inference time. Experiments show that FOCAL-VLA outperforms baselines on both simulation benchmarks and real-world manipulation tasks. Project website: https://zhiyuan-gao.github.io/FOCAL-VLA/.

Zhiyuan Gao, Di Wen, Yanxiang Zhan +4

University of Bremen, Germany · Karlsruhe Institute of Technology (KIT), Germany · Robotics Institute Germany (RIG)

Vision-Language-Action ModelsRobotic ManipulationWorld Model Learning
AskPDF
Sep 17, 2026

MetaPusher: Meta Learning and Planning for Nonprehensile Manipulation of Unseen Objects with Rapid Online Adaption

Manipulating previously unseen objects remains challenging, as their dynamics depend on latent physical properties, such as friction and mass distribution, that cannot be inferred from perception alone. Prior experience across objects can provide an initial estimate of unseen object dynamics, but this estimate remains uncertain and can degrade further during sim-to-real transfer. Adapting the dynamics through interaction can progressively refine the estimation, however, updating the model may invalidate the planned trajectory. Successful and efficient manipulation therefore requires both rapid dynamics adaptation and a planning strategy that can incorporate this evolution. In this work, we introduce MetaPusher, a meta-learning and adaptive planning framework for nonprehensile manipulation of unseen objects without prior object-specific interactions. A meta-learned dynamics model rapidly adapts from interactions during task execution, while an adaptive kinodynamic planner updates long-horizon plans by reusing and refining its existing search tree. This coupling enables manipulation and adaptation without a separate data collection phase. We evaluate MetaPusher on unseen objects in simulation and in sim-to-real scenarios, comparing against fine-tuning and active learning methods, MPPI-based control, and a reinforcement learning policy. It achieves lower prediction error and improves task success rate by up to 20%.

Donghyung Lee, Seyedali Golestaneh, Jaskrit Singh +3

Non-Prehensile Robotic ManipulationRobotic ManipulationRobot LearningRobotic Object PushingRobot Dynamics LearningLong-Horizon Robotic Manipulation+1
AskPDF
Sep 17, 2026

Demonstration Synthesis from a Single Scan via Gaussian Splatting for Visuomotor Policy Learning

Training a visuomotor policy calls for abundant demonstrations that closely match the target environment, yet collecting them anew remains expensive. Existing demonstration synthesis methods reduce this cost but remain constrained by high manual effort, limited visual fidelity, or heavy reliance on physics simulators. This paper introduces GaussianFactory, a high-fidelity data engine that mass-produces demonstrations with a single video scan as its only human input and no physics engine in the generation loop. Specifically, GaussianFactory reconstructs the scene as an editable 3D Gaussian Splatting (3DGS) replica and samples from the object-combination tasks the scene affords. For each task, it plans grasps and trajectories purely kinematically on the geometric reconstruction, rendering photorealistic demonstrations that visually match the target environment. Physical dynamics enter the pipeline only where contact force interactions dictate the outcome---during grasp formation, via a learned contact model pretrained once on an interaction dataset. To evaluate the downstream utility of the synthesized demonstrations, we implement an end-to-end scan-to-deployment workflow in two setups: a simulated scene that stands in for the real world to enable reproducibility, and a real-world workspace with a physical UR10e robot. In each setup, a standard diffusion policy trained solely on the synthesized demonstrations achieves 95.1% and 84.2% success rates, respectively.

Beichen Wang, Yuen-Hei Yeung, V. R. Sridhar Devarakonda +1

George Mason University · New York University

Robot Demonstration GenerationSynthetic Data Generation for RoboticsRobotic ManipulationVisuomotor Policy LearningImitation Learning
AskPDF
Sep 17, 2026

DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures. Project website: https://dextera-project.github.io/

Jin Wu, Lianjie Yuan, Zeyan Sun +4

The University of Texas at Austin · University of Florida · BrainCo

Sim-to-Real TransferSynthetic Data Generation for RoboticsRobotic ManipulationDigital Twins
AskPDF
Sep 17, 2026

Catch Me If You Can: Real-Time Feedback Denoising for Responsive VLAs

Vision-Language-Action (VLA) models have shown strong generalization in robotic manipulation by combining semantic knowledge from pretrained vision-language models with expressive action-generation policies. Diffusion-based action generators are particularly effective for modeling temporally coherent action chunks, but these chunks are typically executed open-loop after inference. This limits responsiveness when objects move, contacts change, or the scene evolves during execution. We propose VLA-Feedback, a two-timescale architecture that combines low-frequency diffusion planning with high-frequency visual feedback. Rather than fully denoising an action chunk before execution, VLA-Feedback retains its final denoising step as a lightweight feedback interface, allowing each action to be corrected using the latest observation before it is executed. This design preserves the expressiveness of the diffusion planner while enabling real-time action correction without rerunning the full vision-language diffusion model. VLA-Feedback matched GR00T on static LIBERO tasks while improving average success on dynamic simulation tasks from 27.5% to 85.0%. On real-robot tasks, it improved average success from 51% to 73%. Additional materials can be found on our project page: https://vla-feedback.github.io.

Yiheng Ji, Xingru Zhou, Luis Sentis +1

The University of Texas at Austin · University of Central Florida

Closed-Loop Control of VLA ModelsVision-Language-Action ModelsRobotic ManipulationVision-Based Robot ControlDiffusion Policy
AskPDF
Sep 17, 2026

Shake to Learn: Dynamic Interrogation of Hidden Object Physics for Robotic Manipulation with Physical Reservoir Computing

Many physical properties relevant to robotic manipulation are hidden from vision. A sealed object, for example, may reveal little about its center of mass (COM) or internal contents until it is lifted, shaken, or otherwise dynamically perturbed. This study shows that such interactions can enable a new modality of robotic perception and learning, in which interaction-induced dynamic responses are used to infer object physics that is inaccessible to conventional sensing. We implement this idea using an origami-inspired soft robotic arm that functions as a physical reservoir computer. After grasping an object, the arm is excited by a fixed shaking input at its base, and the resulting ringdown response is recorded through either camera tracking or embedded sensors. Because the input is held constant across trials, hidden object properties, such as the COM position, are encoded through their effect on the dynamics of the coupled robot-object system. A lightweight linear readout can then decode these dynamics to recover interpretable information about the hidden object physics. Using this framework, the soft robotic arm reservoir completed three tasks of increasing difficulty: inferring the orientation of the object's hidden COM, inferring the COM distance from the grasp point, and using the inferred COM information to guide a subsequent regrasp. We further develop a dynamic summary representation of the ringdown response that improves prediction accuracy. Together, these results establish shake-to-learn mechanical interrogation as a promising strategy for robotic systems to convert brief physical interactions into actionable cues about hidden object properties for downstream manipulation.

Wen Sin Lor, Jun Wang, Suyi Li

Department of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24060, USA · Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109, USA

Robotic PerceptionRobotic ManipulationReservoir Computing
AskPDF
Sep 17, 2026

Coding Agents with Harness for Safe Robot Control

Coding agents have emerged as a promising paradigm for robot manipulation: a language model writes the robot controller as a program, and agents built in this way now operate robots without robot-specific training. Whether this paradigm is also safe, however, has not been asked. We evaluate coding agents under a safety constraint, where each task pairs a manipulation goal with an obstacle the robot must not touch. The agent pursues the goal but collides with the obstacle in most cases, treating task completion as its sole objective. The agent reasons about the obstacle in its traces, and the prompt already forbids touching it, so neither perception nor instruction is at fault; the fault lies in the planning, where the stated constraint never becomes a priority. By decomposing manipulation into a route phase and a contact-rich moment, we locate the source of the failure. Along the route, the model cannot prioritize the safety constraint, having no notion of a clearing route and none of replanning once a chosen route becomes infeasible. At the contact, it is unaware that contact execution is bounded by the same constraint. To close this gap, we present SafeHarness, which equips the model with two obstacle-aware harnesses. Obstacle-aware route planning grounds the objects as bounding boxes and draws candidate routes over them as sequences of waypoints. The agent then plans a route in advance, verifies it, replans when necessary, and only then executes it. Obstacle-aware contact execution instead selects the contact position so that the contact itself avoids the obstacle. SafeHarness attains 81.2% task success and 91.9% collision avoidance with GPT-6-Astra, surpassing the previous SOTA by 13.7 and 23.0 points, and the same agent without harnesses by 31.2 and 57.5 points, respectively.

Bingxin Xu, Yuzhang Shang, Zhen Dong +1

USC · UCF · UCSB

Robot SafetyRobotic ManipulationSafe Motion PlanningRobot Collision AvoidanceSafe Robot Manipulation
AskPDF