Organizations: Shenzhen Ubiquitous Data Enabling Key Lab, Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China. · School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639956, Singapore.
Robotic contact-rich and fine-grained manipulation remains a significant challenge due to complex interaction dynamics and the competing requirements of multi-timescale control. While current visual imitation learning methods excel at long-horizon planning, they often fail to perceive critical interaction cues like friction variations or incipient slip, and struggle to balance global task coherence with local reactive feedback. To address these challenges, we propose M2-ResiPolicy, a novel Master-Micro residual control architecture that synergizes high-level action guidance with low-level correction. The framework consists of a Master-Guidance Policy (MGP) operating at 10 Hz, which generates temporally consistent action chunks via a diffusion-based backbone and employs a tactile-intensity-driven adaptive fusion mechanism to dynamically modulate perceptual weights between vision and touch. Simultaneously, a high-frequency (60 Hz) Micro-Residual Corrector (MRC) utilizes a lightweight GRU to provide real-time action compensation based on TCP wrench feedback. This policy is further integrated with a force-mixed PBIC execution layer, effectively regulating contact forces to ensure interaction safety. Experiments across several demanding tasks including fragile object grasping and precision insertion, demonstrate that M2-ResiPolicy significantly outperforms standard Diffusion Policy (DP) and state-of-the-art Reactive Diffusion Policy (RDP), achieving a 93% damage-free success rate in chip grasping and superior force regulation stability.
Figures & tables
Fig. 1 : M 2 -ResiPolicy Framework. Our dual-stream architecture decouples global planning (MGP) from high-frequency local correction (MRC), unified by a force-mixed PBIC execution layer.
Fig. 2 : Overview of the proposed M 2 -ResiPolicy framework. Left: The 10 Hz Master-Guidance Policy encodes multimodal observations and generates temporally coherent action chunks via confidence-gated cross-attention fusion, while the 60 Hz Micro-Residual Corrector predicts residual compensation from aligned TCP wrench feedback. Right: The reference pose xref is executed through the force-mixed PBIC layer at 125 Hz to produce a compliant pose command xcmd for stable contact interaction.
Fig. 3 : System setup and data streams. A teleoperated UR7e platform collects visuotactile, RGB, pose, and TCP wrench data. Visuotactile signals, RGB, and end-effector pose are sampled at 10 Hz, while TCP wrench is sampled at 60 Hz.
Fig. 4 : Evaluation tasks. Key stages of four benchmarks. From top to bottom: Chip Transfer , Plug Insertion , Whiteboard Wiping , and Block Assembly . Each row illustrates a typical execution sequence, with fingertip visuotactile observations visualized in the corner to reflect local deformations and interaction-state changes during contact.
Fig. 5 : Baseline Architectures. We compare our hierarchical MGP+MRC framework against the open-loop Diffusion Policy (DP) and the end-to-end Reactive Diffusion Policy (RDP).
Method
Chip Transfer
Surface Wiping
Plug Insertion
Block Assembly
Score
Pick
Move
Place
Pick
Contact
Wipe
Grasp
Align
Insert
Grasp
Align
Stack
Avg.
DP (Baseline)
43%
40%
37%
80%
67%
63%
70%
63%
43%
57%
53%
43%
0.55
DP w. tactile
63%
57%
53%
83%
73%
70%
77%
70%
53%
67%
60%
50%
0.65
RDP (Tactile)
90%
87%
83%
97%
93%
90%
90%
87%
73%
93%
87%
77%
0.87
RDP (Force)
70%
63%
60%
97%
93%
93%
87%
87%
80%
87%
83%
80%
0.82
M 2 -ResiPolicy
100%
97%
93%
97%
97%
97%
93%
93%
80%
97%
93%
87%
0.94
TABLE I : Policy Performance Across Task-Specific Execution Stages (Success Rate)
Fig. 6 : Failure Analysis. (Top) Baseline failures: Perception errors cause crushing/slippage; Control rigidity leads to force stops/jamming. (Bottom) Force profiles contrast unsafe baseline spikes (Red) with our stable regulation (Blue).
Contact-rich manipulation is central to many everyday human activities, requiring continuous adaptation to contact uncertainty and external disturbances through multi-modal perception, particularly vision and tactile feedback. While imitation learning has shown strong potential for learning complex manipulation behaviors, most existing approaches rely on action chunking, which fundamentally limits their ability to react to unforeseen observations during execution. This limitation becomes especially critical in contact-rich scenarios, where physical uncertainty and high-frequency tactile feedback demand rapid, reactive control. To address this challenge, we propose Tube Diffusion Policy (TDP), a novel reactive visual-tactile policy learning framework that bridges diffusion-based imitation learning with tube-based feedback control. By leveraging the expressive power of generative models, TDP learns an observation-conditioned feedback flow around nominal action chunks, forming an action tube that enables fast and adaptive reactions during execution. We evaluate TDP on the widely used Push-T benchmark and three additional challenging visual-tactile dexterous manipulation tasks. Across all benchmarks, TDP consistently outperforms state-of-the-art imitation learning baselines. Two real-world experiments further validate its robust reactivity under contact uncertainty and external disturbances. Moreover, the step-wise correction mechanism enabled by action tube significantly reduces the required denoising steps, making TDP well suited for real-time, high-frequency feedback control in contact-rich manipulation.
Teng Xue, Alberto Rigo, Bingjian Huang +4
Meta Reality Labs Research · Idiap Research Institute · École Polytechnique Fédérale de Lausanne (EPFL) +2
Contact-rich manipulation demands human-like integration of perception and force feedback: vision should guide task progress, while high-frequency interaction control must stabilize contact under uncertainty. Existing learning-based policies often entangle these roles in a monolithic network, trading off global generalization against stable local refinement, while control-centric approaches typically assume a known task structure or learn only controller parameters rather than the structure itself. In this paper, we formalize a physically grounded interaction frame, an instantaneous local basis that decouples force regulation from motion execution, and propose a method to recover it from demonstrations. Based on this, we address both issues by proposing Force Policy, a global-local vision-force policy in which a global policy guides free-space actions using vision, and upon contact, a high-frequency local policy with force feedback estimates the interaction frame and executes hybrid force-position control for stable interaction. Real-world experiments across diverse contact-rich tasks show consistent gains over strong baselines, with more robust contact establishment, more accurate force regulation, and reliable generalization to novel objects with varied geometries and physical properties, ultimately improving both contact stability and execution quality. Project page: https://force-policy.github.io/
Hongjie Fang, Shirun Tang, Mingyu Mei +9
Noematrix · Shanghai Jiao Tong University · Zhejiang University +2
Visual imitation policies can organise task motion, yet contact-rich object use demands gripper adjustments beyond coarse opening and closing. We study fine-grained gripper control as a local feedback problem within ongoing task execution. Our approach assigns tactile feedback a dedicated control role during contact-rich task stages. We introduce ContactPolicy, a learned local controller that takes over the gripper while a visual base policy continues to control the arm. ContactPolicy combines tactile and force histories with current robot state to jointly predict gripper increments and their modulation, updating the gripper target from each new observation. An Intent policy manages control handover between the visual base policy and ContactPolicy. Real-world experiments on container opening, object handling, and tool operation show improved task success over both the Visuomotor Policy and Tactile-Force Policy baselines. Execution analysis illustrates how the gripper adjusts contact within an ongoing task, including periods of target retention and fine-grained changes as interaction conditions evolve. Arm-state ablations and failure analysis further examine how execution context supports local regulation. These results support task-local tactile policies for fine-grained gripper control within visual manipulation.
Mingyuan Ba, Yanan Zhou, James Zhao +1
School of Computer Science, The University of Sydney, Australia