ReTac-ACT: A State-Gated Vision-Tactile Fusion Transformer for Precision Assembly

Minchi Ruan; LiangQing Zhou; Hongtong Li; Zongtao Wang; ZhaoMing Lu; Jianwei Zhang; Bin Fang

cs.ROMar 10, 2026

ReTac-ACT: A State-Gated Vision-Tactile Fusion Transformer for Precision Assembly

Authors: Minchi Ruan, LiangQing Zhou, Hongtong Li, Zongtao Wang, ZhaoMing Lu, Jianwei Zhang, Bin Fang

Organizations: Beijing University of Posts and Telecommunications, Beijing, China · SunHDex Intelligent Technology (Beijing) Co. Ltd., Beijing, China · University of Hamburg, Hamburg, Germany

Abstract

Precision assembly requires sub-millimeter corrections in contact-rich "last-millimeter" regions where visual feedback fails due to occlusion from the end-effector and workpiece. We present ReTac-ACT (Reconstruction-enhanced Tactile ACT), a vision-tactile imitation learning policy that addresses this challenge through three synergistic mechanisms: (i) bidirectional cross-attention enabling reciprocal visuo-tactile feature enhancement before fusion, (ii) a proprioception-conditioned gating network that dynamically elevates tactile reliance when visual occlusion occurs, and (iii) a tactile reconstruction objective enforcing learning of manipulation-relevant contact information rather than generic visual textures. Evaluated on the standardized NIST Assembly Task Board M1 benchmark, ReTac-ACT achieves 90% peg-in-hole success, substantially outperforming vision-only and generalist baseline methods, and maintains 80% success at industrial-grade 0.1mm clearance. Ablation studies validate that each architectural component is indispensable. The ReTac-ACT codebase and a vision-tactile demonstration dataset covering various clearance levels with both visual and tactile features will be released to support reproducible research.

Figures & tables

Fig. 1: We present ReTac-ACT, a state-gated vision-tactile policy that extends Action Chunking with Transformers (ACT) to natively process tactile feedback. ReTac-ACT sets a new state of the art for high-precision peg-in-hole tasks on the NIST ATB M1 benchmark provided by ManipulationNet, achieving 90% success at 3 mm clearance and maintaining 80% at industrial-grade 0.1 mm clearance where pure vision fails due to occlusion. It features a proprioception-conditioned gating mechanism to dynamically fuse visual and tactile modalities and is trained with auxiliary tactile reconstruction objectives. The ReTac-ACT code will be made open-source to support the research community.

Fig. 2: Overview of the ReTac-ACT architecture. (a) Multi-Modal Encoders : Visual inputs (3 RGB cameras) and tactile inputs (4 contact images: one sensor per fingertip, two fingers per gripper, bimanual) are processed by separate backbones into feature tokens. (b) Cross-Modal Dynamic Fusion : A proprioception-based gating network dynamically weighs modalities, enhanced by bidirectional cross-attention. (c) Action Generator : A CVAE-based transformer decoder predicts temporal action chunks $\hat{a}_{t:t+k-1}$ , where each action includes 14-DoF bimanual joint targets and 2 gripper commands.

Fig. 3: Tactile representation learning via auxiliary reconstruction. During training, an image reconstruction objective regularizes the tactile encoder. By employing a decoder to reconstruct the raw tactile inputs from the learned latent tokens, the model is explicitly forced to capture fine-grained contact geometry, preventing feature collapse.

Fig. 4: Hardware setup for the bimanual vision-tactile precision-assembly system.

Method	Missed	Grasp	Peg-in-hole
ACT [ 2 ]	40%	60%	40%
DP [ 1 ]	70%	30%	20%
pi05 [ 3 ]	55%	45%	20%
ReTac-ACT (ours)	0%	100%	90%

TABLE I: Quantitative results for peg-in-hole assembly at Level 1 (3 mm clearance). The best results are highlighted in bold.

Variant	Missed	Grasp	Peg-in-hole
w/o Reciprocal Fusion	0%	100%	5%
w/o Cross-Attention	0%	100%	5%
w/o TacRecon	5%	95%	15%
w/o StateGate	0%	100%	35%
ReTac-ACT (full)	0%	100%	90%

TABLE II: Ablation study on key components of ReTac-ACT. The best results are highlighted in bold.

Panel of Figure 7 — Fig. 5: Robustness to tighter clearances. As clearance tightens from 3 mm to 0.1 mm, ReTac-ACT degrades only 11% (90% $\to$ 80%) while ACT degrades 62.5% (40% $\to$ 15%) and Diffusion Policy (DP) fails (20% $\to$ 0%). Dashed lines highlight the degradation trends.

Fig. 6: Visual Attention Heatmap Comparison and Parameter Evolution. (a) Attention heatmaps of ACT, showing attention scattered on background regions across three task stages. (b) Attention heatmaps of DP. (c) Attention heatmaps of our ReTac-ACT. (d) Evolution of $\alpha_{t}$ during the task.

Explore similar work

From Reach to Insert: Tactile-Augmented Precision Assembly under Sub-Millimeter Tolerances
May 6, 2026Xinpan Meng, Siyao Huang, JingPu Yang +6RL for Robotics Contact-Rich Robotic Manipulation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
CAAT: Contact-Aware Attention Scaling and Tactile Masking for Data-Efficient Contact-Rich Manipulation
Aug 2, 2026Jiaming Jiang, Yuzhe Huang, Hao Liang +7Contact-Rich Robotic Manipulation Visuo-Tactile Representation Learning
ShanghaiTech University · Beijing Institute for General Artificial Intelligence · Beihang University +2
TACIT: Tactile Contact Supervision for Spatial Attention in Dexterous Manipulation
Sep 21, 2026Yanhou Lai, Fucai Zhu, Ruiqiang Wang +1Contact-Rich Robotic Manipulation Robot Imitation Learning
Graduate School of Information Sciences, Tohoku University, Sendai 980-8579, Japan

May 6, 2026J/K move · Enter open · S save

From Reach to Insert: Tactile-Augmented Precision Assembly under Sub-Millimeter Tolerances

Xinpan Meng, Siyao Huang, JingPu Yang, Muyuan Ma+5

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China

RL for Robotics Contact-Rich Robotic Manipulation Robotic Insertion Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

High-precision assembly frequently involves tight-tolerance insertions, where even slight pose errors can cause jamming or excessive interaction forces, making robust and safe insertion policies difficult to obtain. This paper proposes a tactile-augmented two-stage method that combines Imitation Learning (IL) and Reinforcement Learning (RL) for precision insertion tasks. In the first stage, IL learns a reaching policy with position generalization that grasps the peg and brings it to the vicinity of the target region. In the second stage, RL executes the insertion and enables recovery from failures during contact-rich interactions. To better exploit tactile feedback, we introduce tactile group sampling to increase coverage of critical contact segments during training, and design a tactile critic to more accurately evaluate policy values, improving insertion performance while maintaining low contact forces. We conduct systematic experiments across five hole geometries and three clearance settings. Results show that our method substantially improves insertion performance across all settings; under the most challenging 0.05,mm clearance, it achieves a 67% success rate while keeping contact forces low, reducing the maximum interaction force by 60% and torque by 44%, thereby validating both effectiveness and safety for precision assembly.

Abstract

Explore similar work

From Reach to Insert: Tactile-Augmented Precision Assembly under Sub-Millimeter Tolerances

CAAT: Contact-Aware Attention Scaling and Tactile Masking for Data-Efficient Contact-Rich Manipulation

TACIT: Tactile Contact Supervision for Spatial Attention in Dexterous Manipulation