Reliable insertion of industrial connectors remains a central challenge in robotics, requiring sub-millimeter precision under uncertainty and often without full visual access. Vision-based approaches struggle with occlusion and limited generalization, while learning-based policies frequently fail to transfer to unseen geometries. To address these limitations, we leverage tactile sensing, which captures local surface geometry at the point of contact and thus provides reliable information even under occlusion and across novel connector shapes. Building on this capability, we present Touch2Insert, a tactile-based framework for arbitrary peg insertion. Our method reconstructs cross-sectional geometry from high-resolution tactile images and estimates the relative pose of the hole with respect to the peg in a zero-shot manner. By aligning reconstructed shapes through registration, the framework enables insertion from a single contact without task-specific training. To evaluate its performance, we conducted experiments with three diverse connectors in both simulation and real-robot settings. The results indicate that Touch2Insert achieved sub-millimeter pose estimation accuracy for all connectors in simulation, and attained an average success rate of 86.7% on the real robot, thereby confirming the robustness and generalizability of tactile sensing for real-world robotic connector insertion.
Figures & tables
Fig. 1 : This paper addresses the problem of inserting an arbitrary peg into an unknown hole without any prior knowledge of their shapes or types. We propose a novel framework in which the robot makes contact with the cross-sections of the peg and hole, and estimates the hole’s relative pose with respect to the peg in SE(2) in a zero-shot manner from 3D point clouds obtained from tactile images.
Fig. 2 : Definition of coordinates. The coordinate frames are defined as follows: Fee denotes the end-effector frame, Fee the frame of the grasped peg, and Fh the hole frame. The objective is to estimate the relative pose of the hole with respect to the peg, hpT^ , which inevitably contains estimation noise. This transformation is then incorporated into the coordinate conversion described in Eq. 1 , yielding the estimated end-effector pose in the world frame. Finally, this pose is issued to the robot as a command for execution.
Fig. 3 : Touch2Insert : Overview of the proposed peg insertion framework. Tactile images are first converted into gradient maps and integrated to reconstruct 3D cross-sectional shapes of the peg and hole. The resulting point clouds are then refined by inverting the hole geometry, applying height-based filtering, projecting onto a 2D plane, and removing background artifacts. The cleaned planar point clouds are aligned using ICP with multiple initializations to estimate the relative SE(2) pose between peg and hole. Finally, the robot performs insertion under stiffness control, compensating for small residual errors without requiring exploratory search.
Fig. 4 : Connectors and example tactile images of the corresponding pegs and holes used in our experiments.
Translation
Rotation
Error (mm)
Error (degs)
OmniGlue
Audio Jack
0.78 (0.39)
-
Lightning
2.25 (1.74)
31.64 (33.45)
USB-C
2.42 (2.13)
28.29 (28.4)
w/o preprocess
Audio Jack
3.6 (1.21)
-
Lightning
3.68 (1.05)
44.56 (30.73)
TABLE I : Average estimation errors (mean values, with standard deviations in parentheses) for each connector. Bold numbers show the best results.
Fig. 5 : Successful and failed examples of OmniGlue baseline that uses edge detection and feature matching for the Lightning connector. For each image, the left shows the edge image of a peg, and the right shows a hole. In the successful case (a), matching occurs roughly at the corresponding positions of the peg and the hole. In contrast, in the failed case (b), there are regions where matching occurs at non-corresponding positions, leading to an inaccurate estimation.
Fig. 6 : Registration results for successful and failed USB-C insertions. The red point cloud represents the peg and the blue point cloud represents the hole. In the failed case, distortions in the reconstruction produced irregularities in the hole point cloud, leading to a small angular misalignment with the peg and subsequent pose estimation errors.
Connector Type
Success rate
Audio Jack
95% (19/20)
Lightning
100% (20/20)
USB-C
65% (13/20)
TABLE II : Insertion success rate for each connector.
Connector Type
MAEGx (mm/pixel)
MAEGy (mm/pixel)
MAEθx (degs)
MAEθy (degs)
MLP
Ours
MLP
Ours
MLP
Ours
MLP
Ours
Audio Jack Peg
0.003
0.002
0.004
0.001
0.146
0.077
0.203
0.069
Audio Jack Hole
0.008
0.007
0.010
0.006
0.436
0.362
0.547
0.326
Lightning Peg
0.005
0.004
0.007
0.003
0.274
0.196
0.369
0.177
Lightning Hole
0.009
0.008
0.011
0.007
0.487
0.450
0.601
0.407
USB-C Peg
0.008
0.007
0.010
0.007
0.457
0.400
0.560
0.374
TABLE III : Mean absolute errors (MAEs) between the predicted and ground-truth components Gx , Gy , θx , and θy for gradient maps estimated by the baseline MLP and the proposed method. In each entry, the left and right values indicate the MLP-based model and the proposed model, respectively, and bold numbers show the better results. The proposed model achieves consistently lower errors, demonstrating more accurate tactile reconstruction.
Fig. 7 : Height maps reconstructed from gradient maps estimated by the baseline MLP model and the proposed model, shown at the same scale. The proposed method produces cleaner and more accurate height maps than the MLP baseline, highlighting the effectiveness of our reconstruction approach and its contribution to reliable connector insertion.
Robots that autonomously determine their actions from language instructions and sensory observations could perform new contact-rich manipulation tasks without task-specific training or hand-designed rules. To perform these tasks, robots must infer how objects contact one another and move as a result, then select actions. For contact inference and action selection, prior approaches involve designing estimation models and tactile feedback control laws, or learning models for object-motion estimation, action-outcome prediction, and action selection from tactile data. Instead, we propose TacZero, which uses a pretrained general-purpose vision-language model (VLM) to interpret visual and tactile observations and select robot actions without additional tactile or manipulation training or task-specific rules for contact interpretation or action selection. TacZero provides the VLM with camera images, robot state, and three-axis tactile responses represented as numerical values or vectors overlaid on the images. From these observations and interaction history, the VLM generates commands specifying target end-effector positions and gripper opening or closing, which a low-level controller executes. In real-world cylindrical-peg insertion experiments, TacZero succeeded in 15 of 20 trials with numerical tactile input, compared with 10 of 20 without tactile input. This study provides a concrete starting point for further research on contact-rich manipulation using general-purpose VLMs and highlights challenges in pursuing this direction.
Kazutoshi Tanaka
OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan
High-precision connector insertion remains challenging for robotic systems due to tight mechanical tolerances, partial observability during contact, and multimodal uncertainty arising from occlusion and contact ambiguity. Successful insertion requires closed-loop contact guidance that continuously integrates global alignment cues with local contact feedback to produce stable corrective actions under interaction. In this work, we present ContactDP (Contact-Guided Diffusion Policy for Tight Insertion Tasks), a multimodal diffusion-policy framework for contact-rich insertion. ContactDP jointly integrates wrist RGB observations, fingertip tactile sensing, and wrist-mounted force-torque measurements to infer contact state and generate temporally consistent corrective motions during insertion. To ensure stable execution under contact, the learned policy operates together with a hybrid position-force controller that provides compliant low-level interaction. We evaluate our approach on a suite of industrial-grade connector insertion tasks with varying connector geometries, grasp conditions, and initial misalignment. Across all tasks, ContactDP significantly outperforms vision-only diffusion policies for performance, reliability and generalization.
Chengyi Xing, Shaoxiong Yao, Diego Romeres +1
Mitsubishi Electric Research Labs, Cambridge, MA, US
Contact-rich precision insertion is a key manipulation skill in robotic assembly. Tight clearances make insertion more sensitive to alignment errors and prone to collisions and jamming, while variations in geometry and clearance across parts further complicate policy reuse. We present a reinforcement learning framework that trains insertion policies entirely in simulation for direct deployment without real-world demonstrations or policy fine-tuning. By combining target poses with compact three-dimensional fingertip force feedback, the policy learns to search for alignment and correct its motion despite errors in the estimated hole position. A decoupled gated reward coordinates alignment and insertion. Force-signal smoothing and state-independent standard deviations stabilize the learning process. The resulting policies perform real-world insertion across multiple hole geometries with a minimum nominal clearance of 0.02 mm and improve success while reducing peak contact forces under hole-position errors. Cross-clearance and cross-geometry evaluations further confirm policy generalization. The system achieved the first perfect score of 20/20 on ManipulationNet's peg-in-hole benchmark under its Human-in-the-Loop protocol, with fully autonomous insertion motions. A single policy trained only on a simulated hexagonal insertion task achieved an overall success rate of 95.0% across eight unseen real-world insertion tasks. These results show that learning entirely in simulation can yield precision insertion skills that can be deployed directly and reused across real-world tasks. The project website (https://mzhsoul.github.io/InsertAnything/) provides open-source simulation and real-robot experiment scripts, assets, and trained checkpoints.
Zhenghua Ma, Xinpan Meng, Zeyu Liu +4
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China, and the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · PaXini AI Technology (Beijing) Co., Ltd., Beijing 100080, China