Touch2Insert: Zero-Shot Peg Insertion by Touching Intersections of Peg and Hole

Masaru Yajima; Yuma Shin; Rei Kawakami; Asako Kanezaki; Kei Ota

cs.ROMar 4, 2026

Touch2Insert: Zero-Shot Peg Insertion by Touching Intersections of Peg and Hole

Authors: Masaru Yajima, Yuma Shin, Rei Kawakami, Asako Kanezaki, Kei Ota

Organizations: Institute of Science Tokyo, Tokyo, Japan. · Mitsubishi Electric, Kanagawa, Japan.

Abstract

Reliable insertion of industrial connectors remains a central challenge in robotics, requiring sub-millimeter precision under uncertainty and often without full visual access. Vision-based approaches struggle with occlusion and limited generalization, while learning-based policies frequently fail to transfer to unseen geometries. To address these limitations, we leverage tactile sensing, which captures local surface geometry at the point of contact and thus provides reliable information even under occlusion and across novel connector shapes. Building on this capability, we present Touch2Insert, a tactile-based framework for arbitrary peg insertion. Our method reconstructs cross-sectional geometry from high-resolution tactile images and estimates the relative pose of the hole with respect to the peg in a zero-shot manner. By aligning reconstructed shapes through registration, the framework enables insertion from a single contact without task-specific training. To evaluate its performance, we conducted experiments with three diverse connectors in both simulation and real-robot settings. The results indicate that Touch2Insert achieved sub-millimeter pose estimation accuracy for all connectors in simulation, and attained an average success rate of 86.7% on the real robot, thereby confirming the robustness and generalizability of tactile sensing for real-world robotic connector insertion.

Figures & tables

Fig. 1 : This paper addresses the problem of inserting an arbitrary peg into an unknown hole without any prior knowledge of their shapes or types. We propose a novel framework in which the robot makes contact with the cross-sections of the peg and hole, and estimates the hole’s relative pose with respect to the peg in $\mathrm{SE}(2)$ in a zero-shot manner from 3D point clouds obtained from tactile images.

Fig. 2 : Definition of coordinates. The coordinate frames are defined as follows: $F_{\mathrm{e}e}$ denotes the end-effector frame, $F_{\mathrm{ee}}$ the frame of the grasped peg, and $F_{\mathrm{h}}$ the hole frame. The objective is to estimate the relative pose of the hole with respect to the peg, ${}^{\mathrm{p}}_{\mathrm{h}}\hat{T}$ , which inevitably contains estimation noise. This transformation is then incorporated into the coordinate conversion described in Eq. 1 , yielding the estimated end-effector pose in the world frame. Finally, this pose is issued to the robot as a command for execution.

Fig. 3 : Touch2Insert : Overview of the proposed peg insertion framework. Tactile images are first converted into gradient maps and integrated to reconstruct 3D cross-sectional shapes of the peg and hole. The resulting point clouds are then refined by inverting the hole geometry, applying height-based filtering, projecting onto a 2D plane, and removing background artifacts. The cleaned planar point clouds are aligned using ICP with multiple initializations to estimate the relative $\mathrm{SE}(2)$ pose between peg and hole. Finally, the robot performs insertion under stiffness control, compensating for small residual errors without requiring exploratory search.

Fig. 4 : Connectors and example tactile images of the corresponding pegs and holes used in our experiments.

		Translation	Rotation
		Error (mm)	Error (degs)
OmniGlue	Audio Jack	0.78 (0.39)	-
	Lightning	2.25 (1.74)	31.64 (33.45)
	USB-C	2.42 (2.13)	28.29 (28.4)
w/o preprocess	Audio Jack	3.6 (1.21)	-
	Lightning	3.68 (1.05)	44.56 (30.73)

TABLE I : Average estimation errors (mean values, with standard deviations in parentheses) for each connector. Bold numbers show the best results.

Panel of Figure 6 — Fig. 5 : Successful and failed examples of OmniGlue baseline that uses edge detection and feature matching for the Lightning connector. For each image, the left shows the edge image of a peg, and the right shows a hole. In the successful case (a), matching occurs roughly at the corresponding positions of the peg and the hole. In contrast, in the failed case (b), there are regions where matching occurs at non-corresponding positions, leading to an inaccurate estimation.

Panel of Figure 7 — Fig. 6 : Registration results for successful and failed USB-C insertions. The red point cloud represents the peg and the blue point cloud represents the hole. In the failed case, distortions in the reconstruction produced irregularities in the hole point cloud, leading to a small angular misalignment with the peg and subsequent pose estimation errors.

Connector Type	Success rate
Audio Jack	95% (19/20)
Lightning	100% (20/20)
USB-C	65% (13/20)

TABLE II : Insertion success rate for each connector.

Connector Type	$\mathrm{MAE}_{G_{x}}$ (mm/pixel)		$\mathrm{MAE}_{G_{y}}$ (mm/pixel)		$\mathrm{MAE}_{\theta_{x}}$ (degs)		$\mathrm{MAE}_{\theta_{y}}$ (degs)
Connector Type	MLP	Ours	MLP	Ours	MLP	Ours	MLP	Ours
Audio Jack Peg	0.003	$\bm{0.002}$	0.004	$\bm{0.001}$	0.146	$\bm{0.077}$	0.203	$\bm{0.069}$
Audio Jack Hole	0.008	$\bm{0.007}$	0.010	$\bm{0.006}$	0.436	$\bm{0.362}$	0.547	$\bm{0.326}$
Lightning Peg	0.005	$\bm{0.004}$	0.007	$\bm{0.003}$	0.274	$\bm{0.196}$	0.369	$\bm{0.177}$
Lightning Hole	0.009	$\bm{0.008}$	0.011	$\bm{0.007}$	0.487	$\bm{0.450}$	0.601	$\bm{0.407}$
USB-C Peg	0.008	$\bm{0.007}$	0.010	$\bm{0.007}$	0.457	$\bm{0.400}$	0.560	$\bm{0.374}$

TABLE III : Mean absolute errors (MAEs) between the predicted and ground-truth components

G_{x}

G_{y}

\theta_{x}

, and

\theta_{y}

for gradient maps estimated by the baseline MLP and the proposed method. In each entry, the left and right values indicate the MLP-based model and the proposed model, respectively, and bold numbers show the better results. The proposed model achieves consistently lower errors, demonstrating more accurate tactile reconstruction.

Fig. 7 : Height maps reconstructed from gradient maps estimated by the baseline MLP model and the proposed model, shown at the same scale. The proposed method produces cleaner and more accurate height maps than the MLP baseline, highlighting the effectiveness of our reconstruction approach and its contribution to reliable connector insertion.

Explore similar work

TacZero: Training-Free Peg Insertion Using a General-Purpose Vision-Language Model with Tactile Feedback
Oct 6, 2026Kazutoshi TanakaZero-Shot Robotic Manipulation Contact-Rich Robotic Manipulation
OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan
ContactDP: Contact-Guided Diffusion Policy for Tight Insertion Tasks
Sep 20, 2026Chengyi Xing, Shaoxiong Yao, Diego Romeres +1Contact-Rich Robotic Manipulation Diffusion Policy
Mitsubishi Electric Research Labs, Cambridge, MA, US
InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality
Sep 21, 2026Zhenghua Ma, Xinpan Meng, Zeyu Liu +4RL for Robotics Contact-Rich Robotic Manipulation
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China, and the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · PaXini AI Technology (Beijing) Co., Ltd., Beijing 100080, China

Oct 6, 2026J/K move · Enter open · S save

TacZero: Training-Free Peg Insertion Using a General-Purpose Vision-Language Model with Tactile Feedback

Kazutoshi Tanaka

OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan

Zero-Shot Robotic Manipulation Contact-Rich Robotic Manipulation Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

Robots that autonomously determine their actions from language instructions and sensory observations could perform new contact-rich manipulation tasks without task-specific training or hand-designed rules. To perform these tasks, robots must infer how objects contact one another and move as a result, then select actions. For contact inference and action selection, prior approaches involve designing estimation models and tactile feedback control laws, or learning models for object-motion estimation, action-outcome prediction, and action selection from tactile data. Instead, we propose TacZero, which uses a pretrained general-purpose vision-language model (VLM) to interpret visual and tactile observations and select robot actions without additional tactile or manipulation training or task-specific rules for contact interpretation or action selection. TacZero provides the VLM with camera images, robot state, and three-axis tactile responses represented as numerical values or vectors overlaid on the images. From these observations and interaction history, the VLM generates commands specifying target end-effector positions and gripper opening or closing, which a low-level controller executes. In real-world cylindrical-peg insertion experiments, TacZero succeeded in 15 of 20 trials with numerical tactile input, compared with 10 of 20 without tactile input. This study provides a concrete starting point for further research on contact-rich manipulation using general-purpose VLMs and highlights challenges in pursuing this direction.

Abstract

Explore similar work

TacZero: Training-Free Peg Insertion Using a General-Purpose Vision-Language Model with Tactile Feedback

ContactDP: Contact-Guided Diffusion Policy for Tight Insertion Tasks

InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality