Vision-Force Admittance Learning for Peg Insertion into a Movable Hole

Yuzhong Chen; Yongqing Liang; Yunzhi Xu; Irving Fang; Chase Kidder; Hui-ping Wang; Raihan Haque; Yubiao Zhang

cs.ROSep 12, 2026

Vision-Force Admittance Learning for Peg Insertion into a Movable Hole

Authors: Yuzhong Chen, Yongqing Liang, Yunzhi Xu, Irving Fang, Chase Kidder, Hui-ping Wang, Raihan Haque, Yubiao Zhang,

Organizations: New York University, Brooklyn, NY 11201, USA · General Motors, Warren, MI 48073, USA

Abstract

Precise manipulation in dynamic environments, whether induced by a mobile robot base or a target with unknown motion, remains a major challenge in robotics. Manipulation in dynamic environments introduces substantial uncertainty, which fundamentally conflicts with the tight precision requirement of precise tasks such as peg-in-the-hole. We propose a Vision-Force Admittance Learning (VFAL) framework that fuses asynchronous visual feedback with a high-frequency force-based model, using visual pose estimations as a regularization term. VFAL adapts insertion strategies online to dynamic motion while maintaining millimeter-level precision. To obtain robust, low-frequency pose information, we employ state-of-the-art vision foundation models for visual pose estimation. Additionally, we incorporate failure recovery mechanisms to enhance overall robustness. We validate our approach in real-world experiments, demonstrating high success rates and strong adaptability to various pegs and dynamic environments.

Figures & tables

Fig. 2: Algorithm Workflow. A two-stage pipeline is designed for the manipulation process. In the first stage, before making contact with the base, the robot approaches the target using the pose estimation result. In the second stage, once contact is detected via the force sensor, the model predicts the optimal next step control based on the current state and force readings, while referencing the asynchronously updated pose estimation as a regularization term.

Fig. 3: Asynchronous Vision–Force Controller. In each control cycle, the force-based model updates itself using force/torque feedback, the robot’s previous control inputs, and its previous state. When available, visual information is incorporated asynchronously. The model predicts the next-step control by minimizing the discrepancy between the expected and the sensed force and pose, using the current robot pose, force/torque readings, and visual input.

Description	Method	Modality	Success Rate	Full Wrench
1D Mobile Base (Cylinder Christmas tree nut)	Vision	Vision	40%	57.11 $(\pm 59.65)$
	Admittance	Force	60%	70.96 $(\pm 29.78)$
	Online Learning	Force	70%	80.95 $(\pm 42.56)$
	Synchronous VFAL	Force, Vision	90%	28.50 $(\pm 14.11)$
	VFAL	Force, Vision	100%	13.95 $(\pm\textbf{5.05})$
3D Mobile Base (Cross-fin Christmas tree nut)	Vision	Vision	20%	114.60 $(\pm 57.30)$

TABLE I: Success rate and Mean Squared Error (MSE) on full wrench across different models for both fixed-base and mobile-base insertions. We mark the best performed results for each experiment in bold .

Fig. 4: TPU-based gripper . It serves as an indicator of force and manipulation status visualizer through its deformation.

Fig. 5: Real world experiment setup . In our real world experiment setup, an RGB-D camera is used for pose estimation. The robot, equipped with a force/torque sensor, holds the deformable pushnut and performs the insertion task. The setup includes two types of bases: a fixed base mounted on a motherboard, and a cable-driven movable base mounted on a fixed linear track. This configuration allows us to evaluate insertion performance under both static and dynamic conditions.

Method	Success Rate	Full Wrench MSE
VFAL without Online Learning	70%	12.11 $(\pm 8.04)$
VFAL	80%	9.22 $(\pm\textbf{5.23})$

TABLE II: Ablation on Online Learning (3D Mobile Base)

Method	Success Rate	Full Wrench MSE
$\lambda=1$	70%	73.68 $(\pm 50.89)$
$\lambda=0.1$	80%	9.22 $(\pm\textbf{5.23})$
$\lambda=0.01$	40%	67.89 $(\pm 68.73)$

TABLE III: Ablation on

\lambda

Selection (3D Mobile Base)

Method	Success Rate	Full Wrench MSE
Synchronous VFAL	60%	37.82 $(\pm 15.83)$
High Frequency Controller	60%	54.95 $(\pm 41.11)$
VFAL	80%	9.22 $(\pm\textbf{5.23})$

TABLE IV: Ablation on the robot controller (3D Mobile Base)

Method	Force	Torque	Full Wrench
Vision	100.32 $(\pm 51.39)$	14.28 $(\pm 5.91)$	114.60 $(\pm 57.30)$
Admittance	95.02 $(N/A)$	11.07 $(N/A)$	106.09 $(N/A)$
Online Learning	N/A $(N/A)$	N/A $(N/A)$	N/A $(N/A)$
Sync VFAL	33.26 $(\pm 13.99)$	4.56 $(\pm 1.94)$	37.82 $(\pm 15.83)$
VFAL	8.07 $(\pm\textbf{4.56})$	1.15 $(\pm\textbf{0.68})$	9.22 $(\pm\textbf{5.23})$

TABLE V: Ablation on force/torque (3D Mobile Base)

Method	Approaching	Insertion	Overall
Vision	10/17	2/10	2/17
Admittance	10/10	1/10	1/10
Online Learning	10/12	0/10	0/12
Synchronous VFAL	10/13	6/10	6/13
VFAL	10/12	8/10	8/12

TABLE VI: Ablation on full system (3D Mobile Base)

Robotic Insertion

Robotic insertion studies how robots place objects through openings or into tissue despite tight tolerances, uncertain alignment, and contact forces.

See all 46 papers

Touch2Insert: Zero-Shot Peg Insertion by Touching Intersections of Peg and Hole
Mar 4, 2026Masaru Yajima, Yuma Shin, Rei Kawakami +2Robotic Insertion Tactile Sensing for Robotic Manipulation
Institute of Science Tokyo, Tokyo, Japan. · Mitsubishi Electric, Kanagawa, Japan.
Visual-Tactile Peg-in-Hole Assembly Learning from Peg-out-of-Hole Disassembly
Apr 22, 2026Yongqiang Zhao, Xuyang Zhang, Zhuo Chen +3RL for Robotics Contact-Rich Robotic Manipulation
Department of Engineering, King’s College London, Strand, London, WC2R 2LS, United Kingdom · Department of Informatics, King’s College London, Strand, London, WC2R 2LS, United Kingdom
InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality
Sep 21, 2026Zhenghua Ma, Xinpan Meng, Zeyu Liu +4RL for Robotics Contact-Rich Robotic Manipulation
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China, and the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · PaXini AI Technology (Beijing) Co., Ltd., Beijing 100080, China

Mar 4, 2026J/K move · Enter open · S save

Touch2Insert: Zero-Shot Peg Insertion by Touching Intersections of Peg and Hole

Masaru Yajima, Yuma Shin, Rei Kawakami, Asako Kanezaki+1

Institute of Science Tokyo, Tokyo, Japan. · Mitsubishi Electric, Kanagawa, Japan.

Robotic Insertion Tactile Sensing for Robotic Manipulation Zero-Shot Robotic Manipulation Object Pose Estimation

Open paper AskPDF arXiv

Reliable insertion of industrial connectors remains a central challenge in robotics, requiring sub-millimeter precision under uncertainty and often without full visual access. Vision-based approaches struggle with occlusion and limited generalization, while learning-based policies frequently fail to transfer to unseen geometries. To address these limitations, we leverage tactile sensing, which captures local surface geometry at the point of contact and thus provides reliable information even under occlusion and across novel connector shapes. Building on this capability, we present Touch2Insert, a tactile-based framework for arbitrary peg insertion. Our method reconstructs cross-sectional geometry from high-resolution tactile images and estimates the relative pose of the hole with respect to the peg in a zero-shot manner. By aligning reconstructed shapes through registration, the framework enables insertion from a single contact without task-specific training. To evaluate its performance, we conducted experiments with three diverse connectors in both simulation and real-robot settings. The results indicate that Touch2Insert achieved sub-millimeter pose estimation accuracy for all connectors in simulation, and attained an average success rate of 86.7% on the real robot, thereby confirming the robustness and generalizability of tactile sensing for real-world robotic connector insertion.

Abstract

Robotic Insertion

Explore similar work

Touch2Insert: Zero-Shot Peg Insertion by Touching Intersections of Peg and Hole

Visual-Tactile Peg-in-Hole Assembly Learning from Peg-out-of-Hole Disassembly

InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality