DeTac: Dual-branch Decoupling of Texture and Force Signals in Vision-Based Tactile Sensors

Zhirui Zhang; Yunkai Tang; Jiahang Li; Fan Fei; Jinnuo Zhang; Kairan Ding; Jieji Ren; Yandong Guo

cs.ROOct 12, 2026

DeTac: Dual-branch Decoupling of Texture and Force Signals in Vision-Based Tactile Sensors

Authors: Zhirui Zhang, Yunkai Tang, Jiahang Li, Fan Fei, Jinnuo Zhang, Kairan Ding, Jieji Ren, Yandong Guo,

Organizations: State Key Laboratory of Multimedia Information Processing, National Engineering Research Center of Visual Technology, and PKU-AI2 Robotics Joint Lab of Embodied AI, School of Computer Science, Peking University · AI2 Robotics · State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, and also with Shanghai Key Laboratory of Intelligent Robotics, Shanghai Jiao Tong University, Shanghai

Abstract

Vision-based tactile sensors enable fine-grained perception for dexterous robotic manipulation, but current marker-based designs tightly couple surface texture and contact force signals within a single observation. Existing data-driven decoupling methods rely on serial mask-based inpainting, which suffers from mask-sizing dilemmas caused by physical indentation effects, error retention, and lack of real paired data for supervision. To address these challenges, we propose DeTac, a novel dual-branch generative framework that concurrently extracts clean shading and marker images from a single mixed tactile image. DeTac leverages full-image synthesis to naturally bypass mask dependency and introduces a cross-branch synergistic optimization that enables mutual supervision, effectively preventing error retention. Furthermore, we construct a strictly aligned, real-world paired tactile dataset using custom hardware to provide high-quality supervision. Extensive experiments demonstrate that DeTac achieves state-of-the-art decoupling accuracy, significantly facilitating downstream surface reconstruction and force estimation tasks while exhibiting cross-sensor generalization.

Figures & tables

Figure 1: (a) Markerless sensors capture shading images to reconstruct surface but lack spatial references for force estimation. (b) Marker-based sensors enable force sensing via embedded markers, but introduce coupled signals and indentation artifacts. (c) Serial pipelines suffer from mask-size dilemmas, error retention, and lack of real paired training data. (d) DeTac uses a dual-branch framework with cross-branch optimization, trained on a self-collected aligned paired real dataset.

Figure 2: Overall pipeline of DeTac. (a) DeTac takes as input a mixed tactile image and consists of two parallel branches ( Sec. 3.1 ): a conditional diffusion-based shading restoration branch and a lightweight marker extraction branch. (b) The cross-branch synergistic optimization ( Sec. 3.2 ) exploits marker-aware spatial structures for mutual supervision using surrounding indentation loss for shading restoration and change-marker consistency loss for marker extraction.

Figure 3: Hardware design and data acquisition setup. (a) Exploded view of the MC-Tac [ 52 ] sensor equipped with an outer shell featuring 20 cylindrical alignment pins arranged in a ring for repeatable positioning. (b) The data acquisition system features a collection pad with a 5 $\times$ 5 array of interchangeable texture units, each surrounded by 20 holes that mate with the sensor’s alignment pins. Rotating the sensor and fully inserting the pins into different indexed holes under consistent force enable repeatable sampling of 20 contact poses per object before and after marker fabrication.

Panel of Figure 4 — Figure 4: 3D printed hardware for dataset collection.

Method	Unseen pose				Unseen object
Method	FID $\downarrow$	KID $\times 10^{-2}$ $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	KID $\times 10^{-2}$ $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
TELEA [ 23 ]	237.95	18.327	31.33	0.870	232.66	17.592	31.29	0.868
TacDiff [ 21 ]	158.94	11.447	31.52	0.866	155.63	10.654	31.40	0.863
LaMa [ 54 ]	358.16	37.535	29.09	0.804	360.27	37.117	29.16	0.803
SD 1.5 [ 20 ]	121.70	7.401	25.44	0.861	114.29	6.703	24.16	0.826
U-Net	108.35	7.489	38.56	0.950	111.28	7.750	39.03	0.942

Table 1: Quantitative results of shading restoration. The best results are highlighted in bold .

	UVTac [ 17 ]	SAM 3.1 [ 55 ]	DeTac (w/o $\mathcal{L}_{\rm sur}$ )	DeTac (w/o $\mathcal{L}_{\rm con}$ )	DeTac (Full)
MCLE $\downarrow$	0.21	0.41	0.00839	0.00832	0.00762

Table 2: Quantitative evaluation of marker extraction. The best result is highlighted in bold .

Figure 5: Qualitative comparison of shading restoration results.

Figure 6: Qualitative comparison of marker extraction results.

Figure 9: Surface texture reconstruction results.

Table 11

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Pixel-wise errors caused by indentation artifacts.

Figure A2: Visualization of paired data samples from each of the 54 texture units, arranged in 12 rows and 9 columns. Odd rows display the mixture image $\mathbf{I}_{\rm mix}$ with markers, and even rows show the corresponding shading image $\mathbf{I}_{\rm sha}$ directly below each mixture image.

Input Type	Success Rate			Overall
Input Type	Fabric A	Fabric B	Fabric C	Overall
Raw mixed image	16.7%	43.3%	90.0%	50.0%
Our marker-removed shading image	83.3 %	73.3 %	96.7 %	84.4 %

Appendix

Table A1: Fabric texture classification success rates across three fabric types.

Vision-Based Tactile Sensing

Vision-based tactile sensing uses cameras to read contact-induced changes in soft sensor surfaces, giving robots detailed information about touch.

See all 60 papers

ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation
May 26, 2025Dongyu Luo, Kelin Yu, Amir-Hossein Shahidzadeh +3Vision-Based Tactile Sensing Synthetic Data Generation for Robotics
The University of Hong Kong · University of Maryland · University of Maryland, College Park
TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception
Jun 24, 2026Lan Wei, Gurmeher Khurana, Sirine Bhouri +6Vision-Based Tactile Sensing Few-Shot Learning
Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK
VQ-Touch: A Data-Efficient Tactile Generation Framework Across Sensors and Scenarios
Jul 16, 2026Kailin Lyu, Long Xiao, Jianing Zeng +3Tactile Sensing Vector-Quantized Variational Autoencoder
Institute of Automation, Chinese Academy of Sciences

May 26, 2025J/K move · Enter open · S save

ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation

Dongyu Luo, Kelin Yu, Amir-Hossein Shahidzadeh, Cornelia Fermüller+2

The University of Hong Kong · University of Maryland · University of Maryland, College Park

Vision-Based Tactile Sensing Synthetic Data Generation for Robotics Tactile Sensing for Robotic Manipulation Controllable Image Generation

Open paper AskPDF arXiv

Vision-based tactile sensing is widely used in perception, reconstruction, and robotic manipulation, yet collecting large-scale tactile data remains costly due to diverse sensor-object interactions and inconsistencies across sensor instances. Existing approaches to scaling tactile data---simulation and free-form tactile generation---often yield unrealistically rendered signals with poor transfer to highly dynamic real-world tasks. We propose \name, a two-stage controllable tactile image generation framework that generates realistic tactile images conditioned on a single reference tactile image, contact force, and contact pose. By grounding generation in these important physical priors, \name synthesizes realistic samples across different sensors while effectively capturing task-relevant variations. Across a series of downstream tasks and real-world experiments, such as object insertion, imitation learning, and object weighting, the augmented datasets using our approach consistently improve performance and demonstrate practical utility in dynamic real-world settings. Project page: https://dongyuluo.github.io/controltac.

Abstract

Vision-Based Tactile Sensing

Explore similar work

ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

VQ-Touch: A Data-Efficient Tactile Generation Framework Across Sensors and Scenarios