PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation

Guo Tang; Yongtao Wang

cs.CVOct 6, 2026

PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation

Authors: Guo Tang, Yongtao Wang

Organizations: Wangxuan Institute of Computer Technology, Peking University · VGI Labs Co., Ltd.

Abstract

Realistic physical interaction is a cornerstone of embodied intelligence, yet collecting paired visual--tactile data remains costly. Visual-to-tactile synthesis offers a promising approach to augmenting such data, but learning this mapping is complicated by the gap between visual appearance and contact-related material properties, as well as spatial misalignment in paired observations. To address these challenges, we present \textbf{PhysTacGen}, a visual-to-optical-tactile image generation framework that integrates material-aware descriptions with geometric conditioning. First, we introduce Group Tactile Policy Optimization (GTPO), a reinforcement learning strategy that refines a vision--language model to generate structured material descriptions using task-specific rewards. Second, we combine DINOv2-based pair curation with monocular relative-depth estimation to select training pairs and provide geometric priors. Finally, an SDXL ControlNet synthesizes optical tactile images conditioned on RGB, relative depth, and GTPO-generated text. Experiments on curated SSVTP data demonstrate improved structural similarity over the compared baselines, while a blinded user study shows a preference for GTPO-generated descriptions. Generated tactile inputs also improve performance on an attribute-derived force-coefficient prediction proxy. Together, these results demonstrate the effectiveness of PhysTacGen for optical tactile image synthesis and its utility in the evaluated downstream task.The code will be available at https://github.com/VDIGPKU/PhysTacGen.

Figures & tables

Figure 1: The PhysTacGen Framework. Given a visual observation, our system operates in three sequential stages: (1) Physics-Aware Reasoning: A VLM fine-tuned via GTPO infers latent physical properties. (2) Data Curation & Spatial Prior: The dataset is semantically purified using DINOv2, and input RGB is depth-estimated to form a 4-channel geometric condition. (3) Multi-Modal Synthesis: A physics-driven SDXL ControlNet synthesises the high-fidelity tactile sensor image $I_{t}$ . Here $I_{v}$ is the external visual RGB image, $I_{d}$ the estimated relative-depth prior, and $c_{\mathrm{phys}}$ the GTPO-generated physical description; the optical tactile image is $I_{t}$ .

Figure 2: Qualitative Synthesis Results. Visualisation of generated tactile images across different settings. SDXL (Baseline) fails to ground the contact location. The Mix-Fusion and four-channel variants change both fusion and loss and therefore do not isolate a single cause of their visual differences. The examples illustrate generated contact appearance under the original conditions.

Method and condition	SSIM $\uparrow$	LPIPS $\downarrow$	legacy nFD $\downarrow$
Pix2Pix / RGB	0.106	0.819	0.829
Pix2Pix-Turbo / RGB + generic prompt	0.417	0.692	0.628
VisGel (adapted) / RGB + train reference	0.715	0.629	0.433
SDXL baseline / physical text only	0.190	0.859	0.838
Mix-Fusion / RGB + depth, physical text	0.938	0.617	0.241
Four-channel / RGB + depth, generic text	0.908	0.637	0.349

Table 1: External baselines and original generator variants. All rows use the same retained SSVTP split. The last column is the legacy Fréchet statistic divided by evaluation-set size (legacy nFD), not FID.

Figure 3: Material and Textural Consistency under Spatial Misalignment. Red boxes highlight areas for qualitative texture comparison.The details in the areas outlined in red in each set of images are similar.

Prompt source	SSIM $\uparrow$	LPIPS $\downarrow$	legacy nFD $\downarrow$
7-label classifier	0.919	0.557	0.305
SFT	0.948	0.570	0.255
GTPO	0.964	0.559	0.278

Table 2: Final matched end-to-end prompt-source comparison. Each prompt source trains and tests a separate generator under the same conditions; legacy nFD is not standard FID.

Input modality	Run 1	Run 2	Mean $\pm$ sample SD
RGB only	0.144	0.147	$0.1455\pm 0.0021$
RGB + generated tactile	0.132	0.134	$0.1330\pm 0.0014$
RGB + real tactile (oracle)	0.130	0.132	$0.1310\pm 0.0014$

Table 3: Test MSE for the heuristic force-coefficient prediction proxy. Sample SD describes variation across from two random seeds; the last column reports their mean and sample standard deviation.

Method	SES $\uparrow$	BLEU $\uparrow$	ROUGE $\uparrow$	PPL $\downarrow$	Judge $\uparrow$	Human $\uparrow$
7-label classifier	0.732	29.3	52.7	12.9	2.94	16%
SFT Qwen	0.461	18.4	32.7	14.3	3.42	27%
GTPO Qwen	0.586	26.8	44.5	10.8	4.65	57%

Table 4: Description evaluation on 452 pairs. The human column is a blinded forced choice among classifier, SFT, and GTPO descriptions by 22 participants, with randomized presentation and one choice per image.

Explore similar work

ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation
May 26, 2025Dongyu Luo, Kelin Yu, Amir-Hossein Shahidzadeh +3Tactile Imitation Learning
The University of Hong Kong · University of Maryland · University of Maryland, College Park
TacGen: Touch Is a Necessary Dimension of Physical-World Representation -- Addressing Tactile Data Scarcity with Scalable Vision-to-Touch Alignment and Generation
Jun 28, 2026Wanghao Ye, Aarosh Das, Sihan Chen +19Tactile Tac
University of Maryland, College Park · Carnegie Mellon University · University of Alabama +2
UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation
Jun 30, 2026Jiahang Tu, Fengyu Yang, Chenyang Ma +8Tactile Multimodal Model
Zhejiang University · Yale University · UNIX AI +3

May 26, 2025, cs.CVJ/K move · Enter open · S save

ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation

Dongyu Luo, Kelin Yu, Amir-Hossein Shahidzadeh, Cornelia Fermüller+2

The University of Hong Kong · University of Maryland · University of Maryland, College Park

Tactile Imitation Learning

Open paper AskPDF arXiv

Vision-based tactile sensing is widely used in perception, reconstruction, and robotic manipulation, yet collecting large-scale tactile data remains costly due to diverse sensor-object interactions and inconsistencies across sensor instances. Existing approaches to scaling tactile data---simulation and free-form tactile generation---often yield unrealistically rendered signals with poor transfer to highly dynamic real-world tasks. We propose \name, a two-stage controllable tactile image generation framework that generates realistic tactile images conditioned on a single reference tactile image, contact force, and contact pose. By grounding generation in these important physical priors, \name synthesizes realistic samples across different sensors while effectively capturing task-relevant variations. Across a series of downstream tasks and real-world experiments, such as object insertion, imitation learning, and object weighting, the augmented datasets using our approach consistently improve performance and demonstrate practical utility in dynamic real-world settings. Project page: https://dongyuluo.github.io/controltac.

Abstract

Explore similar work

ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation

TacGen: Touch Is a Necessary Dimension of Physical-World Representation -- Addressing Tactile Data Scarcity with Scalable Vision-to-Touch Alignment and Generation

UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation