PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation

Guo Tang; Yongtao Wang

cs.CVOct 6, 2026

PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation

Authors: Guo Tang, Yongtao Wang

Organizations: Wangxuan Institute of Computer Technology, Peking University · VGI Labs Co., Ltd.

Abstract

Realistic physical interaction is a cornerstone of embodied intelligence, yet collecting paired visual--tactile data remains costly. Visual-to-tactile synthesis offers a promising approach to augmenting such data, but learning this mapping is complicated by the gap between visual appearance and contact-related material properties, as well as spatial misalignment in paired observations. To address these challenges, we present \textbf{PhysTacGen}, a visual-to-optical-tactile image generation framework that integrates material-aware descriptions with geometric conditioning. First, we introduce Group Tactile Policy Optimization (GTPO), a reinforcement learning strategy that refines a vision--language model to generate structured material descriptions using task-specific rewards. Second, we combine DINOv2-based pair curation with monocular relative-depth estimation to select training pairs and provide geometric priors. Finally, an SDXL ControlNet synthesizes optical tactile images conditioned on RGB, relative depth, and GTPO-generated text. Experiments on curated SSVTP data demonstrate improved structural similarity over the compared baselines, while a blinded user study shows a preference for GTPO-generated descriptions. Generated tactile inputs also improve performance on an attribute-derived force-coefficient prediction proxy. Together, these results demonstrate the effectiveness of PhysTacGen for optical tactile image synthesis and its utility in the evaluated downstream task.The code will be available at https://github.com/VDIGPKU/PhysTacGen.

Figures & tables

Figure 1: The PhysTacGen Framework. Given a visual observation, our system operates in three sequential stages: (1) Physics-Aware Reasoning: A VLM fine-tuned via GTPO infers latent physical properties. (2) Data Curation & Spatial Prior: The dataset is semantically purified using DINOv2, and input RGB is depth-estimated to form a 4-channel geometric condition. (3) Multi-Modal Synthesis: A physics-driven SDXL ControlNet synthesises the high-fidelity tactile sensor image $I_{t}$ . Here $I_{v}$ is the external visual RGB image, $I_{d}$ the estimated relative-depth prior, and $c_{\mathrm{phys}}$ the GTPO-generated physical description; the optical tactile image is $I_{t}$ .

Figure 2: Qualitative Synthesis Results. Visualisation of generated tactile images across different settings. SDXL (Baseline) fails to ground the contact location. The Mix-Fusion and four-channel variants change both fusion and loss and therefore do not isolate a single cause of their visual differences. The examples illustrate generated contact appearance under the original conditions.

Method and condition	SSIM $\uparrow$	LPIPS $\downarrow$	legacy nFD $\downarrow$
Pix2Pix / RGB	0.106	0.819	0.829
Pix2Pix-Turbo / RGB + generic prompt	0.417	0.692	0.628
VisGel (adapted) / RGB + train reference	0.715	0.629	0.433
SDXL baseline / physical text only	0.190	0.859	0.838
Mix-Fusion / RGB + depth, physical text	0.938	0.617	0.241
Four-channel / RGB + depth, generic text	0.908	0.637	0.349

Table 1: External baselines and original generator variants. All rows use the same retained SSVTP split. The last column is the legacy Fréchet statistic divided by evaluation-set size (legacy nFD), not FID.

Figure 3: Material and Textural Consistency under Spatial Misalignment. Red boxes highlight areas for qualitative texture comparison.The details in the areas outlined in red in each set of images are similar.

Prompt source	SSIM $\uparrow$	LPIPS $\downarrow$	legacy nFD $\downarrow$
7-label classifier	0.919	0.557	0.305
SFT	0.948	0.570	0.255
GTPO	0.964	0.559	0.278

Table 2: Final matched end-to-end prompt-source comparison. Each prompt source trains and tests a separate generator under the same conditions; legacy nFD is not standard FID.

Input modality	Run 1	Run 2	Mean $\pm$ sample SD
RGB only	0.144	0.147	$0.1455\pm 0.0021$
RGB + generated tactile	0.132	0.134	$0.1330\pm 0.0014$
RGB + real tactile (oracle)	0.130	0.132	$0.1310\pm 0.0014$

Table 3: Test MSE for the heuristic force-coefficient prediction proxy. Sample SD describes variation across from two random seeds; the last column reports their mean and sample standard deviation.

Method	SES $\uparrow$	BLEU $\uparrow$	ROUGE $\uparrow$	PPL $\downarrow$	Judge $\uparrow$	Human $\uparrow$
7-label classifier	0.732	29.3	52.7	12.9	2.94	16%
SFT Qwen	0.461	18.4	32.7	14.3	3.42	27%
GTPO Qwen	0.586	26.8	44.5	10.8	4.65	57%

Table 4: Description evaluation on 452 pairs. The human column is a blinded forced choice among classifier, SFT, and GTPO descriptions by 22 participants, with randomized presentation and one choice per image.

Explore similar work

May 26, 2025cs.CV

ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation

Vision-based tactile sensing is widely used in perception, reconstruction, and robotic manipulation, yet collecting large-scale tactile data remains costly due to diverse sensor-object interactions and inconsistencies across sensor instances. Existing approaches to scaling tactile data---simulation and free-form tactile generation---often yield unrealistically rendered signals with poor transfer to highly dynamic real-world tasks. We propose \name, a two-stage controllable tactile image generation framework that generates realistic tactile images conditioned on a single reference tactile image, contact force, and contact pose. By grounding generation in these important physical priors, \name synthesizes realistic samples across different sensors while effectively capturing task-relevant variations. Across a series of downstream tasks and real-world experiments, such as object insertion, imitation learning, and object weighting, the augmented datasets using our approach consistently improve performance and demonstrate practical utility in dynamic real-world settings. Project page: https://dongyuluo.github.io/controltac.

Dongyu Luo, Kelin Yu, Amir-Hossein Shahidzadeh +3

The University of Hong Kong · University of Maryland · University of Maryland, College Park

Tactile Imitation Learning

Ask PDF

Jun 28, 2026cs.RO

TacGen: Touch Is a Necessary Dimension of Physical-World Representation -- Addressing Tactile Data Scarcity with Scalable Vision-to-Touch Alignment and Generation

Touch resolves the physical-property ambiguity left by vision: exploratory contact recovers shape, texture, compliance, and material, and visuo-haptic object representations converge in ventral visual cortex. We ask whether representation learning can reproduce this grounding. TacGen mitigates the tactile-data scarcity bottleneck by combining pre-specified V+T contrastive alignment with a latent-space residual-MLP V->T generator that synthesizes tactile latents from RGB for tactile-data scaling. With matched DINOv2 backbones, splits, and probes, V+T improves matched V-only on mass (Delta R^2=+0.570), density (Delta acc=+0.067), hardness (+0.117), and uncertainty-banded force labels (Delta R^2=+0.281); all CIs exclude zero. The same representation lifts matched-capacity TACTO manipulation 0.246->0.979 while V-only capacity scaling accounts for only 4.5% of the gap, preserving 95.5%. The generator reaches cross-seed +0.589, with real tactile +0.585 inside the seed interval; the architecture comparison shows a 13pp downstream gap between reconstruction quality and representation utility. Across five-seed SSVTP/TVL reproductions, YCB-Sight transfer, three-backbone checks, permutation/random-feature controls, hash-verified manifests, and measured-force validation checks, the evidence supports the claim that touch supplies a necessary physical evidence channel for representations of contact-dependent properties.

Wanghao Ye, Aarosh Das, Sihan Chen +19

University of Maryland, College Park · Carnegie Mellon University · University of Alabama +2

Tactile Tac Residual Learning Human Visual Cortex

Ask PDF

Jun 30, 2026cs.RO

UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation

Unified multimodal models (UMMs) have shown great promise in integrating understanding and generation across diverse modalities. However, existing research rarely extends this paradigm to the tactile domain, where both object-level semantics and sensor-level configurations jointly determine the meaning of touch. To address this gap, we propose UniTac, the first UMM designed for tactile understanding and generation. UniTac models the tactile process as a transition from non-contact to contact, capturing the physical interaction between sensors and objects through a dual-level representation that encodes both sensor and object attributes. For tactile understanding, UniTac introduces two tasks, object property description and sensor identification, to enhance reasoning over physical and cross-sensor information. For tactile generation, we design a two-stage training paradigm consisting of reconstruction and alignment, together with a sensor-prior-based sampling strategy that simulates realistic tactile contact. Trained on large-scale multi-sensor datasets, UniTac achieves state-of-the-art performance in tactile understanding and generates realistic tactile signals across sensors.

Jiahang Tu, Fengyu Yang, Chenyang Ma +8

Zhejiang University · Yale University · UNIX AI +3

Tactile Multimodal Model Multi-Modal Sensing

Ask PDF