FlowTouch: View-Invariant Visuo-Tactile Prediction

Seongjin Bien; Carlo Kneissl; Tobias Jülg; Frank Fundel; Thomas Ressler-Antal; Florian Walter; Björn Ommer; Gitta Kutyniok

cs.ROMar 9, 2026

FlowTouch: View-Invariant Visuo-Tactile Prediction

Authors: Seongjin Bien, Carlo Kneissl, Tobias Jülg, Frank Fundel, Thomas Ressler-Antal, Florian Walter, Björn Ommer, Gitta Kutyniok,

Organizations: Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg, Germany · Ludwig Maximilian University of Munich · TUM School of Computation, Information and Technology, Technical University of Munich, Germany

Abstract

Tactile sensation is essential for contact-rich manipulation tasks. It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control. An inherent limitation of tactile sensors is that readings are available only when an object is touched. This precludes their use during planning and the initial execution phase of a task. Predicting tactile information from visual information can bridge this gap. A common approach is to learn a direct mapping from camera images to the output of vision-based tactile sensors. However, the resulting model will depend strongly on the specific setup and on how well the camera can capture the area where an object is touched. In this work, we introduce FlowTouch, a novel model for view-invariant visuo-tactile prediction. Our key idea is to use an object's local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details. FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation. Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances. We further show that the resulting tactile images can be used for downstream grasp stability prediction. Our code, datasets and videos are available at https://flowtouch.github.io/

Figures & tables

Fig. 1: FlowTouch at a glance: The robot first looks at the object and creates a mesh with scene mesh generation foundation models. Given the particular touch point on the mesh and the static background image of the tactile sensor, FlowTouch then predicts the resulting tactile image.

Fig. 2: The FlowTouch architecture: The blue area shows the image-to-PCN sampling pipeline, while the green area shows the generative model’s structure. Snowflakes indicate components with frozen weights.

Fig. 3: Left: A selection of primitive geometries used for generating the simulation data. Right: Taxim GelSight images rendered from a single contact point (corner of a cube), with local translations and rotations.

Name	Sensor	Training # (# obj.)	Validation # (# obj.)
OFR-G	GelSight	887 (36)	135 (6)
SIM-G	GelSight	200K (135)	113K (60)
YCB-D	DIGIT	2578 (7)	1084 (3)
SIM-D	DIGIT	200K (135)	—
SELF-D	DIGIT	—	547 (13)
GS-G	GelSight	13372 (6*)	3379 (6*)

TABLE I: List of datasets used for evaluation. * indicates within-object split.

Variant	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )
Base	24.28	0.52	0.40
BG-Enc	25.52	0.57	0.34
BG-Scene-Enc	24.91	0.54	0.38
BG-Stack	25.53	0.59	0.35

TABLE II: Ablation study on network design choices evaluated on the GelSight OFR dataset. All metrics are reported on the held-out validation set.

Training Method	YCB-D			OFR-G			SELF-D (unseen)
Training Method	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )
Naive Finetuning	$20.23$	$0.612$	$0.284$	$24.58$	$0.553$	$0.410$	$16.05$	$0.355$	$0.415$
Data Augmentation	$-0.56$	$-0.015$	$+0.013$	$-0.43$	$-0.031$	$+0.009$	$-0.37$	$-0.030$	$+0.004$
Sparsh Perceptual Loss	$-0.20$	$-0.006$	$+0.005$	$+0.17$	$-0.001$	$-0.006$	$+0.06$	$-0.000$	$-0.002$
Optimizer Reset	$-0.03$	$-0.001$	$+0.000$	$+0.28$	$-0.005$	$-0.004$	$+0.07$	$+0.001$	$-0.001$
Domain Conditioning	$\mathbf{+2.18}$	$\mathbf{+0.058}$	$\mathbf{-0.055}$	$\mathbf{+0.90}$	$\mathbf{+0.029}$	$\mathbf{-0.045}$	$\mathbf{+0.78}$	$\mathbf{+0.032}$	$\mathbf{-0.021}$

TABLE III: Quantitative comparison of different sim-to-real training techniques across the pretraining SIM-G, finetuning OFR-G, and unseen SELF-D datasets. Values for experimental methods are shown as absolute differences relative to the Naive Finetuning baseline and are calculated on the validation dataset.

Fig. 4: Tactile predictions of the model ablations listed in Tab. III . Samples are taken from the validation dataset.

Fig. 5: Left: Objects used for data collection. Center: Robot setup. Right: Collected grasp poses aligned to generated mesh, with unique color for each finger pair.

Fig. 6: Meshes used for the ObjectFolder Benchmark grasp stability test.

Name	Pre-training ds.	Fine-tuning ds.	Accuracy (%)
GT	—	—	85.83
A	Full GS-G	—	86.06
B	SIM-{D,G}	Full GS-G	85.17
C	Mixed-DG	10% of GS-G	83.74
D	SIM-{D,G}	OFR-G, YCB-D	81.35
D, no Sparsh	SIM-{D,G}	OFR-G, YCB-D	78.59

TABLE IV: Training dataset ablations for grasp stability task. Values are averaged over 100 trainings.

Fig. 7: Tactile predictions from the grasp stability task on the dataset ablations listed in Tab. IV . Samples are taken from the validation dataset. SELF-D has not been seen during training.

Visual-to-Tactile Prediction

See all 17 papers

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision
Aug 15, 2026Wenkang Zhang, Chengbo Yuan, Zicheng Zhang +2Visual-to-Tactile Prediction Contact Force Estimation
Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University +1
PredTac: Learning Contact-Rich Manipulation with Predicted Touch
Sep 14, 2026Weijia Fan, Daqiang GuoVisual-to-Tactile Prediction Tactile Sensing
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation
Jul 22, 2026Zinan Li, Yiyang Ling, Yuming Gu +8Contact-Rich Robotic Manipulation Visuo-Tactile Representation Learning
Viterbi School of Engineering, University of Southern California · Columbia University · Starpilot

Aug 15, 2026J/K move · Enter open · S save

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision

Wenkang Zhang, Chengbo Yuan, Zicheng Zhang, Zhengxue Cheng+1

Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University+1

Visual-to-Tactile Prediction Contact Force Estimation Robotic Tactile Sensing Contact-Rich Manipulation

Open paper AskPDF arXiv

Touch is fundamental to dexterous manipulation, yet most egocentric human data increasingly used for robot learning lacks tactile information. Directly collecting large-scale tactile data is challenging due to sensor limitations, while human video data is abundant, contact-rich, and easily scalable. This motivates a natural question: can tactile signals be inferred purely from vision? To address this, we introduce EgoTac, a generalizable model that predicts rich tactile information directly from egocentric human videos. EgoTac is trained on a unified corpus of over 5.7M image-tactile pairs, covering both continuous force measurements and binary contacts. By learning from this diverse dataset, EgoTac captures nuanced touch dynamics across varied interactions. Experiments demonstrate strong performance: in-domain prediction achieves an average force error below 0.06N. On out-of-domain contact prediction benchmarks, EgoTac consistently outperforms the state-of-the-art contact estimator. It also captures the rise and fall patterns of real tactile data and enables zero-shot predictions on unconstrained real-world videos. Scaling analyses further reveal that both data diversity and volume improve performance steadily. Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning. Related resources can be found at https://egotac.github.io/ .

Abstract

Visual-to-Tactile Prediction

Explore similar work

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision

PredTac: Learning Contact-Rich Manipulation with Predicted Touch

FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation