MultiDiffSense: Diffusion-Based Multi-Modal Visuo-Tactile Image Generation Conditioned on Object Shape and Contact Pose
Authors: Sirine Bhouri, Lan Wei, Jian-Qing Zheng, Dandan Zhang
Organizations: the Department of Bioengineering, Imperial-X Initiative, Imperial College London, London, United Kingdom. · CAMS-Oxford Institute, University of Oxford, Oxford, United Kingdom.
Acquiring aligned visuo-tactile datasets is slow and costly, requiring specialised hardware and large-scale data collection. Synthetic generation is promising, but prior methods are typically single-modality, limiting cross-modal learning. We present MultiDiffSense, a unified diffusion model that synthesises images for multiple vision-based tactile sensors (ViTac, TacTip, ViTacTip) within a single architecture. Our approach uses dual conditioning on CAD-derived, pose-aligned depth maps and structured prompts that encode sensor type and 4-DoF contact pose, enabling controllable, physically consistent multi-modal synthesis. Evaluating on 8 objects (5 seen, 3 novel) and unseen poses, MultiDiffSense outperforms a Pix2Pix cGAN baseline in SSIM by +36.3% (ViTac), +134.6% (ViTacTip), and +64.7% (TacTip). For downstream 3-DoF pose estimation, mixing 50% synthetic with 50% real halves the required real data while maintaining competitive performance. MultiDiffSense alleviates the data-collection bottleneck in tactile sensing and enables scalable, controllable multi-modal dataset generation for robotic applications.
Figures & tables
Fig. 1 : Framework Overview. The model takes a CAD file and textual prompt as inputs. The CAD model is converted into a pose-aligned depth map (control image) fed via zero-convolutions into the ControlNet branch as the geometric condition. The text prompt is encoded with CLIP and injected into the UNet via cross-attention. The decoder then refines the latents based on both conditions to generate an image reflecting the desired object geometry, contact pose, and sensor modality.
Fig. 2 : Control Image Processing Pipeline. The pipeline takes an STL file, target image and a CSV log of end-effector poses (pose annotations) as inputs and consists of four stages: (1) Use STL file to render depth map and preprocess it to extract clean object masks; (2) Align robot coordinates to image pixels via centroid mapping; (3) Scale XY translations using workspace calibration, Incorporate Z-axis depth through geometric scaling and intensity modulation, and Apply yaw rotation using 2D rotation matrices; (4) Centre alignment error is minimised to < 5 pixels ( ≈ 0.6 mm)
Metric
Model
ViTac
ViTacTip
TacTip
SSIM ↑
MultiDiffSense
0.919 ± 0.022
0.877 ± 0.024
0.768 ± 0.058
Pix2Pix cGAN
0.678 ± 0.028
0.362 ± 0.015
0.450 ± 0.012
PSNR ↑
MultiDiffSense
28.27 ± 3.10
25.74 ± 2.06
22.61 ± 2.33
Pix2Pix cGAN
20.57 ± 0.92
17.38 ± 0.242
14.87 ± 0.28
MSE ↓
MultiDiffSense
0.002 ± 0.003
0.003 ± 0.002
0.006 ± 0.004
Pix2Pix cGAN
0.009 ± 0.003
0.018 ± 0.001
0.033 ± 0.002
TABLE I : Seen objects, unseen poses: performance across tactile modalities (ViTac, ViTacTip, TacTip). We compare MultiDiffSense (single unified model) with Pix2Pix cGAN (separate per modality). Metrics are mean±std; ↑/↓ denote higher-/lower-better; best per metric–modality in bold.
Metric
Model
ViTac
ViTacTip
TacTip
SSIM ↑
MultiDiffSense
0.912 ± 0.013
0.835 ± 0.030
0.741 ± 0.066
Pix2Pix cGAN
0.669 ± 0.015
0.356 ± 0.012
0.450 ± 0.012
PSNR ↑
MultiDiffSense
27.314 ± 2.146
23.530 ± 1.608
21.296 ± 2.575
Pix2Pix cGAN
20.165 ± 0.462
17.163 ± 0.217
14.871 ± 0.271
MSE ↓
MultiDiffSense
0.002 ± 0.001
0.005 ± 0.002
0.009 ± 0.006
Pix2Pix cGAN
0.009 ± 0.001
0.019 ± 0.001
0.033 ± 0.002
TABLE II : Quantitative results on unseen objects across three tactile modalities (ViTac, ViTacTip, TacTip). Best per metric–modality in bold.
Fig. 4 : Visualisation of image generation result on unseen objects across three tactile sensor modalities (ViTacTip, ViTac, TacTip). Red dashed boxes highlight regions where the methods differ: MultiDiffSense better preserves contact geometry, marker patterns, and lighting.
Modality
Component
Real Dataset
Mixed Dataset
Synthetic Dataset
MSE ↓
RMSE ↓
MAE ↓
R² ↑
MSE ↓
RMSE ↓
MAE ↓
R² ↑
MSE ↓
RMSE ↓
MAE ↓
R² ↑
TacTip
X
3.607
1.899
1.500
0.610
4.333
2.081
1.636
0.532
6.182
2.486
1.992
0.332
Z
0.066
0.258
0.221
0.789
0.028
0.166
0.129
0.912
0.279
0.528
0.475
0.112
θz
42.527
6.521
5.532
0.982
221.682
14.889
5.700
0.907
602.863
24.553
13.818
0.748
ViTac
X
0.183
0.428
0.309
0.980
0.131
0.361
0.280
0.986
0.903
0.950
0.761
0.902
Z
0.068
0.261
0.213
0.782
0.051
0.226
0.183
0.837
0.594
0.770
0.689
-0.893
TABLE III : Pose estimation performance comparison across sensor modalities and training dataset types. Mixed datasets achieve performance comparable to or superior to real-only training. Best results for each row (per component) for each metric are shown in bold.
Metric
Seen Objects
Unseen Objects
Control-only
Prompt + Control
Control-only
Prompt + Control
SSIM ↑
0.852 ± 0.001
0.853 ± 0.001
0.820 ± 0.001
0.812 ± 0.001
FID ↓
2.464 ± 0.050
2.021 ± 0.035
3.224 ± 0.274
3.866 ± 0.021
TABLE IV : Ablation 1: Impact of geometric Control (CAD-derived depth) on MultiDiffSense. Evaluated on seen objects–unseen poses and unseen objects . Best results are bolded. Metrics are mean ± std over three runs.
Metric
Seen Objects
Unseen Objects
Short Prompt (1)
Long Prompt (2)
Short Prompt (1)
Long Prompt (2)
SSIM ↑
0.8768
0.8394
0.8349
0.8069
PSNR ↑
25.74
23.91
23.53
22.24
MSE ↓
3.04E-03
4.53E-03
4.81E-03
6.29E-03
LPIPS ↓
0.0593
0.0765
0.0738
0.0893
FID ↓
2.212
2.1993
2.7982
3.5497
TABLE V : Ablation 2: Impact of prompt length on reconstruction quality for seen and unseen objects . Best results per case are bolded.
Fig. 5 : Effect of prompt length on reconstruction quality. Real vs Generated images by the two different model variants under the two testing scenarios (seen object but unseen poses and unseen objects).