Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying tactile and thermal interaction. Existing 3D datasets and reconstruction methods primarily represent object-scale geometry and visual appearance, overlooking microscale surface structure for high-fidelity haptic rendering and transient temperature dynamics for temperature-aware interaction. We present TouchTherm, a framework for constructing simulation-ready visuo-tactile-thermal object assets from real-world objects. For visual and tactile reconstruction, we combine structured-light scanning with multiview normal maps obtained from photometric stereo. The normal maps are registered to the scanned geometry and transformed into tangent space to recover local micro-height fields for optical tactile rendering, while the coarse mesh handles collision detection. For thermal reconstruction, we capture synchronized multiview infrared videos of natural cooling following controlled heating and reconstruct a physics-regularized dynamic thermal field. Experiments on 20 objects show that the reconstructed micro-height fields preserve dominant surface structures and recover higher-frequency details beyond the coarse geometry, while the thermal fields achieve held-out surface-temperature MAEs of 0.465 degrees C and 0.592 degrees C at 30 s and 45 s, respectively. The resulting tactile assets support synthetic-to-real object recognition from tactile observations, while a glove-based VR system demonstrates spatially and temporally varying thermal feedback. These results highlight the potential of TouchTherm for multimodal sensory simulation and temperature-aware virtual interaction.
Figures & tables
Fig. 1: Overview of TouchTherm . From real-object observations, our pipeline constructs simulation-ready assets combining visual geometry, contact-aligned tactile microgeometry, and observation-driven dynamic thermal fields for optical tactile rendering and temperature-field simulation.
Fig. 2: Overview of the TouchTherm reconstruction pipeline. (a) Structured-light scanning, multiview photometric stereo, and thermal imaging acquire geometry, normals, and infrared videos. (b) Coarse geometry is decoupled from registered microgeometry, reconstructed into contact-conditioned height fields for tactile rendering. (c) Infrared observations support physics-regularized dynamic thermal reconstruction and parameter identification for rollout. (d) The asset comprises a coarse collision mesh, normal maps, tactile micro-height fields, and a dynamic thermal field.
Fig. 3: Overview of the 20 real objects used in our dataset . The collection spans flat, curved, cylindrical, and free-form objects with diverse materials and surface relief.
Fig. 4: Experimental setup for real-to-simulation tactile evaluation . The real and simulated sensors follow matched approach-and-press procedures.
Fig. 5: Real-to-simulation tactile appearance comparison across 20 objects. Each object shows a real GelSight Mini measurement and the corresponding simulated rendering under an approximately matched contact region and sensor configuration. Similarity metrics are computed after bounded 2D alignment to compensate for residual image-plane mismatch.
Fig. 6: Representative tactile-microgeometry ablation on four objects . Each object is with two contacts (C1–C2). Real images are captured at the corresponding physical contact regions. The evaluation in Table IV-B covers all 20 objects.
Method
G-SSIM ↑
HF-NCC ↑
Coarse Geometry
0.0480±0.0283
0.0116±0.0328
Image-space Height
0.0629±0.0242
0.0081±0.0208
Ours
0.0701±0.0246
0.0911±0.0648
Table 7
Fig. 7: Comparison of reference and reconstructed thermal evolution for four objects at 0 , 30 , 60 , and 90s . Within each object block, the upper row shows the reference field and the lower row shows the reconstructed field.
Fig. 8: Thermal-feedback VR system . (a) VR and thermal-feedback hardware and (b) interaction with a reconstructed dynamic thermal field.