Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying tactile and thermal interaction. Existing 3D datasets and reconstruction methods primarily represent object-scale geometry and visual appearance, overlooking microscale surface structure for high-fidelity haptic rendering and transient temperature dynamics for temperature-aware interaction. We present TouchTherm, a framework for constructing simulation-ready visuo-tactile-thermal object assets from real-world objects. For visual and tactile reconstruction, we combine structured-light scanning with multiview normal maps obtained from photometric stereo. The normal maps are registered to the scanned geometry and transformed into tangent space to recover local micro-height fields for optical tactile rendering, while the coarse mesh handles collision detection. For thermal reconstruction, we capture synchronized multiview infrared videos of natural cooling following controlled heating and reconstruct a physics-regularized dynamic thermal field. Experiments on 20 objects show that the reconstructed micro-height fields preserve dominant surface structures and recover higher-frequency details beyond the coarse geometry, while the thermal fields achieve held-out surface-temperature MAEs of 0.465 degrees C and 0.592 degrees C at 30 s and 45 s, respectively. The resulting tactile assets support synthetic-to-real object recognition from tactile observations, while a glove-based VR system demonstrates spatially and temporally varying thermal feedback. These results highlight the potential of TouchTherm for multimodal sensory simulation and temperature-aware virtual interaction.
Figures & tables
Fig. 1: Overview of TouchTherm . From real-object observations, our pipeline constructs simulation-ready assets combining visual geometry, contact-aligned tactile microgeometry, and observation-driven dynamic thermal fields for optical tactile rendering and temperature-field simulation.
Fig. 2: Overview of the TouchTherm reconstruction pipeline. (a) Structured-light scanning, multiview photometric stereo, and thermal imaging acquire geometry, normals, and infrared videos. (b) Coarse geometry is decoupled from registered microgeometry, reconstructed into contact-conditioned height fields for tactile rendering. (c) Infrared observations support physics-regularized dynamic thermal reconstruction and parameter identification for rollout. (d) The asset comprises a coarse collision mesh, normal maps, tactile micro-height fields, and a dynamic thermal field.
Fig. 3: Overview of the 20 real objects used in our dataset . The collection spans flat, curved, cylindrical, and free-form objects with diverse materials and surface relief.
Fig. 4: Experimental setup for real-to-simulation tactile evaluation . The real and simulated sensors follow matched approach-and-press procedures.
Fig. 5: Real-to-simulation tactile appearance comparison across 20 objects. Each object shows a real GelSight Mini measurement and the corresponding simulated rendering under an approximately matched contact region and sensor configuration. Similarity metrics are computed after bounded 2D alignment to compensate for residual image-plane mismatch.
Fig. 6: Representative tactile-microgeometry ablation on four objects . Each object is with two contacts (C1–C2). Real images are captured at the corresponding physical contact regions. The evaluation in Table IV-B covers all 20 objects.
Method
G-SSIM ↑
HF-NCC ↑
Coarse Geometry
0.0480±0.0283
0.0116±0.0328
Image-space Height
0.0629±0.0242
0.0081±0.0208
Ours
0.0701±0.0246
0.0911±0.0648
Table 7
Fig. 7: Comparison of reference and reconstructed thermal evolution for four objects at 0 , 30 , 60 , and 90s . Within each object block, the upper row shows the reference field and the lower row shows the reconstructed field.
Fig. 8: Thermal-feedback VR system . (a) VR and thermal-feedback hardware and (b) interaction with a reconstructed dynamic thermal field.
We propose a multimodal, physically grounded approach for metric-scale amodal object reconstruction and pose estimation under severe hand occlusion. Unlike prior occlusion-aware 3D generation methods that rely only on vision, we leverage physical interaction signals: proprioception provides the posed hand geometry, and multi-contact touch constrains where the object surface must lie, reducing ambiguity in occluded regions. We represent object structure as a pose-aware, camera-aligned signed distance field (SDF) and learn a compact latent space with a Structure-VAE. In this latent space, we train a conditional flow-matching diffusion model, pretraining on vision-only images and finetuning on occluded manipulation scenes while conditioning on visible RGB evidence, occluder/visibility masks, the hand latent representation, and tactile information. Crucially, we incorporate physics-based objectives and differentiable decoder-guidance during finetuning and inference to reduce hand--object interpenetration and to align the reconstructed surface with contact observations. Because our method produces a metric, physically consistent structure estimate, it integrates naturally into existing two-stage reconstruction pipelines, where a downstream module refines geometry and predicts appearance. Experiments in simulation show that adding proprioception and touch substantially improves completion under occlusion and yields physically plausible reconstructions at correct real-world scale compared to vision-only baselines; we further validate transfer by deploying the model on a real humanoid robot with an end-effector different from those used during training.
Gabriele Mario Caddeo, Pasquale Marra, Lorenzo Natale
Human tactile perception of materials relies on complex multisensory touch cues, yet the relationship between low-level tactile signals and perceptual representations remains poorly understood. This knowledge gap hinders the integration of touch in digital environments and the development of robots capable of human-like tactile perception. Here, we present an interpretable computational framework for modeling human material perception and recognition using multisensory touch data. Our framework comprises three interconnected models: Model 1 maps finger-surface interaction features to psychophysical sensory attributes, Model 2 classifies materials based on these perceptual representations, and Model 3 directly classifies materials from tactile features. The results showed that combining information from pressing, static contact, and sliding interactions improves prediction accuracy, and that thermal cues are particularly informative for both perceptual modeling and material classification. These findings highlight the importance of thermal and compliance cues, which remain underrepresented in current robotic fingers and haptic displays. Incorporating such cues may enhance artificial systems' ability to approximate human material perception and guide the design of more perceptually grounded haptic interfaces.
Li Zou, Yasemin Vardar
Delft University of Technology (TU Delft), Department of Cognitive Robotics, Delft, 2628 CD, The Netherlands
Large-scale datasets are essential for training generalist robot control policies. Collecting real-world tactile data is costly and time-consuming, motivating the use of tactile simulations. However, current tactile simulators capture only overall contact geometry and miss fine details like texture. This results in a significant domain shift between simulated and real tactile data. To address this gap, we introduce Norm2Tex, a plug-in method that augments simulations of vision-based tactile sensors with high-frequency surface details from normal map textures. By modifying the target object's depth map before a tactile simulator's rendering pipeline, Norm2Tex seamlessly integrates into different tactile simulators. We also evaluate sim-to-real transfer using material classification and a reinforcement learning task. Our results show that Norm2Tex preserves material-dependent tactile information across domains, improving texture recognition and producing material-dependent control behavior in the real world.