Cross-Modal Visuo-Tactile Object Perception

Anirvan Dutta; Simone Tasciotti; Claudia Cusseddu; Ang Li; Panayiota Poirazi; Julijana Gjorgjieva; Etienne Burdet; Patrick van der Smagt

cs.ROApr 2, 2026

Cross-Modal Visuo-Tactile Object Perception

Authors: Anirvan Dutta, Simone Tasciotti, Claudia Cusseddu, Ang Li, Panayiota Poirazi, Julijana Gjorgjieva, Etienne Burdet, Patrick van der Smagt,

Organizations: BMW Group AG, München, Germany. · Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, UK. · Department of Biology, University of Crete, Heraklion, Greece. · Institute of Molecular Biology and Biotechnology, Foundation for Research and Technology-Hellas, Heraklion, Greece. · School of Medicine and Health, Institute for Neuroscience, Technical University of Munich, Munich, Germany. · School of Life Sciences, Technical University of Munich, Freising, Germany. · School of Computation, Information and Technology, Technical University of Munich, Germany. · Faculty of Informatics, Eötvös Lor´and University, Budapest, Hungary. · Foundation Robotics Labs, München, Germany. · Electrical Engineering, Eindhoven University of Technology, Netherlands.

Abstract

Estimating physical properties is critical for safe and efficient autonomous robotic manipulation, particularly during contact-rich interactions. In such settings, vision and tactile sensing provide complementary information about object geometry, pose, inertia, stiffness, and contact dynamics, such as stick-slip behavior. However, these properties are only indirectly observable and cannot always be modeled precisely (e.g., deformation in non-rigid objects coupled with nonlinear contact friction), making the estimation problem inherently complex and requiring sustained exploitation of visuo-tactile sensory information during action. Existing visuo-tactile perception frameworks have primarily emphasized forceful sensor fusion or static cross-modal alignment, with limited consideration of how uncertainty and beliefs about object properties evolve over time. Inspired by human multi-sensory perception and active inference, we propose the Cross-Modal Latent Filter (CMLF) to learn a structured, causal latent state-space of physical object properties. CMLF supports bidirectional transfer of cross-modal priors between vision and touch and integrates sensory evidence through a Bayesian inference process that evolves over time. Real-world robotic experiments demonstrate that CMLF improves the efficiency and robustness of latent physical properties estimation under uncertainty compared to baseline approaches. Beyond performance gains, the model exhibits perceptual coupling phenomena analogous to those observed in humans, including susceptibility to cross-modal illusions and similar trajectories in learning cross-sensory associations. Together, these results constitutes a significant step toward generalizable, robust and physically consistent cross-modal integration for robotic multi-sensory perception.

Figures & tables

Figure 1: Concept of the proposed Cross-Modal Latent Filter (CMLF) perception framework. Inspired by human multi-sensory processing, the model uses extrinsic visual cues to form priors over intrinsic object properties, and vice versa, in a fully unsupervised manner from raw visual and tactile data. Bayesian integration (BI) underpins this cross-modal inference, improving the robustness and efficiency of object property estimation by exploiting statistical regularities across modalities.

Figure 2: Robotic experimental setup and data collection pipeline. a) The figure illustrates the main stages used to construct the visuo-tactile interaction dataset. The process begins with initial estimation of the object shape and pose, which is used to autonomously perform prehensile robotic interactions. These interactions generate rich, time-varying visual and tactile observations capturing both geometric and contact dynamics. The resulting multi-sensory streams are then processed by the proposed Cross-Modal Latent Filter (CMLF) to infer latent object properties and interaction dynamics over time. b) To emulate everyday non-rigid objects, we designed a set of synthetic objects with configurable material properties, enabling controlled variation in flexibility. This allows us to systematically control intrinsic (e.g. mass, stiffness, surface friction) and extrinsic (e.g. size, shape, visual texture) physical attributes and to study their cross-modal associations.

Figure 3: Cross-modal latent filter architecture with cross-modal connections from vision to tactile ( CM-V2T ) and tactile to vision ( CM-T2V ) modalities. The initial priors are set to $q^{filt}(\mathbf{z}_{0}^{V/T})\sim\mathcal{N}(0,1)$ .

Figure 4: Classification and regression to evaluate inference performance. Statistical significance is assessed using paired t-tests with Holm–Bonferroni correction for multiple comparisons, with *** denoting

p<0.001

, **

p<0.01

, *

p<0.05

, and non-significant comparisons are omitted. (a) Classification accuracy for each latent feature with higher values indicate better performance and error bars denote

\pm\,1

standard deviation. (b) Regression performance measured by trajectory NMSE for each physical property, with error bars denote

\pm\,1

standard deviation; lower values indicate better estimation. (c) Temporal evolution of NMSE for intrinsic and extrinsic properties, with shaded regions indicating

\pm 0.1

standard deviation across interactions; lower values indicate better estimation. The results show that cross-modal priors from extrinsic properties significantly improve the efficiency of intrinsic object property inference compared to baseline approaches.

		Nominal	Noise		Corruption
Parameters	Methods	$\sigma,c$ =0	$\sigma$ =0.2	$\sigma$ =0.4	$c$ =0.15	$c$ =0.35
Shape	Joint	11 $\pm$ 1.1	11 $\pm$ 1.3	11 $\pm$ 1.6	12 $\pm$ 0.86	13 $\pm$ 0.82
	wo-CM	9.7 $\pm$ 0.34	11 $\pm$ 0.78	12 $\pm$ 0.78	11 $\pm$ 0.49	11 $\pm$ 0.45
	w-CM	9.6 $\pm$ 0.36	11 $\pm$ 0.83	11 $\pm$ 0.54	10 $\pm$ 0.53	11 $\pm$ 0.86
Size	Joint	11 $\pm$ 1.3	12 $\pm$ 1.5	12 $\pm$ 1.9	12 $\pm$ 1.3	14 $\pm$ 1.2
	wo-CM	10 $\pm$ 1.1	12 $\pm$ 1.3	13 $\pm$ 1.7	12 $\pm$ 0.88	13 $\pm$ 0.62

Table 1: Perturbation study on noise and corruption, with heatmap in cells. Brighter hue signifies higher error and worst performance, values are scaled by 100 to improve clarity of presentation.

Figure 5: CMLF perceptual similarity to human inference. a) Mean trajectory error on the surprise set . Statistical significance is evaluated using paired t-tests with Holm–Bonferroni correction. b) Effect of delayed activation of cross-modality priors on inference of intrinsic properties. c) Temporal evolution of NMSE on the surprise set , with shaded regions indicating

\pm 0.1

standard deviation. d) Representative examples illustrating how latent filtering and cross-modal priors differ between the aligned and surprise sets. In the aligned set, the visual-to-tactile prior (

CM-V2T

) provides informative cues derived from extrinsic properties, enabling faster convergence of latent estimates. With the surprise set , the prior trained on aligned set’s cross-modal association becomes misleading; however Bayesian integration gradually corrects the estimate toward the ground-truth.

Figure 6: Experimental setup of the configurable non-rigid objects.

Figure 7: Graphical visualization of the multi-view shape perception and leveraging the estimated shape for segmentation of visual observations.

Figure 8: Visualization of the different noise and corruption levels used to evaluate the proposed approach. $\sigma=0.2,0.4$ denotes the standard deviation of zero mean Gaussian white noise. $c=0.15,0.35$ denotes the fraction of observations along the trajectory that are replaced with zero values or missing data.

Figure S1: Probabilistic Graphical Model of the Latent Filter

Figure S2: Qualitative analysis of latent space

Figure S3: Temporal evolution of the NMSE error of pose estimation, with the shaded region depicting

\pm 0.1

standard deviation, representing the variability of the mean error.

Visuo-Tactile Representation Learning

Visuo-tactile representation learning aligns vision and touch to infer contact, object properties, and actions for physical interaction.

See all 63 papers

TactX: Learning Shared Tactile Representations Across Diverse Sensors
Jun 30, 2026Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2Cross-Modal Representation Learning Contact-Rich Robotic Manipulation
UC San Diego · Seoul National University · Amazon FAR
Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation
Jun 29, 2026Shengqi Xu, Guojin Zhong, Yang Liu +7Robot Policy Learning Contact-Rich Robotic Manipulation
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · NeoteAI +1
VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
Aug 21, 2026Congsheng Xu, Qiaochu Yang, Fangyuan Shi +9Visuo-Tactile Representation Learning Visuomotor Policy Learning
Shanghai Jiao Tong University · Xense Robotics

Jun 30, 2026J/K move · Enter open · S save

TactX: Learning Shared Tactile Representations Across Diverse Sensors

Junsung Park, Sachin Bhadang, Carmelo Sferrazza, Sha Yi+1

UC San Diego · Seoul National University · Amazon FAR

Cross-Modal Representation Learning Contact-Rich Robotic Manipulation Visuo-Tactile Representation Learning Generalization in Robotic Manipulation Cross-Embodiment Robot Learning Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

Tactile sensors provide critical information for contact-rich manipulation, yet tactile representations and policies remain tightly coupled to each specific sensor, limiting transferability across robots and hardware platforms. We propose TactX, a framework for learning a transferable tactile representation across sensors spanning three fundamentally different transduction modalities: resistive, magnetic, and vision-based. TactX maps heterogeneous tactile observations into a shared latent space through modality-specific encoders trained on paired contact data. Such paired interactions provide a natural alignment signal across modalities, and the encoders are jointly trained across all sensor pairs, inducing a consistent latent space for all sensor types. Our experiments show that TactX aligns tactile representations across sensors while preserving object-level contact information, as evidenced by sensor-identity prediction and object classification in the learned latent space. We evaluate TactX on four contact-rich manipulation tasks: pick-and-place, plug insertion, board wiping, and object reorientation, and show that policies trained with one sensor transfer zero-shot to physically distinct sensors through the shared latent. This improves the average success rate from 27.5% for vision-only policy to 45.9%, providing a step toward sensor-agnostic tactile manipulation.

Abstract

Visuo-Tactile Representation Learning

Explore similar work

TactX: Learning Shared Tactile Representations Across Diverse Sensors

Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation

VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation