Authors: Anirvan Dutta, Simone Tasciotti, Claudia Cusseddu, Ang Li, Panayiota Poirazi, Julijana Gjorgjieva, Etienne Burdet, Patrick van der Smagt, Mohsen Kaboli
Organizations: BMW Group AG, München, Germany. · Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, UK. · Department of Biology, University of Crete, Heraklion, Greece. · Institute of Molecular Biology and Biotechnology, Foundation for Research and Technology-Hellas, Heraklion, Greece. · School of Medicine and Health, Institute for Neuroscience, Technical University of Munich, Munich, Germany. · School of Life Sciences, Technical University of Munich, Freising, Germany. · School of Computation, Information and Technology, Technical University of Munich, Germany. · Faculty of Informatics, Eötvös Lor´and University, Budapest, Hungary. · Foundation Robotics Labs, München, Germany. · Electrical Engineering, Eindhoven University of Technology, Netherlands.
Estimating physical properties is critical for safe and efficient autonomous robotic manipulation, particularly during contact-rich interactions. In such settings, vision and tactile sensing provide complementary information about object geometry, pose, inertia, stiffness, and contact dynamics, such as stick-slip behavior. However, these properties are only indirectly observable and cannot always be modeled precisely (e.g., deformation in non-rigid objects coupled with nonlinear contact friction), making the estimation problem inherently complex and requiring sustained exploitation of visuo-tactile sensory information during action. Existing visuo-tactile perception frameworks have primarily emphasized forceful sensor fusion or static cross-modal alignment, with limited consideration of how uncertainty and beliefs about object properties evolve over time. Inspired by human multi-sensory perception and active inference, we propose the Cross-Modal Latent Filter (CMLF) to learn a structured, causal latent state-space of physical object properties. CMLF supports bidirectional transfer of cross-modal priors between vision and touch and integrates sensory evidence through a Bayesian inference process that evolves over time. Real-world robotic experiments demonstrate that CMLF improves the efficiency and robustness of latent physical properties estimation under uncertainty compared to baseline approaches. Beyond performance gains, the model exhibits perceptual coupling phenomena analogous to those observed in humans, including susceptibility to cross-modal illusions and similar trajectories in learning cross-sensory associations. Together, these results constitutes a significant step toward generalizable, robust and physically consistent cross-modal integration for robotic multi-sensory perception.
Figures & tables
Figure 1: Concept of the proposed Cross-Modal Latent Filter (CMLF) perception framework. Inspired by human multi-sensory processing, the model uses extrinsic visual cues to form priors over intrinsic object properties, and vice versa, in a fully unsupervised manner from raw visual and tactile data. Bayesian integration (BI) underpins this cross-modal inference, improving the robustness and efficiency of object property estimation by exploiting statistical regularities across modalities.
Figure 2: Robotic experimental setup and data collection pipeline. a) The figure illustrates the main stages used to construct the visuo-tactile interaction dataset. The process begins with initial estimation of the object shape and pose, which is used to autonomously perform prehensile robotic interactions. These interactions generate rich, time-varying visual and tactile observations capturing both geometric and contact dynamics. The resulting multi-sensory streams are then processed by the proposed Cross-Modal Latent Filter (CMLF) to infer latent object properties and interaction dynamics over time. b) To emulate everyday non-rigid objects, we designed a set of synthetic objects with configurable material properties, enabling controlled variation in flexibility. This allows us to systematically control intrinsic (e.g. mass, stiffness, surface friction) and extrinsic (e.g. size, shape, visual texture) physical attributes and to study their cross-modal associations.
Figure 3: Cross-modal latent filter architecture with cross-modal connections from vision to tactile ( CM-V2T ) and tactile to vision ( CM-T2V ) modalities. The initial priors are set to qfilt(z0V/T)∼N(0,1) .
Figure 4: Classification and regression to evaluate inference performance. Statistical significance is assessed using paired t-tests with Holm–Bonferroni correction for multiple comparisons, with *** denoting p<0.001 , ** p<0.01 , * p<0.05 , and non-significant comparisons are omitted. (a) Classification accuracy for each latent feature with higher values indicate better performance and error bars denote ±1 standard deviation. (b) Regression performance measured by trajectory NMSE for each physical property, with error bars denote ±1 standard deviation; lower values indicate better estimation. (c) Temporal evolution of NMSE for intrinsic and extrinsic properties, with shaded regions indicating ±0.1 standard deviation across interactions; lower values indicate better estimation. The results show that cross-modal priors from extrinsic properties significantly improve the efficiency of intrinsic object property inference compared to baseline approaches.
Nominal
Noise
Corruption
Parameters
Methods
σ,c =0
σ =0.2
σ =0.4
c =0.15
c =0.35
Shape
Joint
11 ± 1.1
11 ± 1.3
11 ± 1.6
12 ± 0.86
13 ± 0.82
wo-CM
9.7 ± 0.34
11 ± 0.78
12 ± 0.78
11 ± 0.49
11 ± 0.45
w-CM
9.6 ± 0.36
11 ± 0.83
11 ± 0.54
10 ± 0.53
11 ± 0.86
Size
Joint
11 ± 1.3
12 ± 1.5
12 ± 1.9
12 ± 1.3
14 ± 1.2
wo-CM
10 ± 1.1
12 ± 1.3
13 ± 1.7
12 ± 0.88
13 ± 0.62
Table 1: Perturbation study on noise and corruption, with heatmap in cells. Brighter hue signifies higher error and worst performance, values are scaled by 100 to improve clarity of presentation.
Figure 5: CMLF perceptual similarity to human inference. a) Mean trajectory error on the surprise set . Statistical significance is evaluated using paired t-tests with Holm–Bonferroni correction. b) Effect of delayed activation of cross-modality priors on inference of intrinsic properties. c) Temporal evolution of NMSE on the surprise set , with shaded regions indicating ±0.1 standard deviation. d) Representative examples illustrating how latent filtering and cross-modal priors differ between the aligned and surprise sets. In the aligned set, the visual-to-tactile prior ( CM−V2T ) provides informative cues derived from extrinsic properties, enabling faster convergence of latent estimates. With the surprise set , the prior trained on aligned set’s cross-modal association becomes misleading; however Bayesian integration gradually corrects the estimate toward the ground-truth.
Figure 6: Experimental setup of the configurable non-rigid objects.
Figure 7: Graphical visualization of the multi-view shape perception and leveraging the estimated shape for segmentation of visual observations.
Figure 8: Visualization of the different noise and corruption levels used to evaluate the proposed approach. σ=0.2,0.4 denotes the standard deviation of zero mean Gaussian white noise. c=0.15,0.35 denotes the fraction of observations along the trajectory that are replaced with zero values or missing data.
Figure S1: Probabilistic Graphical Model of the Latent Filter
Figure S2: Qualitative analysis of latent space
Figure S3: Temporal evolution of the NMSE error of pose estimation, with the shaded region depicting ±0.1 standard deviation, representing the variability of the mean error.
Tactile sensors provide critical information for contact-rich manipulation, yet tactile representations and policies remain tightly coupled to each specific sensor, limiting transferability across robots and hardware platforms. We propose TactX, a framework for learning a transferable tactile representation across sensors spanning three fundamentally different transduction modalities: resistive, magnetic, and vision-based. TactX maps heterogeneous tactile observations into a shared latent space through modality-specific encoders trained on paired contact data. Such paired interactions provide a natural alignment signal across modalities, and the encoders are jointly trained across all sensor pairs, inducing a consistent latent space for all sensor types. Our experiments show that TactX aligns tactile representations across sensors while preserving object-level contact information, as evidenced by sensor-identity prediction and object classification in the learned latent space. We evaluate TactX on four contact-rich manipulation tasks: pick-and-place, plug insertion, board wiping, and object reorientation, and show that policies trained with one sensor transfer zero-shot to physically distinct sensors through the shared latent. This improves the average success rate from 27.5% for vision-only policy to 45.9%, providing a step toward sensor-agnostic tactile manipulation.
Junsung Park, Sachin Bhadang, Carmelo Sferrazza +2
UC San Diego · Seoul National University · Amazon FAR
Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation. These sensors achieve high spatial resolution and multi-dimensional force sensing by utilizing an internal camera to monitor the deformation of their elastic gel surface, thereby indirectly inferring tactile cues. Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge. Existing methods typically use either raw images or cumulative motion fields to represent tactile cues. However, both are prone to perception ambiguity. Raw tactile images mainly capture appearance changes, while cumulative motion fields only reflect the aggregate gel deformation. Consequently, distinct fine-grained contact states can exhibit highly similar patterns, making it difficult to explicitly distinguish subtle contact variations. To address this issue, we explore the dynamic priors of tactile motion and discover that the correlation between transient and cumulative motion can explicitly distinguish fine-grained contact states. Based on this insight, we propose a motion-aware tactile representation to facilitate contact-rich manipulation. Beyond tactile representation, effective fusion of tactile and visual modalities is also critical. Most existing fusion methods either directly concatenate features from each modality or train modality-specific networks separately and fuse their outputs. However, these strategies struggle to simultaneously model cross-modal interactions and preserve modality-specific characteristics. In this work, we take advantage of the Mixture-of-Transformers architecture and propose a unified modality-aware visuo-tactile policy that captures cross-modal complementarity while maintaining modality-specific properties.
Shengqi Xu, Guojin Zhong, Yang Liu +7
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · NeoteAI +1
We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most methods focus on observations at the current time step and overlook the temporal evolution of contact. VT-MUSE addresses both limitations through a two-stage representation learning framework. In Stage I, modality specific encoders are jointly adapted via cross-modal temporal alignment and masked-view consistency. In Stage II, a conditional variational latent model processes masked visual sequences together with full tactile histories. Auxiliary decoders reconstruct the masked recent visual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves substantial improvements in real-world experiments.