Vision and touch provide complementary cues for object pose and local contact geometry, but external cameras are often occluded at the contact site. We present GelSlim 5.0, a fingertip sensor that acquires visual and tactile observations with one fixed camera and a continuous UV-responsive contact layer. By switching illumination, the camera sequentially acquires through-membrane visual images and contact-sensitive tactile images from a shared viewpoint, avoiding inter-camera extrinsic calibration. To combine these observations, we introduce VisTacFusion, a multimodal transformer with separate prediction paths for planar object pose and contact geometry. Asymmetric bottleneck fusion incorporates visual context while preserving tactile spatial features for depth and surface-normal estimation. Modality dropout enables inference with paired inputs or either modality alone. On real validation data from 12 known objects, paired inputs reduce depth MSE, angular error, and translation L1 by 61.2%, 72.3%, and 61.3%, respectively, relative to tactile-only inference with the same model. Relative to visual-only inference, the corresponding reductions are 93.7%, 6.1%, and 8.6%.
Figures & tables
Fig. 1: GelSlim 5.0 enables visual and tactile feedback with a single fingertip camera. By switching illumination, the sensor operates in proximity mode with UV off, allowing the camera to see through the membrane, and in contact-sensitive tactile mode with UV on. Both modes share the same viewpoint and require no inter-camera extrinsic calibration. VisTacFusion combines their complementary signals to estimate planar object rotation and translation, as well as contact depth and surface normals. The observations are acquired sequentially under approximately stationary contact.
Sensor construction and physical size
Mode selection
Observation and reconstruction
Sensor
Cameras
Contact interface
Size [mm] H: head; I: interface
Selected element
Control
Tactile cue
Contact representation
Acquisition
Persistent transparency or parallel spectral channels
ViTacTip [ 8 , 9 ]
1 shared
Clear skin; marker-tipped pins
H: NR I: 2500mm2S
None
None
Pin / marker motion
Marker deformation
Same frame
M 3 Tac [ 10 ]
3 bands a
Spectrally selective membrane
H: ∅65×80R I: film t=0.2R ; area NR
Spectral channels
Filters + LEDs
Photometric shading
Surface reconstruction
Parallel bands
Fixed interface with illumination-dependent observations
STS [ 3 ]
1 shared
Semitransparent reflective skin
H: NR I: 150×150R pad ( t=5 )
Lighting ratio
LEDs
Photometric shading
Contact images
Alternating / mixed
TABLE I: Representative sensor designs and physical sizes for scene and contact perception. Rows summarize cited implementations, not a common accuracy benchmark.
Fig. 2: GelSlim 5.0 hardware and observation states. (a) Exploded assembly with the UV-responsive film at the contact interface. (b) Illumination-controlled visual and tactile acquisition. (c) Prototype in (i) tactile mode and (ii) proximity mode; scale marker: 30 mm.
Fig. 3: VisTacFusion: task-specific estimation of object pose and contact geometry. Frozen encoders provide tactile features and visual memory M . The pose path uses a 16-token visual bottleneck to condition tactile-initialized queries over four layers; the final query state Q4 predicts planar pose. The geometry path retains tactile maps from blocks 3, 6, 9, and 12, applies gated conditioning from visual-context state B4 , and decodes contact depth and surface normals with DPT. Calibration inputs, embeddings, and pre-normalized residual connections are omitted. Geometry thumbnails illustrate estimation targets; missing-input routing is defined in Sec. IV-E .
Fig. 4: Common-view visual–tactile observations. UV-on tactile images (top) and UV-off proximity images (bottom) are acquired sequentially from the same fixed camera under quasi-static contact.
Geometry
Pose
Model
Depth MSE ↓
Normal MSE ↓
Rot. L1↓
Angle (∘)↓
Trans. L1↓
(a) Vis only
MAE [ 19 ]
0.0756
0.0403
0.0135
0.678
0.0060
DINOv3 [ 21 ]
0.0968
0.0358
0.0256
1.260
0.0110
CLIP [ 23 ]
0.1501
0.0380
0.0554
2.554
0.0294
SigLIP [ 24 ]
0.1449
0.0392
0.0525
2.526
0.0277
TABLE II: Contact geometry and object pose on real validation data. Bold: lowest error in the paired-input configuration; ties at displayed precision.
Fig. 5: Encoder and training ablations with paired input. (a,b) Depth MSE and angular error for four tactile encoders (rows) and four visual encoders (columns); outlines mark minima. (c) Visual-conditioning sweep with a fixed Both/Tac/Vis mixture of 55/35/10, digitized from the two-seed mean curve. Blue circles show depth MSE (left axis); red squares show angular error (right axis). (d) Modality mixtures at pinject=0.75 ; labels give Both/Tac/Vis percentages, and “None” denotes paired-only training. The star marks the selected mixture. Depth is in 10−3mm2 and angle in degrees. These sweeps are separate from the main-table run.
Geometry
Pose
Input
Model
Depth MSE ↓
Normal MSE ↓
Rot. L1↓
Angle (∘)↓
Trans. L1↓
Vis
Sym. MBT [ 31 ]
0.1059
0.0387
0.0168
0.831
0.0069
VisTacFusion
0.1055
0.0388
0.0138
0.694
0.0058
Tac
Sym. MBT [ 31 ]
0.0167
0.0123
0.0452
2.178
0.0120
VisTacFusion
0.0170
0.0128
0.0488
2.350
0.0137
Both
Sym. MBT [ 31 ]
0.0062
0.0071
0.0149
0.747
0.0054
TABLE III: Fusion configuration comparison with SITR+MAE and identical heads. Bottleneck tokens: 4 (MBT), 16 (ours). Checkpoints as in Table II ; bold: lower error in the paired-input comparison.
Fig. 6: Real–simulation data mixing with paired input. Completed configurations are compared by (a) depth MSE, (b) mean angular error, and (c) normalized translation L1 . Stars mark the main-experiment ratio of 1:1.27. Depth uses the depth-selected checkpoint; both pose metrics use the angle-selected checkpoint.
Fig. 7: Depth and normal predictions on real validation samples. Sim inputs are shown for comparison: tactile images are generated from each sample’s reference depth, and visual images are rendered at the corresponding pose. Predictions use the real inputs and one depth-selected checkpoint across all three modes. Depth uses a shared color range within each row, and normals are RGB-encoded.
High-fidelity visuo-tactile sensing is important for precise robotic manipulation, yet most vision-based tactile sensors rely on opaque coatings that enable tactile sensing but block direct visual observation. We propose MuxGel, a spatially multiplexed sensor that captures both external visual information and contact-induced tactile signals through a single camera. By using a checkerboard coating pattern, MuxGel interleaves tactile-sensitive regions with transparent windows for external vision. This design maintains standard form factors, allowing for plug-and-play integration into GelSight-style sensors by simply replacing the gel pad. To recover dense visual and tactile signals from the multiplexed inputs, we develop a U-Net-based reconstruction framework trained with a sim-to-real pipeline. Experiments on unseen objects demonstrate the framework's generalization and accuracy. We further demonstrate MuxGel in grasping tasks, where visual feedback supports alignment and tactile feedback supports contact interaction. Results show that MuxGel enables single-camera dual-modal sensing within a GelSight-style implementation, providing local visual feedback and reconstructed tactile feedback with potential extension to other optical tactile sensors. Project webpage: https://zhixianhu.github.io/muxgel/.
Zhixian Hu, Zhengtong Xu, Sheeraz Athar +2
Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA.
We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most methods focus on observations at the current time step and overlook the temporal evolution of contact. VT-MUSE addresses both limitations through a two-stage representation learning framework. In Stage I, modality specific encoders are jointly adapted via cross-modal temporal alignment and masked-view consistency. In Stage II, a conditional variational latent model processes masked visual sequences together with full tactile histories. Auxiliary decoders reconstruct the masked recent visual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves substantial improvements in real-world experiments.
Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation. These sensors achieve high spatial resolution and multi-dimensional force sensing by utilizing an internal camera to monitor the deformation of their elastic gel surface, thereby indirectly inferring tactile cues. Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge. Existing methods typically use either raw images or cumulative motion fields to represent tactile cues. However, both are prone to perception ambiguity. Raw tactile images mainly capture appearance changes, while cumulative motion fields only reflect the aggregate gel deformation. Consequently, distinct fine-grained contact states can exhibit highly similar patterns, making it difficult to explicitly distinguish subtle contact variations. To address this issue, we explore the dynamic priors of tactile motion and discover that the correlation between transient and cumulative motion can explicitly distinguish fine-grained contact states. Based on this insight, we propose a motion-aware tactile representation to facilitate contact-rich manipulation. Beyond tactile representation, effective fusion of tactile and visual modalities is also critical. Most existing fusion methods either directly concatenate features from each modality or train modality-specific networks separately and fuse their outputs. However, these strategies struggle to simultaneously model cross-modal interactions and preserve modality-specific characteristics. In this work, we take advantage of the Mixture-of-Transformers architecture and propose a unified modality-aware visuo-tactile policy that captures cross-modal complementarity while maintaining modality-specific properties.
Shengqi Xu, Guojin Zhong, Yang Liu +7
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · NeoteAI +1