GelSlim 5.0: Unifying Vision and Touch with a Single Fingertip Camera

Xiangdong Wang; I-Hsuan Huang; Xiaohao Xu; Xili Yi; Nima Fazeli; Xiaonan Huang

cs.ROOct 22, 2026

GelSlim 5.0: Unifying Vision and Touch with a Single Fingertip Camera

Authors: Xiangdong Wang, I-Hsuan Huang, Xiaohao Xu, Xili Yi, Nima Fazeli, Xiaonan Huang

Organizations: University of Michigan, Ann Arbor, MI, USA

Abstract

Vision and touch provide complementary cues for object pose and local contact geometry, but external cameras are often occluded at the contact site. We present GelSlim 5.0, a fingertip sensor that acquires visual and tactile observations with one fixed camera and a continuous UV-responsive contact layer. By switching illumination, the camera sequentially acquires through-membrane visual images and contact-sensitive tactile images from a shared viewpoint, avoiding inter-camera extrinsic calibration. To combine these observations, we introduce VisTacFusion, a multimodal transformer with separate prediction paths for planar object pose and contact geometry. Asymmetric bottleneck fusion incorporates visual context while preserving tactile spatial features for depth and surface-normal estimation. Modality dropout enables inference with paired inputs or either modality alone. On real validation data from 12 known objects, paired inputs reduce depth MSE, angular error, and translation $L_1$ by 61.2%, 72.3%, and 61.3%, respectively, relative to tactile-only inference with the same model. Relative to visual-only inference, the corresponding reductions are 93.7%, 6.1%, and 8.6%.

Figures & tables

Fig. 1: GelSlim 5.0 enables visual and tactile feedback with a single fingertip camera. By switching illumination, the sensor operates in proximity mode with UV off, allowing the camera to see through the membrane, and in contact-sensitive tactile mode with UV on. Both modes share the same viewpoint and require no inter-camera extrinsic calibration. VisTacFusion combines their complementary signals to estimate planar object rotation and translation, as well as contact depth and surface normals. The observations are acquired sequentially under approximately stationary contact.

Sensor construction and physical size				Mode selection		Observation and reconstruction
Sensor	Cameras	Contact interface	Size [mm] H: head; I: interface	Selected element	Control	Tactile cue	Contact representation	Acquisition
Persistent transparency or parallel spectral channels
ViTacTip [ 8 , 9 ]	1 shared	Clear skin; marker-tipped pins	H: NR I: $2500\,\mathrm{mm}^{2\,\mathrm{S}}$	None	None	Pin / marker motion	Marker deformation	Same frame
M 3 Tac [ 10 ]	3 bands a	Spectrally selective membrane	H: $\varnothing 65\!\times\!80^{\mathrm{R}}$ I: film $t=0.2^{\mathrm{R}}$ ; area NR	Spectral channels	Filters + LEDs	Photometric shading	Surface reconstruction	Parallel bands
Fixed interface with illumination-dependent observations
STS [ 3 ]	1 shared	Semitransparent reflective skin	H: NR I: $150\!\times\!150^{\mathrm{R}}$ pad ( $t=5$ )	Lighting ratio	LEDs	Photometric shading	Contact images	Alternating / mixed

TABLE I: Representative sensor designs and physical sizes for scene and contact perception. Rows summarize cited implementations, not a common accuracy benchmark.

Fig. 2: GelSlim 5.0 hardware and observation states. (a) Exploded assembly with the UV-responsive film at the contact interface. (b) Illumination-controlled visual and tactile acquisition. (c) Prototype in (i) tactile mode and (ii) proximity mode; scale marker: 30 mm.

Fig. 3: VisTacFusion: task-specific estimation of object pose and contact geometry. Frozen encoders provide tactile features and visual memory $M$ . The pose path uses a 16-token visual bottleneck to condition tactile-initialized queries over four layers; the final query state $Q_{4}$ predicts planar pose. The geometry path retains tactile maps from blocks 3, 6, 9, and 12, applies gated conditioning from visual-context state $B_{4}$ , and decodes contact depth and surface normals with DPT. Calibration inputs, embeddings, and pre-normalized residual connections are omitted. Geometry thumbnails illustrate estimation targets; missing-input routing is defined in Sec. IV-E .

Fig. 4: Common-view visual–tactile observations. UV-on tactile images (top) and UV-off proximity images (bottom) are acquired sequentially from the same fixed camera under quasi-static contact.

(a) Vis only
	Geometry		Pose
Model	Depth MSE $\downarrow$	Normal MSE $\downarrow$	Rot. $L_{1}$ $\downarrow$	Angle $({}^{\circ})$ $\downarrow$	Trans. $L_{1}$ $\downarrow$
MAE [ 19 ]	0.0756	0.0403	0.0135	0.678	0.0060
DINOv3 [ 21 ]	0.0968	0.0358	0.0256	1.260	0.0110
CLIP [ 23 ]	0.1501	0.0380	0.0554	2.554	0.0294
SigLIP [ 24 ]	0.1449	0.0392	0.0525	2.526	0.0277

TABLE II: Contact geometry and object pose on real validation data. Bold: lowest error in the paired-input configuration; ties at displayed precision.

Fig. 5: Encoder and training ablations with paired input. (a,b) Depth MSE and angular error for four tactile encoders (rows) and four visual encoders (columns); outlines mark minima. (c) Visual-conditioning sweep with a fixed Both/Tac/Vis mixture of 55/35/10, digitized from the two-seed mean curve. Blue circles show depth MSE (left axis); red squares show angular error (right axis). (d) Modality mixtures at

p_{\rm inject}=0.75

; labels give Both/Tac/Vis percentages, and “None” denotes paired-only training. The star marks the selected mixture. Depth is in

10^{-3}\,\mathrm{mm}^{2}

and angle in degrees. These sweeps are separate from the main-table run.

		Geometry		Pose
Input	Model	Depth MSE $\downarrow$	Normal MSE $\downarrow$	Rot. $L_{1}$ $\downarrow$	Angle $({}^{\circ})$ $\downarrow$	Trans. $L_{1}$ $\downarrow$
Vis	Sym. MBT [ 31 ]	0.1059	0.0387	0.0168	0.831	0.0069
	VisTacFusion	0.1055	0.0388	0.0138	0.694	0.0058
Tac	Sym. MBT [ 31 ]	0.0167	0.0123	0.0452	2.178	0.0120
	VisTacFusion	0.0170	0.0128	0.0488	2.350	0.0137
Both	Sym. MBT [ 31 ]	0.0062	0.0071	0.0149	0.747	0.0054

TABLE III: Fusion configuration comparison with SITR+MAE and identical heads. Bottleneck tokens: 4 (MBT), 16 (ours). Checkpoints as in Table II ; bold: lower error in the paired-input comparison.

Fig. 6: Real–simulation data mixing with paired input. Completed configurations are compared by (a) depth MSE, (b) mean angular error, and (c) normalized translation

L_{1}

. Stars mark the main-experiment ratio of 1:1.27. Depth uses the depth-selected checkpoint; both pose metrics use the angle-selected checkpoint.

Fig. 7: Depth and normal predictions on real validation samples. Sim inputs are shown for comparison: tactile images are generated from each sample’s reference depth, and visual images are rendered at the corresponding pose. Predictions use the real inputs and one depth-selected checkpoint across all three modes. Depth uses a shared color range within each row, and normals are RGB-encoded.

Tactile Sensing for Robotic Manipulation

Tactile sensing for robotic manipulation uses contact signals to help robots perceive forces, prevent slips, and handle objects reliably.

See all 178 papers

MuxGel: Simultaneous Dual-Modal Visuo-Tactile Sensing via Spatially Multiplexing and Deep Reconstruction
Mar 10, 2026Zhixian Hu, Zhengtong Xu, Sheeraz Athar +2Vision-Based Tactile Sensing Tactile Sensing for Robotic Manipulation
Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA.
VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
Aug 21, 2026Congsheng Xu, Qiaochu Yang, Fangyuan Shi +9Visuo-Tactile Representation Learning Visuomotor Policy Learning
Shanghai Jiao Tong University · Xense Robotics
Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation
Jun 29, 2026Shengqi Xu, Guojin Zhong, Yang Liu +7Robot Policy Learning Contact-Rich Robotic Manipulation
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · NeoteAI +1

Mar 10, 2026J/K move · Enter open · S save

MuxGel: Simultaneous Dual-Modal Visuo-Tactile Sensing via Spatially Multiplexing and Deep Reconstruction

Zhixian Hu, Zhengtong Xu, Sheeraz Athar, Juan Wachs+1

Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA.

Vision-Based Tactile Sensing Tactile Sensing for Robotic Manipulation Robotic Perception Image Reconstruction

Open paper AskPDF arXiv

High-fidelity visuo-tactile sensing is important for precise robotic manipulation, yet most vision-based tactile sensors rely on opaque coatings that enable tactile sensing but block direct visual observation. We propose MuxGel, a spatially multiplexed sensor that captures both external visual information and contact-induced tactile signals through a single camera. By using a checkerboard coating pattern, MuxGel interleaves tactile-sensitive regions with transparent windows for external vision. This design maintains standard form factors, allowing for plug-and-play integration into GelSight-style sensors by simply replacing the gel pad. To recover dense visual and tactile signals from the multiplexed inputs, we develop a U-Net-based reconstruction framework trained with a sim-to-real pipeline. Experiments on unseen objects demonstrate the framework's generalization and accuracy. We further demonstrate MuxGel in grasping tasks, where visual feedback supports alignment and tactile feedback supports contact interaction. Results show that MuxGel enables single-camera dual-modal sensing within a GelSight-style implementation, providing local visual feedback and reconstructed tactile feedback with potential extension to other optical tactile sensors. Project webpage: https://zhixianhu.github.io/muxgel/.

Abstract

Tactile Sensing for Robotic Manipulation

Explore similar work

MuxGel: Simultaneous Dual-Modal Visuo-Tactile Sensing via Spatially Multiplexing and Deep Reconstruction

VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation

Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation