High-fidelity visuo-tactile sensing is important for precise robotic manipulation, yet most vision-based tactile sensors rely on opaque coatings that enable tactile sensing but block direct visual observation. We propose MuxGel, a spatially multiplexed sensor that captures both external visual information and contact-induced tactile signals through a single camera. By using a checkerboard coating pattern, MuxGel interleaves tactile-sensitive regions with transparent windows for external vision. This design maintains standard form factors, allowing for plug-and-play integration into GelSight-style sensors by simply replacing the gel pad. To recover dense visual and tactile signals from the multiplexed inputs, we develop a U-Net-based reconstruction framework trained with a sim-to-real pipeline. Experiments on unseen objects demonstrate the framework's generalization and accuracy. We further demonstrate MuxGel in grasping tasks, where visual feedback supports alignment and tactile feedback supports contact interaction. Results show that MuxGel enables single-camera dual-modal sensing within a GelSight-style implementation, providing local visual feedback and reconstructed tactile feedback with potential extension to other optical tactile sensors. Project webpage: https://zhixianhu.github.io/muxgel/.
Figures & tables
Fig. 1: Grasping a chip with MuxGel: (a) A Robotiq gripper grasping a chip using MuxGel for simultaneous visuo-tactile sensing via spatial multiplexing. (b) MuxGel with the 4 × 4 checkerboard configuration integrated into a GelSight Mini by replacing only the gel pad, without optical or mechanical redesign. (c) Raw multiplexed sensor output. (d) Reconstructed tactile image. (e) Reconstructed visual image.
Fig. 2: MuxGel configurations and raw observations. (a) A 4 × 4 sensor integrated with a Robotiq gripper during a grasp. (b)-(f) Hardware patches (left) and captured images (right) for pure vision, pure tactile, and multiplexed (2 × 2, 4 × 4, 8 × 8) configurations.
Fig. 3: Large-scale physics-based simulation pipeline for visual-tactile data generation. Bg: Background; Obj: Object; Tac: Tactile; Ref: Reference.
Fig. 4: Overview of the dual-stream MuxNet architecture. A shared ResNet-34 encoder processes either a 3-channel fused image or a 6-channel tensor formed by concatenating the multiplexed image with a non-contact reference image. Two task-specific decoders reconstruct the visual and tactile modalities, respectively. The tactile branch supports absolute prediction (Option A) and residual prediction (Option B), where the predicted contact residual is added to the non-contact tactile image. Asterisks ( ∗ ) denote activation functions specific to real-data fine-tuning. Chns: channels. BN: BatchNorm. Conv: Convolution.
Fig. 5: Real-world data collection and dataset overview. (a) Automated 3-axis linear motion platform, where the x,y-axes align with the sensor contact plane for precise spatial sampling. (b) Indenter diversity. Top: five distinct geometries (sphere, edge, square, hollow hexagon, solid hexagon). Bottom: one geometry (hollow hexagon) in five colors (white, black, red, green, blue) to improve visual robustness. Black scale bars correspond to 1 cm. (c) Representative data samples. Top: a white sphere indenter at 1.5 mm depth across five gel configurations. Bottom: a hollow hexagon indenter in five colors captured with the 4 × 4 patterned gel pad at 1.5 mm depth.
Fig. 6: Visuo-tactile reconstruction of real unseen objects using the DI-ResT after real-world fine-tuning. Row 1: Nine objects (screw nut, almond, stub, LEGO block, stone, plastic golf ball, potato, plastic strawberry, avocado) used to test model generalization. Black scale bars correspond to 1 cm. Row 2: Raw multiplexed inputs from the 4 × 4 checkerboard configuration. Row 3: Reconstructed vision outputs. Row 4: Reconstructed tactile outputs.
Fig. 7: Qualitative reconstruction results on the almond object. Top: vision; bottom: tactile. Left to right: Ground Truth; zero-shot simulation (Sim) models (SI, DI-AbsT, DI-ResT); and real-data fine-tuned models (SI, DI-AbsT, DI-ResT).
Training Stage
Architecture
Tactile Reconstruction
Vision Reconstruction
RMSE ↓
(1-SSIM) ↓
LPIPS ↓
PSNR ↑
RMSE ↓
(1-SSIM) ↓
LPIPS ↓
PSNR ↑
Simulation (Zero-shot)
SI
0.0929
0.1667
0.3749
20.6384
0.2198
0.3651
0.3900
13.1892
DI-AbsT
0.1260
0.1697
0.2025
18.0412
0.1294
0.3182
0.3119
17.7973
DI-ResT
0.0830
0.1227
0.1094
21.6298
0.1553
0.3319
0.3398
16.2726
Real (Fine-tuned)
SI
0.0332
0.0956
0.0977
29.5972
0.1030
0.2822
0.3557
19.7613
DI-AbsT
0.0770
0.0877
0.1082
24.6348
0.1118
0.2799
0.3339
19.1292
TABLE I: Quantitative evaluation of the models on real unseen objects.
Configuration
Tactile Reconstruction
Vision Reconstruction
RMSE ↓
(1-SSIM) ↓
LPIPS ↓
PSNR ↑
RMSE ↓
(1-SSIM) ↓
LPIPS ↓
PSNR ↑
2×2
0.0301
0.0879
0.0484
30.6372
0.1185
0.3132
0.3457
18.5543
4×4
0.0285
0.0869
0.0481
31.2080
0.1021
0.3053
0.3136
19.8327
8×8
0.0290
0.0891
0.0513
31.0296
0.0935
0.3015
0.3059
20.5986
TABLE II: Quantitative evaluation across MuxGel configurations using the real-data fine-tuned DI-ResT model tested on real unseen objects.
Fig. 8: Visuo-tactile servoing grasping experiment. (a) Experimental setup featuring the proposed sensor integrated with a Robotiq gripper. (b)-(j) Successful grasp executions across a diverse set of unseen objects. For each object, the left panel depicts the stable physical grasp, while the right panel displays the real-time system interface: capturing the raw sensor input, the reconstructed vision output with contour-center tracking for alignment, and the reconstructed GelSight-style tactile image upon contact. The tested objects include: (b) cherry tomato, (c) plastic strawberry, (d) stone, (e) hex key, (f) potato, (g) red potato, (h) avocado, (i) apple, and (j) pickleball.
The sense of touch is central to manipulation, especially when vision is occluded or ambiguous. Although combining vision and touch improves manipulation, learning robust visuo-tactile policies requires substantial tactile data. Such data remains scarcer than visual data, because tactile sensors are fragile, specialized, and hard to standardize. To address this, we present Feature-Extracted Latent Tactile (FELT), a learning-based framework that synthesizes per-finger pressure tactile images from RGB observations, reducing the need for tactile-equipped data collection. FELT uses a large frozen visual encoder and a lightweight query decoder to predict tactile signals in a single feed-forward pass. To respect the physical topology of dual-finger tactile sensors, FELT decodes the left and right tactile sensor panels through separate branches, capturing the asymmetric contact patterns during interactions such as wiping, insertion, and in-hand rotation. At inference time, FELT only requires RGB data, allowing us to augment existing vision-only data with tactile observations, either as generated tactile images or as latent tactile features. Experiments on four contact-rich manipulation tasks demonstrate that both generated tactile images and latent tactile features improve policy success over vision-only baselines, with latent feature requiring no real tactile sensor during policy training or deployment. Supplementary material is available on our anonymous website: https://felt-tactile.github.io/.
Zinan Li, Yiyang Ling, Yuming Gu +8
Viterbi School of Engineering, University of Southern California · Columbia University · Starpilot
Touch sensing is beneficial for solving a wide variety of manipulation tasks. While there exists a wide range of tactile sensors with different properties, exploiting the fusion of multiple heterogeneous tactile sensors to improve manipulation learning remains underexplored. We present Multi-Resolution Tactile Sensing (MiTaS), a representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks. We propose a novel architecture using modality-specific convolutional stems and transformer-based fusion that effectively fuses information from an RGB camera stream, a vision-based GelSight Mini sensor and a high-frequency event-based Evetac sensor. This multi-sensor representation then conditions a flow-matching policy for solving downstream tasks. Experimental results across five contact-rich manipulation tasks demonstrate the effectiveness of multi-resolution tactile features in imitation learning. MiTaS achieves an average success rate of 80 %, while vision-only (31 %) and visual-tactile (54 %) baselines cannot solve the task reliably. Co-training a visuo-tactile model with multi-tactile data boosts performance by over 10 % in certain tasks, without having access to the Evetac sensor during policy evaluation. A detailed sensor-reading and attention analysis reveals the importance of different sensors throughout task execution, validating our multi-resolution tactile sensing approach. Project Page: http://mitas-touch.github.io.
Rickmer Krohn, Erik Helmut, Niklas Funk +3
Interactive Robot Perception & Learning, TU Darmstadt · Hessian AI · Robotics Institute Germany +1
We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most methods focus on observations at the current time step and overlook the temporal evolution of contact. VT-MUSE addresses both limitations through a two-stage representation learning framework. In Stage I, modality specific encoders are jointly adapted via cross-modal temporal alignment and masked-view consistency. In Stage II, a conditional variational latent model processes masked visual sequences together with full tactile histories. Auxiliary decoders reconstruct the masked recent visual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves substantial improvements in real-world experiments.