MuxGel: Simultaneous Dual-Modal Visuo-Tactile Sensing via Spatially Multiplexing and Deep Reconstruction

Zhixian Hu; Zhengtong Xu; Sheeraz Athar; Juan Wachs; Yu She

cs.ROMar 10, 2026

MuxGel: Simultaneous Dual-Modal Visuo-Tactile Sensing via Spatially Multiplexing and Deep Reconstruction

Authors: Zhixian Hu, Zhengtong Xu, Sheeraz Athar, Juan Wachs, Yu She

Organizations: Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA.

Abstract

High-fidelity visuo-tactile sensing is important for precise robotic manipulation, yet most vision-based tactile sensors rely on opaque coatings that enable tactile sensing but block direct visual observation. We propose MuxGel, a spatially multiplexed sensor that captures both external visual information and contact-induced tactile signals through a single camera. By using a checkerboard coating pattern, MuxGel interleaves tactile-sensitive regions with transparent windows for external vision. This design maintains standard form factors, allowing for plug-and-play integration into GelSight-style sensors by simply replacing the gel pad. To recover dense visual and tactile signals from the multiplexed inputs, we develop a U-Net-based reconstruction framework trained with a sim-to-real pipeline. Experiments on unseen objects demonstrate the framework's generalization and accuracy. We further demonstrate MuxGel in grasping tasks, where visual feedback supports alignment and tactile feedback supports contact interaction. Results show that MuxGel enables single-camera dual-modal sensing within a GelSight-style implementation, providing local visual feedback and reconstructed tactile feedback with potential extension to other optical tactile sensors. Project webpage: https://zhixianhu.github.io/muxgel/.

Figures & tables

Fig. 1: Grasping a chip with MuxGel: (a) A Robotiq gripper grasping a chip using MuxGel for simultaneous visuo-tactile sensing via spatial multiplexing. (b) MuxGel with the 4 $\times$ 4 checkerboard configuration integrated into a GelSight Mini by replacing only the gel pad, without optical or mechanical redesign. (c) Raw multiplexed sensor output. (d) Reconstructed tactile image. (e) Reconstructed visual image.

Fig. 2: MuxGel configurations and raw observations. (a) A 4 $\times$ 4 sensor integrated with a Robotiq gripper during a grasp. (b)-(f) Hardware patches (left) and captured images (right) for pure vision, pure tactile, and multiplexed (2 $\times$ 2, 4 $\times$ 4, 8 $\times$ 8) configurations.

Fig. 3: Large-scale physics-based simulation pipeline for visual-tactile data generation. Bg: Background; Obj: Object; Tac: Tactile; Ref: Reference.

Fig. 4: Overview of the dual-stream MuxNet architecture. A shared ResNet-34 encoder processes either a 3-channel fused image or a 6-channel tensor formed by concatenating the multiplexed image with a non-contact reference image. Two task-specific decoders reconstruct the visual and tactile modalities, respectively. The tactile branch supports absolute prediction (Option A) and residual prediction (Option B), where the predicted contact residual is added to the non-contact tactile image. Asterisks (

*

) denote activation functions specific to real-data fine-tuning. Chns: channels. BN: BatchNorm. Conv: Convolution.

Fig. 5: Real-world data collection and dataset overview. (a) Automated 3-axis linear motion platform, where the x,y-axes align with the sensor contact plane for precise spatial sampling. (b) Indenter diversity. Top: five distinct geometries (sphere, edge, square, hollow hexagon, solid hexagon). Bottom: one geometry (hollow hexagon) in five colors (white, black, red, green, blue) to improve visual robustness. Black scale bars correspond to 1 cm. (c) Representative data samples. Top: a white sphere indenter at 1.5 mm depth across five gel configurations. Bottom: a hollow hexagon indenter in five colors captured with the 4 $\times$ 4 patterned gel pad at 1.5 mm depth.

Fig. 6: Visuo-tactile reconstruction of real unseen objects using the DI-ResT after real-world fine-tuning. Row 1: Nine objects (screw nut, almond, stub, LEGO block, stone, plastic golf ball, potato, plastic strawberry, avocado) used to test model generalization. Black scale bars correspond to 1 cm. Row 2: Raw multiplexed inputs from the 4 $\times$ 4 checkerboard configuration. Row 3: Reconstructed vision outputs. Row 4: Reconstructed tactile outputs.

Fig. 7: Qualitative reconstruction results on the almond object. Top: vision; bottom: tactile. Left to right: Ground Truth; zero-shot simulation (Sim) models (SI, DI-AbsT, DI-ResT); and real-data fine-tuned models (SI, DI-AbsT, DI-ResT).

Training Stage	Architecture	Tactile Reconstruction				Vision Reconstruction
Training Stage	Architecture	RMSE $\downarrow$	(1-SSIM) $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	RMSE $\downarrow$	(1-SSIM) $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$
Simulation (Zero-shot)	SI	0.0929	0.1667	0.3749	20.6384	0.2198	0.3651	0.3900	13.1892
	DI-AbsT	0.1260	0.1697	0.2025	18.0412	0.1294	0.3182	0.3119	17.7973
	DI-ResT	0.0830	0.1227	0.1094	21.6298	0.1553	0.3319	0.3398	16.2726
Real (Fine-tuned)	SI	0.0332	0.0956	0.0977	29.5972	0.1030	0.2822	0.3557	19.7613
	DI-AbsT	0.0770	0.0877	0.1082	24.6348	0.1118	0.2799	0.3339	19.1292

TABLE I: Quantitative evaluation of the models on real unseen objects.

Configuration	Tactile Reconstruction				Vision Reconstruction
Configuration	RMSE $\downarrow$	(1-SSIM) $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	RMSE $\downarrow$	(1-SSIM) $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$
$2\times 2$	0.0301	0.0879	0.0484	30.6372	0.1185	0.3132	0.3457	18.5543
$4\times 4$	0.0285	0.0869	0.0481	31.2080	0.1021	0.3053	0.3136	19.8327
$8\times 8$	0.0290	0.0891	0.0513	31.0296	0.0935	0.3015	0.3059	20.5986

TABLE II: Quantitative evaluation across MuxGel configurations using the real-data fine-tuned DI-ResT model tested on real unseen objects.

Fig. 8: Visuo-tactile servoing grasping experiment. (a) Experimental setup featuring the proposed sensor integrated with a Robotiq gripper. (b)-(j) Successful grasp executions across a diverse set of unseen objects. For each object, the left panel depicts the stable physical grasp, while the right panel displays the real-time system interface: capturing the raw sensor input, the reconstructed vision output with contour-center tracking for alignment, and the reconstructed GelSight-style tactile image upon contact. The tested objects include: (b) cherry tomato, (c) plastic strawberry, (d) stone, (e) hex key, (f) potato, (g) red potato, (h) avocado, (i) apple, and (j) pickleball.

Explore similar work

FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation
Jul 22, 2026Zinan Li, Yiyang Ling, Yuming Gu +8Contact-Rich Robotic Manipulation Visuo-Tactile Representation Learning
Viterbi School of Engineering, University of Southern California · Columbia University · Starpilot
Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation
Jun 4, 2026Rickmer Krohn, Erik Helmut, Niklas Funk +3Tactile Sensing Contact-Rich Robotic Manipulation
Interactive Robot Perception & Learning, TU Darmstadt · Hessian AI · Robotics Institute Germany +1
VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
Aug 21, 2026Congsheng Xu, Qiaochu Yang, Fangyuan Shi +9Visuo-Tactile Representation Learning Visuomotor Policy Learning
Shanghai Jiao Tong University · Xense Robotics

Jul 22, 2026J/K move · Enter open · S save

FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation

Zinan Li, Yiyang Ling, Yuming Gu, Binghao Huang+7

Viterbi School of Engineering, University of Southern California · Columbia University · Starpilot

Contact-Rich Robotic Manipulation Visuo-Tactile Representation Learning Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

The sense of touch is central to manipulation, especially when vision is occluded or ambiguous. Although combining vision and touch improves manipulation, learning robust visuo-tactile policies requires substantial tactile data. Such data remains scarcer than visual data, because tactile sensors are fragile, specialized, and hard to standardize. To address this, we present Feature-Extracted Latent Tactile (FELT), a learning-based framework that synthesizes per-finger pressure tactile images from RGB observations, reducing the need for tactile-equipped data collection. FELT uses a large frozen visual encoder and a lightweight query decoder to predict tactile signals in a single feed-forward pass. To respect the physical topology of dual-finger tactile sensors, FELT decodes the left and right tactile sensor panels through separate branches, capturing the asymmetric contact patterns during interactions such as wiping, insertion, and in-hand rotation. At inference time, FELT only requires RGB data, allowing us to augment existing vision-only data with tactile observations, either as generated tactile images or as latent tactile features. Experiments on four contact-rich manipulation tasks demonstrate that both generated tactile images and latent tactile features improve policy success over vision-only baselines, with latent feature requiring no real tactile sensor during policy training or deployment. Supplementary material is available on our anonymous website: https://felt-tactile.github.io/.

Abstract

Explore similar work

FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation

Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation

VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation