We present GelSphere, a spherical vision-based tactile sensor designed for real-time continuous surface scanning. Unlike traditional vision-based tactile sensors that can only sense locally and are damaged when slid across surfaces, and cylindrical tactile sensors that can only roll along a fixed direction, our design enables omnidirectional rolling on surfaces. We accomplish this through our novel sensing system design, which has steel balls inside the sensor, forming a bearing layer between the gel and the rigid housing that allows rolling motion in all axes. The sensor streams tactile images through Wi-Fi, with online large-surface reconstruction capabilities. We present quantitative results for both reconstruction accuracy and image fusion performance. The results show that our sensor maintains geometric fidelity and high reconstruction accuracy even under multi-directional rolling, enabling uninterrupted surface scanning.
Figures & tables
Fig. 1: Our novel vision-based tactile sensor, GelSphere, achieves omnidirectional continuous scanning of large surfaces through a unique mechanical design. It can be easily human-operated for online surface reconstruction or potentially used in robotic platforms for automation. We use tactile SLAM methods to stitch the contact patches together, forming a continuous surface model.
Fig. 2: Optical and mechanical design of GelSphere. (A) The simulated tactile image is shown alongside a real sensor image under a spherical indentation, presenting a close visual similarities between the optical simulation and the fabricated sensor. The RGB LED ring consists of 18 LEDs (6 of each color), and three color groups are spaced by approximately 120∘ , which provides directional illumination from every direction. (B) Cross-sectional view of the GelSphere sensor. The internal optical module (camera, LED right, and battery) is mounted on a base that is magnetically suspended, which helps maintain a consistent camera orientation while the outer shell rolls. A magnetic switch enables power on/off control without opening the housing. The steel ball bearing layer ensures low-friction contact between the sensing gel assembly and the housing, enabling smooth rolling during scanning. The sensor is self-contained, with onboard power and wireless image streaming.
Fig. 3: GelSphere sensor design. (A) Fabricated GelSphere components. The sensing gel pad is removable to allow battery replacement and maintenance. The cap is fastened to the housing to support the magnetic suspension structure and retain the steel balls in the bearing race. (B) Assembled prototype. The sensor is able to roll omnidirectionally when fully assembled.
Fig. 4: Fabrication Process for making the gel pad and the shell. The silicone mold (Mold Star 15) is created using the gel pad replica, and XP-565 silicone elastomer is added to the cured silicone mold. The off-the-shelf plastic shell is added to the top mold and pressed carefully with the bottom part of the mold, ensuring no air is caught. The cured elastomer is air-sprayed with the desired coating material to finish the gel pad.
(A) Single-frame (Hex Indenter)
(B) Large-surface (Feather)
Coating
Dot Product
MSE
MAE
SSIM
PSNR [dB]
Dist. MSE [ mm2 ]
Dist. MAE [ mm ]
Angle MSE [ ∘2 ]
Angle MAE [ ∘ ]
Specular
0.965
0.023
0.111
0.950
22.62
1.6169
1.070
3.7
1.3
Matte
0.939
0.041
0.156
0.930
20.08
2.1813
1.251
12.8
2.7
TABLE I: Comparison of reconstruction performance under specular (bronze) and matte coatings for single-frame and large-surface reconstruction. Higher Dot Product, SSIM, and PSNR, and lower MSE/MAE indicate better alignment with ground truth.
Fig. 5: GelSphere’s single frame sensing performance. (A) A 2D normal-estimation accuracy plot was generated from the indentation of a hexagonal pyramid at multiple locations. The RGB direction labels in the dot-product maps are for visual reference. The three LED groups are positioned 120∘ apart around the ring. Left: specular, Right: matte. (B) A key and an M8 screw were pressed into both specular and matte sensing gel, showcasing the reconstruction performance.
Fig. 6: Large surface reconstruction of an SLA-printed feather using specular and matte-coated GelSphere sensors. (A) The printed ground-truth and the reconstructed meshes from specular and matte coatings. (B) Surface normal maps are used for error analysis. 24 correspondence points are manually selected on the ground truth and each reconstructed normal map. These points are used to compare pair-wise distances and local normal directions between corresponding points. The specular coating yields lower error than the matte coating, as shown in Table I .
Fig. 7: Contact normal force estimation using GelSphere. (A) We use linear guides to constrain motion to the z-axis in the force application setup. The ground truth force is measured using a force-torque sensor (NRS-6050-D50, Nordbo Robotics), while indenters of different radii are manually pushed onto the sensor surface. (B) The size of the contact area increases as the force increases. The specular surface has a much higher visual sensitivity compared to the matte coating. (C) The specular-coated sensor estimate forces with MAE of 0.44 N while the Matte coating has almost double the amount, resulting from a lower signal-to-noise ratio compared to specular.
High-fidelity visuo-tactile sensing is important for precise robotic manipulation, yet most vision-based tactile sensors rely on opaque coatings that enable tactile sensing but block direct visual observation. We propose MuxGel, a spatially multiplexed sensor that captures both external visual information and contact-induced tactile signals through a single camera. By using a checkerboard coating pattern, MuxGel interleaves tactile-sensitive regions with transparent windows for external vision. This design maintains standard form factors, allowing for plug-and-play integration into GelSight-style sensors by simply replacing the gel pad. To recover dense visual and tactile signals from the multiplexed inputs, we develop a U-Net-based reconstruction framework trained with a sim-to-real pipeline. Experiments on unseen objects demonstrate the framework's generalization and accuracy. We further demonstrate MuxGel in grasping tasks, where visual feedback supports alignment and tactile feedback supports contact interaction. Results show that MuxGel enables single-camera dual-modal sensing within a GelSight-style implementation, providing local visual feedback and reconstructed tactile feedback with potential extension to other optical tactile sensors. Project webpage: https://zhixianhu.github.io/muxgel/.
Zhixian Hu, Zhengtong Xu, Sheeraz Athar +2
Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA.
Vision-based tactile sensors (VBTS) provide rich contact information for robotic manipulation, but existing designs can be hard to simplify and adapt to the size and constraints of humanoid fingertips. We introduce GlowTact, a pressure-responsive vision-based tactile sensing mechanism that directly visualizes contact pressure. GlowTact requires only single-color, non-directional illumination, and the raw tactile image directly represents the pressure distribution without explicit geometry reconstruction. This simple sensing principle enables compact, customizable tactile sensors while preserving high sensitivity and rich spatial detail. We demonstrate gram-scale contact detection, accurate normal-force estimation, and reconstruction of fine contact geometry, including M1 screw threads. These results establish GlowTact as a practical new sensing technology for compact humanoid fingertips, combining a durable nitrile membrane and simple optical design with sensitive and information-rich tactile perception.
Yuxiang Ma, Megha Tippur, Pengfei Ye +4
Massachusetts Institute of Technology, Cambridge, MA, USA · Harvard University, Cambridge, MA, USA
Vision and touch provide complementary cues for object pose and local contact geometry, but external cameras are often occluded at the contact site. We present GelSlim 5.0, a fingertip sensor that acquires visual and tactile observations with one fixed camera and a continuous UV-responsive contact layer. By switching illumination, the camera sequentially acquires through-membrane visual images and contact-sensitive tactile images from a shared viewpoint, avoiding inter-camera extrinsic calibration. To combine these observations, we introduce VisTacFusion, a multimodal transformer with separate prediction paths for planar object pose and contact geometry. Asymmetric bottleneck fusion incorporates visual context while preserving tactile spatial features for depth and surface-normal estimation. Modality dropout enables inference with paired inputs or either modality alone. On real validation data from 12 known objects, paired inputs reduce depth MSE, angular error, and translation L1 by 61.2%, 72.3%, and 61.3%, respectively, relative to tactile-only inference with the same model. Relative to visual-only inference, the corresponding reductions are 93.7%, 6.1%, and 8.6%.