cs.ROAug 16, 2026
SaveVision-Based Tactile Intelligence for Robotics: Sensing, Learning, and Embodied Manipulation
Abstract
Tactile sensing is essential for robots in contact-rich tasks, yet many tactile sensors still provide sparse, low-dimensional signals that do not capture sufficient information for complex robotic perception and interaction. Vision-based tactile sensors (VBTSs) offer a powerful alternative by con-verting contact-induced deformation of a soft interface into im-ages. The image-based formulation gives VBTSs high-resolution, information-rich tactile observations that enable complex robotic tasks. This review surveys the full VBTS pipeline and treats sensing hardware, learning methods, simulation, and datasets as an integrated sensing-and-learning system. We 1) organize representative VBTSs into a hardware taxonomy structured by deformable elastomer design, sensor size and shape, and optical system design to guide future sensor development; 2) present a hierarchical view of learning-based tactile intelligence from low-level signal understanding to task-level policies and foundation models; and 3) examine simulation platforms and tactile datasets as a scaling layer, together with sim-to-real transfer and cross-sensor adaptation for training, benchmarking, and deployment. Finally, we identify open challenges and future directions for VBTSs in robotics. By providing a holistic view of how hardware, AI architectures, simulation, and datasets interact, this review aims to advance tactile intelligence for contact-rich robotic tasks.
Figures & tables
Fig. 1: Organizational framework of this survey.
Fig. 2: Structure and working principle of a vision-based tactile sensor. (a) A representative VBTS consists of a deformable elastomer, illumination subsystem, imaging optics, and one or more image sensors, with the illumination and imaging components forming the optical system. (b) Contact deforms the elastomer, produces an optical response, and generates tactile images from which contact-related physical quantities are inferred.
| Method | Primary cue | Strengths | Limitations | Representative sensor |
|---|---|---|---|---|
| Marker tracking | Discrete feature motion | Interpretable deformation and shear cues | Limited by marker density and tracking quality | DM-Tac [ 29 ] , TacTip [ 18 ] |
| Photometric stereo | Continuous intensity variation | High spatial resolution and fine geometry recovery | Requires careful illumination and calibration | GelSight [ 16 ] , DIGIT [ 19 ] |
| Stereo vision | Multi view disparity | Metric 3D geometry from triangulation | Higher hardware complexity and lower resolution | GelStereo [ 30 ] |
| Shading based reconstruction | Shading and appearance variation | Greater flexibility under non ideal optics | More model ambiguity and stronger dependence on inference | TacShade [ 31 ] , RainbowSight [ 32 ] |
TABLE I: Comparison of optical transduction mechanisms in vision-based tactile sensors.
Fig. 3: Categorical overview of representative vision-based tactile sensors, grouped by hardware design dimension and category to clarify the design choices within each. Each sensor is shown as a block containing an image, sensor name, and publication year; when no explicit sensor name is reported, the first author’s surname is given in parentheses. The color of a block’s border encodes the publication year, while the border style indicates the sensor’s role in that category: a solid border marks the category corresponding to the sensor’s main contribution, and a dashed border marks a secondary contribution. A sensor that spans several categories therefore appears once with a solid border in its primary category and again with dashed borders in its secondary categories.
| Design dimension | Category | Main design focus | Representative sensors and studies |
|---|---|---|---|
| Early architectures | Early VBTS designs | Track visual markers on or within transparent deformable interfaces, or use reflective coatings and controlled illumination to reconstruct contact geometry. | (Ferrier) [ 12 ] , (Kamiyama) [ 13 ] , (Obinata) [ 14 ] , GelSight [ 15 ] . |
| Deformable elastomer design | Multimodal skins | Co-register visual and tactile modes or add non-optical transduction within the elastomer interface. | (Yamaguchi) [ 35 ] , FingerVision [ 36 ] , STS [ 37 ] , Finger-STS [ 38 ] , SpecTac [ 39 ] , TIRgel [ 40 ] , TacTHru [ 41 ] , V-T Palm [ 42 ] , FingerEye [ 43 ] , Vi2TaP [ 44 ] , MVTS [ 45 ] , CompdVision [ 46 ] , (Halwani) [ 47 ] , MagicGel [ 48 ] , FVSight [ 49 ] , OmniSense v2 [ 50 ] , PolyTouch [ 51 ] , PiezoSight [ 52 ] , VibTac [ 53 ] . |
| Marker/pattern encoding | Use sparse markers or dense coded patterns to render deformation fields observable. | TacTip [ 18 ] , DotView [ 54 ] , HSVTac [ 55 ] , TacShade [ 31 ] , TacShade 2.0 [ 56 ] , DelTact [ 57 ] , (Li) [ 58 ] , (Xue) [ 59 ] . | |
| Elastomer-body design | Tune elastomer mechanics, surface structures, and compliant morphologies for sensitivity, resolution, and force reconstruction. | (Zhang) [ 60 ] , (Zaid) [ 61 ] , ML-VBTS [ 62 ] , DenseTact 2.0 [ 63 ] , HiVTac [ 64 ] , (Shiratori) [ 65 ] , UHDA [ 66 ] , Subblescope [ 67 ] , LightTac [ 68 ] , BiTac [ 69 ] , MagicTac [ 70 ] , TacFinRay [ 71 ] . | |
| Sensor size and shape | Broad coverage | Extend sensing over large surfaces, palms, hands, or curved multi-sided bodies. | TacLink [ 72 ] , Pro Tac [ 73 ] , GelGripper [ 74 ] , GelBelt [ 75 ] , Hydropalm [ 76 ] , Tactile SoftHand-A [ 77 ] , ViART [ 78 ] , SI-VTS [ 79 ] . |
| Manipulator integration | Package VBTSs as finger-shaped or gripper-integrated modules for manipulation. | Finger-Shaped GelForce [ 80 ] , Insight [ 81 ] , GelFinger [ 82 ] , (Jiang) [ 83 ] , (Feng) [ 84 ] , GelStereo BioTip [ 30 ] , (Tomomizu) [ 85 ] , GelSight Svelte [ 86 ] , DIGIT [ 19 ] , DigiTac [ 87 ] , R-Tac [ 88 ] , 9DTact [ 89 ] , L 3 F-TOUCH [ 90 ] , Minsight [ 91 ] , MagicGripper [ 92 ] , PneuGelSight [ 93 ] , FlexiRay [ 94 ] , RoTipBot [ 95 ] . |
TABLE II: Hardware taxonomy of vision-based tactile sensors, categorizing representative sensors by design dimension and summarizing the main focus of each design category.
Fig. 4: Hierarchical view of AI-based tactile perception and learning for VBTSs. Raw tactile images or event streams are transformed into representations for physical dynamics estimation, geometry and pose reconstruction, recognition and property inference, multimodal fusion, task-oriented manipulation, and generalizable tactile learning. The side columns indicate representative inputs and outputs at each layer.
| Year | Model | Pretraining objective | Training data | Sensor name(s) | Aligned modalities | Ref. |
|---|---|---|---|---|---|---|
| 2022 | See-Hear-Feel | Self-attention multisensory fusion | Dense packing and pouring data | GelSight, RGB camera, contact microphone | T+V+Au+Ac | [ 205 ] |
| 2024 | UniTouch | Multimodal alignment to ImageBind | Mixed multimodal pairs | GelSight, DIGIT, Taxim | T+V+L+Ac | [ 206 ] |
| 2024 | Sparsh | SSL: masked + self-distillation (DINO / IJEPA / MAE) | 460,000+ tactile images | DIGIT, GelSight, GelSight Mini | T | [ 140 ] |
| 2024 | T3 | Multi-sensor / multi-task SSL | FoTa, 3M+ samples, 11 tasks | GelSight variants (7 in total), DIGIT, DenseTact 2.0 | T | [ 141 ] |
| 2024 | AnyTouch | Static-dynamic masked modelling + multimodal alignment | TacQuad | GelSight Mini, DIGIT, DuraGel, Tac3D | T+V | [ 207 ] |
| 2024 | MMWand | ImageBind-style alignment | MMWand | GelSight Mini | T+V+L | [ 208 ] |
TABLE III: Comparison of general-purpose tactile learning models and foundation policies. ’T’, ’V’, ’L’, ’Au’, and ’Ac’ denote touch, vision, language, audio, and action modalities, respectively.
| Year | Platform/Dataset | Type | Sensor(s) | Suitable tasks | Ref. |
|---|---|---|---|---|---|
| 2024 | TacBench | Benchmark | Multi-sensor | Benchmarking tactile representations across perception and manipulation | [ 140 ] |
| 2026 | ManiFeel | Benchmark | Visuotactile simulation | Evaluating supervised visuotactile manipulation policies | [ 215 ] |
| 2021 | YCB-Sight | Dataset | GelSight, Azure Kinect RGB-D | Tactile shape mapping and perception on household objects | [ 216 ] |
| 2021 | ObjectFolder | Dataset | Simulated DIGIT | Multimodal (vision/audio/touch) representation learning | [ 217 ] |
| 2022 | ObjectFolder 2.0 | Dataset | Simulated | Multisensory learning and sim-to-real transfer | [ 218 ] |
| 2022 | Touch and Go | Dataset | GelSight | Visuotactile representation learning and material/texture | [ 219 ] |
TABLE IV: Tactile simulation platforms and datasets.
Fig. 5: Chronological overview of representative datasets, simulation platforms and benchmarks, and foundation models in vision-based tactile sensing research. Entries are arranged by publication year along the horizontal axis, with entries from 2022 or earlier grouped under 2022, and by the number of supported modalities along the vertical axis. Each entry shows an affiliation logo, the resource or model name, and its supported modalities in parentheses. The circle border color encodes the category: green denotes datasets, blue denotes simulation platforms and benchmarks, and red denotes foundation models. T, V, L, F, Ac, and Au denote tactile, vision, language, force, action, and audio, respectively.