We present a zero-shot deformation reconstruction framework for soft robots that operates without any visual supervision at inference time. In this work, zero-shot deformation reconstruction is defined as the ability to infer object-wide deformations on previously unseen soft robots without collecting object-specific deformation data or performing any retraining during deployment. Our method assumes access to a static geometric proxy of the undeformed object, which can be obtained from a STL model. During operation, the system relies exclusively on tactile sensing, enabling camera-free deformation inference. The proposed framework integrates a flexible piezoresistive sensor array with a geometry-aware, cage-based 3D Gaussian deformation model. Local tactile measurements are mapped to low-dimensional cage control signals and propagated to dense Gaussian primitives to generate globally consistent shape deformations. A graph attention network regresses cage displacements from tactile input, enforcing spatial smoothness and structural continuity via boundary-aware propagation. Given only a nominal geometric proxy and real-time tactile signals, the system performs zero-shot deformation reconstruction of unseen soft robots in bending and twisting motions, while rendering photorealistic RGB in real time. It achieves 0.67 IoU, 0.65 SSIM, and 3.48 mm Chamfer distance, demonstrating strong zero-shot generalization through explicit coupling of tactile sensing and structured geometric deformation.
Figures & tables
Fig. 1: Overview of our camera-free framework for real-time reconstruction and RGB rendering of deformable objects from a flexible sensor array. (a) Real-time resistance map acquired from the flexible tactile array. (b) A Graph Attention Network maps encoded sensor features to cage node displacements. (c) Low-dimensional cage representation controlling dense surface deformation. (d) Gaussian primitives updated via cage-based interpolation. (e) Real-time rendering of the deformed object using Gaussian splatting.
Fig. 2: The architecture of cage-based deformation networks. The cage is represented as a graph, where nodes correspond to cage vertices and edges encode local geometric adjacency. A graph attention (GAT) mechanism is employed to adaptively weight neighboring node contributions during deformation propagation, allowing the network to selectively emphasize structurally relevant neighbors under different deformation patterns. This attention-based message passing enables spatially coherent yet flexible deformation modeling, capturing non-uniform and anisotropic deformation behaviors across the cage.
Fig. 3: Tileable tactile sensor patch. (a) Photograph of the fabricated 3×3 cm flexible tactile patch. (b) Exploded view illustrating the three-layer structure, consisting of (i) a 10×10 orthogonal grid of silver-plated nylon electrodes (ii) a Velostat piezoresistive layer, and (iii) TPU encapsulation.
Fig. 4: Two pneumatically actuated soft robots for zero-shot evaluation: (a) twisting and (b) bending.
Fig. 5: 3DGS initialization and qualitative results. (a) Overview of the proposed geometry proxy initialization (b) Qualitative comparison for pneumatic twisting on an unseen soft robot. (c) Qualitative comparison for pneumatic bending on an unseen soft robot. Each comparison shows the predicted deformation cage, the reconstructed 3D Gaussian model (with viewpoints slightly rotated to better reveal surface indentations and protrusions), the rendered RGB view, and the corresponding real-world observation.
Geometric Accuracy
Method
IoU ↑
Chamfer (mm) ↓
Full Model (Cage + GAT + IDW)
0.67
3.48
w/o Cage (Direct Regression)
0.41
4.62
Cage + MLP (no GAT)
0.62
3.97
TABLE I: Ablation Study on Deformation Representation (Full Robot)
Deformation
IoU ↑
SSIM ↑
Chamfer (mm) ↓
Bending (center)
0.76±0.05
–
2.18±0.32
Bending (full)
0.69±0.18
0.692±0.172
3.12±0.86
Twisting (center)
0.72±0.07
–
2.47±0.44
Twisting (full)
0.65±0.21
0.603±0.186
3.84±1.05
Average (full)
0.67±0.20
0.65±0.18
3.48±0.96
TABLE II: Quantitative Results of Zero-Shot Deformation Reconstruction
Fig. 6: The top row (A–D) shows the real captured deformation scenes, while the bottom row (E–H) presents the corresponding results rendered in real time from the sensor. The deformation angles are manually measured and annotated using red lines and numerical labels. Specifically, A, B, E, and F correspond to bending cases, whereas C, D, G, and H correspond to twisting cases. Each top–bottom pair (A–E, B–F, C–G, D–H) represents the same deformation condition, enabling a direct comparison between the real captured angles and the sensor-based reconstruction results.
Method
Occlusion Robust
Low-Latency
Zero-Shot
High-Fidelity Geometry
Angle Error ( ∘ )
Chamfer (mm)
Vision-based [ 29 ]
✗
✗
✗
✓
–
< 1.20
EIT-based [ 3 , 14 ]
✓
✓
✗
✗
1.90
2.32
Tactile / Glove-based [ 10 ]
✓
✓
✗
✗
4.16
4.02
Ours
✓
✓
✓
✓
4.7
3.48
TABLE III: Comparison with Existing Deformation Reconstruction Methods
Fig. 7: Proof-of-concept tactile deformation reconstruction on an everyday object. (a) A human hand presses a wallet instrumented with a surface-mounted flexible tactile sensor. (b–e) Time-sequential renderings of the reconstructed 3DGS model, showing progressive deformation under increasing pressure. The results demonstrate real-time pressure-to-geometry reconstruction beyond soft robotic platforms.
Estimating the full shape of a deformable object is especially challenging when vision is unavailable: in the dark, inside an opaque bag, behind the manipulating hand, or under heavy self-occlusion. Touch is the natural sensor in these settings, but touches are sparse and local. We present a single topology-agnostic estimator that reconstructs the full mesh of a deformable object from only a few touches and no vision, using one permutation-invariant cross-attention architecture that handles a 1D rope, a 2D cloth, and a 3D volumetric soft body. The learned estimator reduces reconstruction error by roughly two-thirds relative to non-learned geometric mesh completion and a Gaussian-process surface baseline, and it outperforms a simpler global-pool set encoder, with the gap growing as more touches are observed. We then show that the estimator's deep-ensemble uncertainty can be used to learn where to touch next, which lowers error further and beats both random touching and a Gaussian-process active baseline at sparse budgets. This gain is modest on average but grows with self-occlusion and on the error tail. When vision is also available, where to touch barely matters, motivating the vision-free setting we study.
Real-time prediction of deformation in highly compliant soft materials remains a significant challenge in soft robotics. While vision-based soft tactile sensors can track internal marker displacements, learning-based models for 3D contact estimation heavily depend on their training datasets, inherently limiting their ability to generalize to complex scenarios such as multi-point sensing. To address this limitation, we introduce TactiVerse, a U-Net-based framework that formulates contact geometry estimation as a spatial heatmap prediction task. Even when trained exclusively on a limited dataset of single-point indentations, our architecture achieves highly accurate single-point sensing, yielding a superior mean absolute error of 0.0589 mm compared to the 0.0612 mm of a conventional regression-based CNN baseline. Furthermore, we demonstrate that augmenting the training dataset with multi-point contact data substantially enhances the sensor's multi-point sensing capabilities, significantly improving the overall mean MAE for two-point discrimination from 1.214 mm to 0.383 mm. By successfully extrapolating complex contact geometries from fundamental interactions, this methodology unlocks advanced multi-point and large-area shape sensing. Ultimately, it significantly streamlines the development of marker-based soft sensors, offering a highly scalable solution for real-world tactile perception.
Junhui Lee, Hyosung Kim, Saekwang Nam
Graduate School of Data Science at Kyungpook National University, 41566 Daegu, Republic of Korea · Major in Data Science Convergence
Flexible sensors are increasingly employed in soft robotics and wearable devices to provide proprioception of freeform deformations.Although supervised learning can train shape predictors from sensor signals, prediction accuracy strongly depends on sensor layout, which is typically determined heuristically or through trial-and-error. This work introduces a model-free, data-driven computational pipeline that jointly optimizes the number, length, and placement of flexible length-measurement sensors together with the parameters of a shape prediction network for large freeform deformations. Unlike model-based approaches, the proposed method relies solely on datasets of deformed shapes, without requiring physical simulation models, and is therefore broadly applicable to diverse robotic sensing tasks. The pipeline incorporates differentiable loss functions that account for both prediction accuracy and manufacturability constraints. By co-optimizing sensor layouts and network parameters, the method significantly improves deformation prediction accuracy over unoptimized layouts while ensuring practical feasibility. The effectiveness and generality of the approach are validated through numerical and physical experiments on multiple soft robotic and wearable systems.
Yingjun Tian, Guoxin Fang, Aoran Lyu +5
Department of Mechanical and Aerospace Engineering, The University of Manchester, United Kingdom · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong