Soft robots and tactile sensors have demonstrated great potential in delicate manipulation tasks. Soft pneumatic robots enable safe contact through compliance, and vision-based tactile sensors offer high-resolution touch perception. However, learning tactile manipulation with compliant robots has been challenging, bottlenecked by the lack of efficient simulation. Existing simulators typically model them in isolation, and exhibit large calibration gaps that are difficult to overcome efficiently. We present PneuTac, a unified framework for tactile-feedback manipulation with soft pneumatic robots. We leverage the material point method (MPM) for modelling the dynamics of the soft robot and the deformable tactile membrane, and 3D Gaussian splatting (3DGS) for rendering. Real-to-sim modelling is done with a simple vision-based method, to then train action and perception networks for efficient simulation with surrogate models. We use the framework to drive a tactile-guided pipeline to collect demonstrations in simulation. Through experiments on a custom-designed pneumatic soft finger with a tactile sensing tip, together with additional cross-device evaluations, we show that PneuTac is capable of accurately modelling soft robots with tactile sensors, and that policies trained with simulation-augmented demonstrations outperform baselines trained on the same real data on three real-world contact-rich compliant manipulation tasks, making it a practical framework for tactile manipulation on compliant hardware.
Figures & tables
Fig. 1: Overview of PneuTac. The framework leverages MPM-3DGS coupling for efficient real-to-sim modelling from CAD models. This supports accurate modelling of soft robot deformation and tactile contact rendering, which facilitates policy learning for real-world contact-rich compliant manipulation.
Fig. 2: PneuTac pipeline. Left: image-based real-to-sim calibration. Middle: residual-torque prediction ( fr ) and probe-to-MPM indentation mapping ( fm ) for fast surrogate simulation. Right: generation of simulated demonstrations guided by real trajectories and tactile observations.
Fig. 3: Real-to-sim comparison of the soft finger and three-chamber arm in novel configurations. The finger’s back view is rendered from the side opposite the calibration camera.
View
PneuTac
Tex-Mesh
Par-Splat
Def-NeRF
Front ↓
50.07±0.83
51.13±1.12
54.58±1.59
55.06±11.72
Back ↓
77.61±11.72
79.09±9.08
78.62±8.43
68.96±12.77
TABLE I: RMSE comparison of soft finger rendering methods for front and back views. Values are reported as mean ± SEM.
Metric
PneuTac
Tacchi
DOT-Sim
Taxim
RMSE ↓
10.18±0.58
10.86±0.29
14.14±0.23
14.13±0.61
PSNR (dB) ↑
28.11±0.54
27.44±0.23
25.13±0.14
25.20±0.36
NCC ↑
0.982±0.001
0.952±0.004
0.913±0.004
0.915±0.011
TABLE II: Comparison of tactile rendering methods. Values are reported as mean ± SEM.
Fig. 4: Tactile image comparison. Columns 1-3: the Digit sensor attached to the soft finger; columns 4-5: a second, unmodified Digit sensor.
Task
Sim+Real
Real
N-Aug
Real-20
No-Tac
PPO
Switch
83.3±3.3
60.0±5.8
73.3±3.3
90.0±5.8
30.0±5.8
63.3±3.3
Egg
90.0±5.8
26.7±8.8
46.7±6.7
40.0±10.0
56.7±3.3
80.0±5.8
Card
66.7±3.3
33.3±8.8
40.0±5.8
50.0±5.8
63.3±3.3
10.0±5.8
TABLE III: Real-world task success rates (%). Values are reported as mean ± SEM across 3 seeds, with 10 trials per seed and task.
Fig. 5: Contact-rich manipulation with the tactile soft finger. Left to right: switch flipping, egg carton opening, and card pulling.
Tactile sensors are increasingly integrated into dexterous robotic manipulators to enhance contact perception. However, learning manipulation policies that rely on tactile sensing remains challenging, primarily due to the trade-off between fidelity and computational cost of soft-body simulations. To address this, we present ETac, a tactile simulation framework that models elastomeric soft-body interactions with both high fidelity and efficiency. ETac employs a lightweight data-driven deformation propagation model to capture soft-body contact dynamics, achieving high simulation quality and boosting efficiency that enables large-scale policy training. When serving as the simulation backend, ETac produces surface deformation estimates comparable to FEM and demonstrates applicability for modeling real tactile sensors. Then, we showcase its capability in training a blind grasping policy that leverages large-area tactile feedback to manipulate diverse objects. Running on a single RTX 4090 GPU, ETac supports reinforcement learning across 4,096 parallel environments, achieving a total throughput of 869 FPS. The resulting policy reaches an average success rate of 84.45% across four object types, underscoring ETac's potential to make tactile-based skill learning both efficient and scalable.
Zhe Xu, Feiyu Zhao, Xiyan Huang +1
School of Information Science and Technology at ShanghaiTech University, Shanghai 201210, China
Tactile perception is key to dexterous manipulation, yet simulating high-resolution elastomer deformation remains computationally prohibitive. Finite element methods (FEM) deliver high fidelity but demand costly remeshing, while Material Point Methods (MPM) suffer from heavy particle-memory tradeoffs. We propose a {reduced-order neural simulation framework} that couples coarse-grained MPM dynamics with an implicit neural decoder to reconstruct sub-particle tactile details from compact latent states. The framework learns a continuous deformation manifold from paired high- and low-resolution simulations, enabling physically consistent, differentiable inference. Compared to the TacIPC, our method achieves over 65% faster simulation and {40% lower memory usage}, while maintaining better geometric fidelity. In tactile rendering and 3D surface reconstruction, our methods further improve accuracy by 25% and produce realistic depth images and surface mesh within a faster inference speed. These results demonstrate that the proposed reduced-order neural model enables high-detail, physically grounded tactile simulation with substantial efficiency gains for robotic interaction and optimization.
Yuhu Guo, Zhikai Shen, Jiasheng Qu +4
The University of Manchester, UK · The Chinese University of Hong Kong, Hong Kong SAR, China · University of Leeds, UK +1
Tactile sensing can substantially improve contact-rich robotic manipulation, yet its practical deployment remains limited by the fragility, calibration requirements, and maintenance burden of tactile hardware. This raises a fundamental question: can robots benefit from tactile knowledge without requiring tactile sensors at deployment? We present TacImag, a tactile imagination framework that predicts tactile observations from vision and proprioception and uses the generated signals to guide manipulation policies. Trained from paired visuotactile demonstrations, TacImag enables touch-informed manipulation using only visual observations at test time. We evaluate TacImag on six simulated and four real-world manipulation tasks. Across simulation and real-world experiments, imagined tactile observations consistently improve manipulation performance without requiring tactile hardware. In real-world experiments, imagined force fields improve contact-sensitive tasks by 44.4% on average, whereas imagined tactile images improve texture-sensitive tasks by 23.3%, revealing that the effectiveness of tactile imagination depends strongly on the relationship between tactile representation and task requirements. Our results further suggest that tactile imagination does not simply recover missing tactile measurements. Instead, it acts as a form of contact-aware supervision that transforms subtle visual interaction cues into representations that are easier for manipulation policies to exploit.
Zhiyuan Zhang, Adeesh Desai, Jyun-Chi Hu +7
Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA