Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China. · School of Future Technology, Harbin Institute of Technology, Harbin, China.
Contact-rich manipulation requires precise interaction feedback. While vision-centric imitation learning is prevalent, external visual observations provide indirect and ambiguous cues about contact states, particularly under occlusion or subtle object--gripper interactions; dedicated tactile or force sensors can provide rich contact information but introduce additional hardware complexity, calibration requirements, and deployment costs. To bridge this gap, we propose VISTA-Policy, an imitation learning paradigm that utilizes the Visual Deformation Field (VDF), a 3D displacement representation of a compliant gripper, as high-dimensional visuo-physical feedback. The framework integrates: 1) a Physics-Aware Encoding Engine for real-time VDF decoding; 2) an Energy Aggregation Denoising Mechanism to isolate true interaction signals; and 3) a Deformation-Augmented Policy Network with incremental gripper actions for precise closed-loop correction. Extensive evaluations on Cross-Scale Object Grasping, Cap Unscrewing, and Calligraphy Writing demonstrate that VISTA-Policy outperforms the strong pure-vision baseline 3D Diffusion Policy and the tactile baseline. VISTA-Policy further demonstrates substantial out-of-distribution generalization to unseen object scales and robustness against dynamic disturbances, offering a durable and cost-effective route toward general-purpose fine-grained manipulation in unstructured environments. Project videos and supplementary materials are available at: https://sites.google.com/view/vista-policy.
Figures & tables
Fig. 1: Overview of VISTA-Policy. By extracting the Visual Deformation Field from a passive compliant gripper, our framework provides high-dimensional visuo-physical feedback for robust contact-rich manipulation without dedicated tactile or force sensors.
Fig. 2: Structure of the passive compliant gripper and its VDF. (a) Front view of the gripper. (b) Observable reactive deformation along the x -axis and z -axis under external contact, providing the physical basis for VDF representation.
Fig. 3: Architecture of VISTA-Policy, comprising three core components: a Physics-Aware Encoding Engine for 3D VDF extraction, an Energy Aggregation Denoising Mechanism for contact denoising, and a Deformation-Augmented Policy Network driven by a relative incremental gripper action space.
Method
Observation Modality
Rel. Gripper Action ( Δ )
Global PC
Wrist PC
VDF
Tactile
DP3-Wrist-Abs
✓
✓
✕
✕
✕
VISTA-Abs
✓
✕
✓
✕
✕
DP3
✓
✕
✕
✕
✓
DP3-Wrist
✓
✓
✕
✕
✓
TDF-DM
✓
✕
✕
✓
✓
TABLE I: Systematic Comparison of Component Configurations between Different Methods
Fig. 4: Overview of the three contact-rich manipulation tasks: (left) Cross-Scale Object Grasping of diverse objects; (center) Cap Unscrewing with varied diameters; (right) Calligraphy Writing on platforms of different heights. The top row illustrates the object sets, while the bottom row shows the task execution sequences.
Fig. 5: Statistical analysis of VDF during the contact phase across the three tasks. VISTA-Policy exhibits lower dispersion than the compared baselines, indicating more consistent contact-induced deformation during execution. (For methods that do not use VDF as policy input, VDF is computed only for post-hoc evaluation.)
Method
Cross-Scale Object Grasping
Cap Unscrewing
Calligraphy Writing
SR (%) ↑
FR (%) ↓
SR (%) ↑
DR (%) ↓
HSR (%) ↑
WSR (%) ↑
DP3
40
70
50
20
50
40
DP3-Wrist
50
50
40
30
90
40
TDF-DM
60
60
50
80
80
20
VISTA ( ours )
100
0
90
0
100
90
TABLE II: Evaluation Results of the Three Tasks
Method
Seen (4 cm)
Unseen
SR (%) ↑
FR (%) ↓
SR (%) ↑
FR (%) ↓
DP3-Wrist-Abs
100
0
50
50
VISTA-Abs
100
0
40
60
DP3-Wrist
60
40
50
50
VISTA ( ours )
100
0
100
0
TABLE III: OOD Scale Generalization Results and Gripper Action Space Ablation
Fig. 6: Demonstration of various execution situations across the three tasks. (a) Several grasping situations in the Grasping task. (b) Robustness evaluations in the Grasping task, including dynamic disturbances and the manipulation of tofu and playing cards. (c) Different execution situations in the Cap Unscrewing task. (d) Various execution situations in the Calligraphy Writing task and the corresponding stroke effects.
Method
Seen
Unseen
SR (%) ↑
FR (%) ↓
SR (%) ↑
FR (%) ↓
DP3
40
60
40
80
DP3-Wrist
60
40
40
60
VISTA-Cat
80
80
20
80
VISTA-MLP
80
40
40
60
TDF-DM
80
40
40
60
TABLE IV: Performance Comparison Under Multi-Scale Training Conditions
Fig. 7: Quantitative evaluation of sample efficiency and disturbance robustness in Cross-Scale Object Grasping.
Method
Minimum-Scale Training (2 cm)
Maximum-Scale Training (6 cm)
Seen
Unseen
Seen
Unseen
SR (%) ↑
DR (%) ↓
SR (%) ↑
DR (%) ↓
SR (%) ↑
DR (%) ↓
SR (%) ↑
DR (%) ↓
DP3-Wrist
20
40
20
60
60
20
40
20
VISTA ( ours )
100
0
80
20
80
40
80
0
TABLE V: Generalization Results for Cap Unscrewing under Boundary-Scale Single-Object Training
Method
SR (%) ↑
DR (%) ↓
DP3
50
20
DP3-Wrist
40
30
VISTA-Cat
60
40
TDF-DM
50
80
VISTA ( ours )
100
0
TABLE VI: Quantitative Results on Multi-Object Cap Unscrewing Experiments
Fig. 8: Evaluation of calligraphy stroke quality and metrics: (a) Scatter plot of Stroke Quality Index (SQI) versus stroke length across different methods to reveal degenerate cases with high SQI but insufficient stroke length; (b) Comparison of average calligraphy stroke metrics (length, width, and SQI) with error bars across VISTA-Policy and the baselines.
Method
HSR (%) ↑
WSR (%) ↑
DP3
50
40
DP3-Wrist
90
40
VISTA-Cat
70
30
VISTA-MLP
100
60
TDF-DM
80
20
VISTA ( ours )
100
90
TABLE VII: Quantitative Results of the Calligraphy Writing Experiment
Fig. 9: Quantitative results of sample efficiency and robustness evaluation in Calligraphy Writing .
Fig. 10: Feature similarity analysis for validating contact representations. (a) Feature consistency across objects with different scales and geometries under similar local contact states. (b) Comparison of contact-phase separation between the point-cloud features and VDF features across sequential non-contact frames (1–3) and contact frames (4–6).
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · TARS Robotics +2