ForceLDM: A Force-aware Latent Dynamics Model for Contact-Rich Manipulation
Authors: Peiyuan Tang, Yao Li, Xiaodong Zhang, Wuyang Zhang, Qin Xia, Yanyong Zhang, Zijiang Yang
Organizations: School of Computer Science and Technology, Xi’an Jiaotong University · Suzhou Artificial Intelligence Laboratory · School of Artificial Intelligence and Data Science, University of Science and Technology of China · Suzhou Institute for Advanced Research, University of Science and Technology of China · School of Computer Science and Technology, University of Science and Technology of China · Synkrotron, Inc.
Contact-rich manipulation requires fine-grained control and complex physical interactions with the environment, posing significant challenges for robot policy learning. Vision-only policies struggle with such tasks, as contact states and interaction forces are difficult to infer from visual observations alone. Existing methods mitigate this issue by incorporating force or tactile sensing. However, they often treat these signals as auxiliary observations without fully exploiting their role in modeling future interaction dynamics. To address this, we propose ForceLDM, a force-aware latent dynamics model that learns future contact and motion dynamics to guide action generation. Rather than directly predicting future observations, which can entangle task-relevant dynamics with irrelevant sensory details, ForceLDM learns a compact representation of future dynamics through knowledge distillation. Specifically, a teacher network with privileged future force/torque and optical flow signals extracts future dynamic features, which are then distilled into a student network trained only on current observations. This enables the student model to reason about upcoming contacts and physical dynamics without relying on future sensory inputs at deployment. To improve robustness and generalization, we further introduce a curriculum-based progressive noise injection strategy to mitigate over-reliance on future features. Experiments on five real-world contact-rich manipulation tasks demonstrate that ForceLDM outperforms state-of-the-art baselines, generalizes effectively to novel objects, and remains robust to environmental perturbations. Project page and code are available at https://tpy001.github.io/ForceLDM/.
Figures & tables
Figure 1: Comparison with prior methods. (a) Force-aware VLAs incorporate force feedback but lack explicit modeling of future interaction dynamics. (b) Video-based WAMs model future dynamics through visual prediction, but often lack force/tactile feedback and may fit task-irrelevant visual details. (c) ForceLDM learns compact, action-relevant future dynamics by distilling future motion and force cues from a privileged teascher.
Figure 2: Overview of ForceLDM. (a) Both teacher and student generate actions from current multimodal observations and language instructions. The teacher additionally receives future force and motion tokens, and its future-aware representations are distilled into the student’s future queries. (b) The future dynamics encoder converts future force/torque and optical flow into compact force and motion tokens. (c) Progressive noise injection perturbs the student’s future queries with scheduled noise to improve generalization.
Figure 3: Five real-world contact-rich manipulation tasks used in our experiments.
Figure 4: Out-of-distribution generalization settings used in our real-world evaluation.
Model
USB Insertion
Gear Assembly
Box Flipping
Board Wiping
VGA Plugging
Avg.
π0 [ 5 ] w/o force
44.0
46.7
55.0
60.0
0.0
41.1
π0 [ 5 ] w/ force
64.0
83.3
70.0
70.0
5.0
58.5
TA-VLA [ 52 ]
80.0
93.3
80.0
75.0
10.0
67.7
ForceVLA [ 46 ]
52.0
66.7
65.0
65.0
5.0
50.7
Seer [ 42 ] w/ Force
76.0
70.0
95.0
70.0
50.0
72.2
Ours
92.0
83.3
95.0
80.0
85.0
87.1
Table 1: Comparison of success rates (%) on in-distribution (ID) contact-rich manipulation tasks.
Model
USB Insertion
Gear Assembly
Box Flipping
Avg.
Object
Occlusion
Height
Color
Material
Size
π0 [ 5 ] w/o force
20.0
26.7
0.0
40.0
26.7
0.0
18.9
π0 [ 5 ] w/ force
40.0
73.3
0.0
46.7
26.7
33.3
36.7
TA-VLA [ 52 ]
20.0
66.7
13.3
66.7
13.3
26.7
34.5
ForceVLA [ 46 ]
26.7
20.0
0.0
40.0
0.0
13.3
16.7
Seer [ 42 ] w/ Force
46.7
20.0
53.3
60.0
73.3
40.0
48.9
Table 2: Comparison of success rates (%) on out-of-distribution (OOD) tasks under visual occlusion and changes in object instance, color, assembly height, material, and size.
Future Target
Success Rate (%)
Visual
Force
Motion
USB Insertion
Box Flipping
VGA Plugging
Avg.
–
–
✓
92.0
60.0
60.0
70.7
–
✓
–
84.0
90.0
30.0
68.0
✓
✓
–
80.0
85.0
35.0
66.7
–
✓
✓
92.0
95.0
85.0
90.7
Table 3: Ablation of future modalities used for dynamics distillation. Visual, Force, and Motion denote future RGB observations, force/torque signals, and optical flow, respectively. We report success rates (%) on USB Insertion, Box Flipping, and VGA Plugging.
Configuration
USB Insertion
Box Flipping
VGA Plugging
Avg.
ID
OOD
ID
OOD
w/o Ldistill
44.0
30.0
85.0
30.0
25.0
42.8
w/o LT
60.0
36.7
80.0
16.7
10.0
40.7
w/o Stop Gradient
68.0
50.0
60.0
16.7
20.0
42.9
Full Model
92.0
86.7
95.0
66.7
85.0
85.1
Table 4: Ablation of teacher–student distillation components. Ldistill and LT denote the representation alignment loss and teacher action loss, respectively. Stop Gradient denotes gradient stopping on teacher targets. We report success rates (%) under both in-distribution (ID) and out-of-distribution (OOD) conditions.
Training Strategy
USB Insertion
Box Flipping
VGA Plugging
Avg.
ID
OOD
ID
OOD
Joint Prediction
84.0
76.7
65.0
20.0
65.0
62.1
w/o Progressive Noise
84.0
73.3
90.0
40.0
85.0
74.1
Full Model
92.0
86.7
95.0
66.7
85.0
85.1
Table 5: Ablation of the training strategies used in ForceLDM. Joint Prediction combines action learning with direct future-dynamics prediction, while w/o Progressive Noise removes progressive noise injection into the student’s future queries.
Figure 5: Probing Latent Dynamics. We freeze the trained model and fit lightweight probes to predict optical flow and contact forces from its intermediate hidden states. Top: Predicted future 2D motion vectors derived from optical flow (bright orange-red arrows) at five stages of the USB insertion task. Bottom: Measured contact force (blue solid) and predicted force (red dashed).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Real-world experimental setups for (a) the main xArm 7 experiments and (b) the supplementary Piper-X evaluation.
Component
xArm 7
Piper-X
DoF
7
6
Force input
6D F/T sensor
Joint torque
Cameras
RealSense L515 + Gemini 336L
RealSense D435 + Gemini 335
Policy backbone
π0
GR00T N1.5
Appendix
Table 6: Comparison of experimental setups across robot platforms.
Robot Platform
Task
Demos
Frames
Duration (h)
xArm 7
USB Insertion
100
55,667
0.52
Gear Assembly
80
64,091
0.59
Box Flipping
80
71,795
0.66
Board Wiping
80
70,810
0.66
VGA Plugging
80
69,874
0.65
Total
420
332,237
3.08
Appendix
Table 7: Dataset statistics for the xArm 7 and Piper-X robot platforms.
Parameter
Value
Optimization
Training hardware
1 NVIDIA A100, 80 GB
Training steps
30,000
Batch size
16
Optimizer
AdamW
Peak learning rate
2.5×10−5
Appendix
Table 8: Training hyperparameters for ForceLDM.
Method
USB Insertion
Charger Insertion
Avg.
GR00T N1.5
15.0
25.0
20.0
GR00T N1.5 + Torque Input
25.0
10.0
17.5
TA-VLA
70.0
55.0
62.5
ForceLDM (Ours)
80.0
60.0
70.0
Appendix
Table 9: Real-world manipulation success rates (%) on the Piper-X platform. Each method is evaluated over 20 trials per task, with the average success rate reported across both tasks.
Horizon H
Box Flipping
Gear Assembly
Average
16
100.0
86.7
93.4
32
100.0
90.0
95.0
50
90.0
70.0
80.0
Appendix
Table 10: Effect of the prediction horizon on task success rates (%).
Configuration
USB Insertion
Charger Insertion
Avg.
GR00T N1.5
15.0
25.0
20.0
TA-VLA
70.0
55.0
62.5
ForceLDM (RAFT)
80.0
60.0
70.0
ForceLDM (DPFlow)
75.0
70.0
72.5
Appendix
Table 11: Comparison of optical-flow estimators on real-world manipulation success rates (%) using Piper-X.
Figure 7: Successful executions of five real-world manipulation tasks: USB Insertion, Gear Assembly, Box Flipping, Board Wiping, and VGA Plugging. Each row shows seven frames from the initial state to task completion.
Figure 8: Successful executions under OOD conditions: (a) an unseen USB connector, (b) an unseen gear color, (c) an increased assembly height, (d) an unseen box material, and (e) an unseen box size. Each row shows seven frames from the initial state to task completion.
Figure 9: Representative failure cases: (a) the USB plug failing to fully insert into the target port, (b) a missed gear grasp, (c) a box falling back during flipping under OOD conditions, (d) the VGA connector released before being fully inserted, and (e) small residual marks after wiping. Each row shows seven frames from the initial state to the terminal failure state.
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
Contact-rich manipulation requires policies to generate precise actions by reasoning over contact forces, robot configurations, and interaction histories beyond visual observations. Existing methods passively condition on force feedback rather than actively predicting future contact dynamics, limiting their ability to generate high-precision actions. To address this problem, we introduce Predictive Sensorimotor Representation (PSR) learning, a framework that learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy. Specifically, during a pretraining stage, a multimodal Transformer is trained to learn a hierarchy of predictive representations by jointly forecasting future interaction dynamics. The learned hierarchy subsequently augments the action stream, enabling the resulting policy to exploit contact-relevant cues at multiple depths. We further instantiate PSR within a Vision-Language-Action (VLA) model, resulting in PSR-VLA, and evaluate it on six real-world contact-rich manipulation tasks. Experimental results show that PSR-VLA achieves 91.7% overall success, improving over π0.5, ForceVLA-π0.5, and ForceVLA2-π0.5 by 30.0, 22.5, and 19.2 percentage points, respectively. These results demonstrate the effectiveness of the proposed PSR for force-aware, contact-rich manipulation. Videos of the tasks and stability tests are available at https://psr-vla.pages.dev/.
Vision-Language-Action (VLA) models have shown remarkable generalization by mapping web-scale knowledge to robotic control, yet they remain blind to physical contact. Consequently, they struggle with contact-rich manipulation tasks that require reasoning about force, texture, and slip. While some approaches incorporate low-dimensional tactile signals, they fail to capture the high-resolution dynamics essential for such interactions. To address this limitation, we introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future. Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-vision inputs coupled with wrist-camera local vision and third-person macro vision. To reconcile these multi-scale sensory streams, we first train a unified policy with a Hierarchical Spatial Alignment (HSA) loss that aligns tactile tokens with their spatial counterparts in the wrist and third-person views. To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals. To mitigate tactile data scarcity and the wear-prone nature of tactile sensors, we construct a hybrid large-scale dataset sourced from both high-fidelity digital twin and real-world experiments. By anticipating upcoming tactile states, DreamTacVLA acquires a rich model of contact physics and conditions its actions on both real observations and imagined consequences. Across contact-rich manipulation tasks, it outperforms state-of-the-art VLA baselines, achieving up to 95% success, highlighting the importance of understanding physical contact for robust, touch-aware robotic agents.
Contact-rich manipulation demands human-like integration of perception and force feedback: vision should guide task progress, while high-frequency interaction control must stabilize contact under uncertainty. Existing learning-based policies often entangle these roles in a monolithic network, trading off global generalization against stable local refinement, while control-centric approaches typically assume a known task structure or learn only controller parameters rather than the structure itself. In this paper, we formalize a physically grounded interaction frame, an instantaneous local basis that decouples force regulation from motion execution, and propose a method to recover it from demonstrations. Based on this, we address both issues by proposing Force Policy, a global-local vision-force policy in which a global policy guides free-space actions using vision, and upon contact, a high-frequency local policy with force feedback estimates the interaction frame and executes hybrid force-position control for stable interaction. Real-world experiments across diverse contact-rich tasks show consistent gains over strong baselines, with more robust contact establishment, more accurate force regulation, and reliable generalization to novel objects with varied geometries and physical properties, ultimately improving both contact stability and execution quality. Project page: https://force-policy.github.io/
Hongjie Fang, Shirun Tang, Mingyu Mei +9
Noematrix · Shanghai Jiao Tong University · Zhejiang University +2