ForceLDM: A Force-aware Latent Dynamics Model for Contact-Rich Manipulation
Authors: Peiyuan Tang, Yao Li, Xiaodong Zhang, Wuyang Zhang, Qin Xia, Yanyong Zhang, Zijiang Yang
Organizations: School of Computer Science and Technology, Xi’an Jiaotong University · Suzhou Artificial Intelligence Laboratory · School of Artificial Intelligence and Data Science, University of Science and Technology of China · Suzhou Institute for Advanced Research, University of Science and Technology of China · School of Computer Science and Technology, University of Science and Technology of China · Synkrotron, Inc.
Contact-rich manipulation requires fine-grained control and complex physical interactions with the environment, posing significant challenges for robot policy learning. Vision-only policies struggle with such tasks, as contact states and interaction forces are difficult to infer from visual observations alone. Existing methods mitigate this issue by incorporating force or tactile sensing. However, they often treat these signals as auxiliary observations without fully exploiting their role in modeling future interaction dynamics. To address this, we propose ForceLDM, a force-aware latent dynamics model that learns future contact and motion dynamics to guide action generation. Rather than directly predicting future observations, which can entangle task-relevant dynamics with irrelevant sensory details, ForceLDM learns a compact representation of future dynamics through knowledge distillation. Specifically, a teacher network with privileged future force/torque and optical flow signals extracts future dynamic features, which are then distilled into a student network trained only on current observations. This enables the student model to reason about upcoming contacts and physical dynamics without relying on future sensory inputs at deployment. To improve robustness and generalization, we further introduce a curriculum-based progressive noise injection strategy to mitigate over-reliance on future features. Experiments on five real-world contact-rich manipulation tasks demonstrate that ForceLDM outperforms state-of-the-art baselines, generalizes effectively to novel objects, and remains robust to environmental perturbations. Project page and code are available at https://tpy001.github.io/ForceLDM/.
Figures & tables
Figure 1: Comparison with prior methods. (a) Force-aware VLAs incorporate force feedback but lack explicit modeling of future interaction dynamics. (b) Video-based WAMs model future dynamics through visual prediction, but often lack force/tactile feedback and may fit task-irrelevant visual details. (c) ForceLDM learns compact, action-relevant future dynamics by distilling future motion and force cues from a privileged teascher.
Figure 2: Overview of ForceLDM. (a) Both teacher and student generate actions from current multimodal observations and language instructions. The teacher additionally receives future force and motion tokens, and its future-aware representations are distilled into the student’s future queries. (b) The future dynamics encoder converts future force/torque and optical flow into compact force and motion tokens. (c) Progressive noise injection perturbs the student’s future queries with scheduled noise to improve generalization.
Figure 3: Five real-world contact-rich manipulation tasks used in our experiments.
Figure 4: Out-of-distribution generalization settings used in our real-world evaluation.
Model
USB Insertion
Gear Assembly
Box Flipping
Board Wiping
VGA Plugging
Avg.
π0 [ 5 ] w/o force
44.0
46.7
55.0
60.0
0.0
41.1
π0 [ 5 ] w/ force
64.0
83.3
70.0
70.0
5.0
58.5
TA-VLA [ 52 ]
80.0
93.3
80.0
75.0
10.0
67.7
ForceVLA [ 46 ]
52.0
66.7
65.0
65.0
5.0
50.7
Seer [ 42 ] w/ Force
76.0
70.0
95.0
70.0
50.0
72.2
Ours
92.0
83.3
95.0
80.0
85.0
87.1
Table 1: Comparison of success rates (%) on in-distribution (ID) contact-rich manipulation tasks.
Model
USB Insertion
Gear Assembly
Box Flipping
Avg.
Object
Occlusion
Height
Color
Material
Size
π0 [ 5 ] w/o force
20.0
26.7
0.0
40.0
26.7
0.0
18.9
π0 [ 5 ] w/ force
40.0
73.3
0.0
46.7
26.7
33.3
36.7
TA-VLA [ 52 ]
20.0
66.7
13.3
66.7
13.3
26.7
34.5
ForceVLA [ 46 ]
26.7
20.0
0.0
40.0
0.0
13.3
16.7
Seer [ 42 ] w/ Force
46.7
20.0
53.3
60.0
73.3
40.0
48.9
Table 2: Comparison of success rates (%) on out-of-distribution (OOD) tasks under visual occlusion and changes in object instance, color, assembly height, material, and size.
Future Target
Success Rate (%)
Visual
Force
Motion
USB Insertion
Box Flipping
VGA Plugging
Avg.
–
–
✓
92.0
60.0
60.0
70.7
–
✓
–
84.0
90.0
30.0
68.0
✓
✓
–
80.0
85.0
35.0
66.7
–
✓
✓
92.0
95.0
85.0
90.7
Table 3: Ablation of future modalities used for dynamics distillation. Visual, Force, and Motion denote future RGB observations, force/torque signals, and optical flow, respectively. We report success rates (%) on USB Insertion, Box Flipping, and VGA Plugging.
Configuration
USB Insertion
Box Flipping
VGA Plugging
Avg.
ID
OOD
ID
OOD
w/o Ldistill
44.0
30.0
85.0
30.0
25.0
42.8
w/o LT
60.0
36.7
80.0
16.7
10.0
40.7
w/o Stop Gradient
68.0
50.0
60.0
16.7
20.0
42.9
Full Model
92.0
86.7
95.0
66.7
85.0
85.1
Table 4: Ablation of teacher–student distillation components. Ldistill and LT denote the representation alignment loss and teacher action loss, respectively. Stop Gradient denotes gradient stopping on teacher targets. We report success rates (%) under both in-distribution (ID) and out-of-distribution (OOD) conditions.
Training Strategy
USB Insertion
Box Flipping
VGA Plugging
Avg.
ID
OOD
ID
OOD
Joint Prediction
84.0
76.7
65.0
20.0
65.0
62.1
w/o Progressive Noise
84.0
73.3
90.0
40.0
85.0
74.1
Full Model
92.0
86.7
95.0
66.7
85.0
85.1
Table 5: Ablation of the training strategies used in ForceLDM. Joint Prediction combines action learning with direct future-dynamics prediction, while w/o Progressive Noise removes progressive noise injection into the student’s future queries.
Figure 5: Probing Latent Dynamics. We freeze the trained model and fit lightweight probes to predict optical flow and contact forces from its intermediate hidden states. Top: Predicted future 2D motion vectors derived from optical flow (bright orange-red arrows) at five stages of the USB insertion task. Bottom: Measured contact force (blue solid) and predicted force (red dashed).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Real-world experimental setups for (a) the main xArm 7 experiments and (b) the supplementary Piper-X evaluation.
Component
xArm 7
Piper-X
DoF
7
6
Force input
6D F/T sensor
Joint torque
Cameras
RealSense L515 + Gemini 336L
RealSense D435 + Gemini 335
Policy backbone
π0
GR00T N1.5
Appendix
Table 6: Comparison of experimental setups across robot platforms.
Robot Platform
Task
Demos
Frames
Duration (h)
xArm 7
USB Insertion
100
55,667
0.52
Gear Assembly
80
64,091
0.59
Box Flipping
80
71,795
0.66
Board Wiping
80
70,810
0.66
VGA Plugging
80
69,874
0.65
Total
420
332,237
3.08
Appendix
Table 7: Dataset statistics for the xArm 7 and Piper-X robot platforms.
Parameter
Value
Optimization
Training hardware
1 NVIDIA A100, 80 GB
Training steps
30,000
Batch size
16
Optimizer
AdamW
Peak learning rate
2.5×10−5
Appendix
Table 8: Training hyperparameters for ForceLDM.
Method
USB Insertion
Charger Insertion
Avg.
GR00T N1.5
15.0
25.0
20.0
GR00T N1.5 + Torque Input
25.0
10.0
17.5
TA-VLA
70.0
55.0
62.5
ForceLDM (Ours)
80.0
60.0
70.0
Appendix
Table 9: Real-world manipulation success rates (%) on the Piper-X platform. Each method is evaluated over 20 trials per task, with the average success rate reported across both tasks.
Horizon H
Box Flipping
Gear Assembly
Average
16
100.0
86.7
93.4
32
100.0
90.0
95.0
50
90.0
70.0
80.0
Appendix
Table 10: Effect of the prediction horizon on task success rates (%).
Configuration
USB Insertion
Charger Insertion
Avg.
GR00T N1.5
15.0
25.0
20.0
TA-VLA
70.0
55.0
62.5
ForceLDM (RAFT)
80.0
60.0
70.0
ForceLDM (DPFlow)
75.0
70.0
72.5
Appendix
Table 11: Comparison of optical-flow estimators on real-world manipulation success rates (%) using Piper-X.
Figure 7: Successful executions of five real-world manipulation tasks: USB Insertion, Gear Assembly, Box Flipping, Board Wiping, and VGA Plugging. Each row shows seven frames from the initial state to task completion.
Figure 8: Successful executions under OOD conditions: (a) an unseen USB connector, (b) an unseen gear color, (c) an increased assembly height, (d) an unseen box material, and (e) an unseen box size. Each row shows seven frames from the initial state to task completion.
Figure 9: Representative failure cases: (a) the USB plug failing to fully insert into the target port, (b) a missed gear grasp, (c) a box falling back during flipping under OOD conditions, (d) the VGA connector released before being fully inserted, and (e) small residual marks after wiping. Each row shows seven frames from the initial state to the terminal failure state.
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.