Organizations: Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · Fudan University · Shanghai Innovation Institute · Tsinghua University
Vision-language-action (VLA) models have shown great potential in building generalist robots, but still face a dilemma-misalignment of 2D image forecasting and 3D action prediction. Besides, such a vision-action entangled training manner limits model learning from large-scale, action-free web video data. To address these issues, we propose DeFI, a novel framework that Decouples visual Forward and Inverse dynamics pretraining to exploit respective data sources, wherein video generation and action prediction are disentangled. We introduce the General Forward Dynamics Model (GFDM), pretrained on diverse human and robot videos for future prediction, and the General Inverse Dynamics Model (GIDM), trained via self-supervised learning to infer latent actions from unlabeled video transitions. These models are then integrated into a unified architecture for end-to-end finetuning on downstream tasks. In this manner, GFDM and GIDM first shine separately and then cooperate for mutual benefit. Extensive experiments on CALVIN ABC-D and SimplerEnv demonstrate state-of-the-art performance, with DeFI achieving an average task length of 4.51 for CALVIN, 51.2% success rate on SimplerEnv-Fractal benchmark and 81.3% success rate in real-world deployment, significantly outperforming prior methods.
Figures & tables
Figure 1: We disentangle robot learning via two decoupled components: a visual forward dynamics model pretrained on large-scale mixed videos via video generation, and an inverse dynamics model pretrained on mixed videos through self-supervised action representation. Then they are coupled during fine-tuning in an end-to-end manner to adapt to downstream tasks. This decoupled pretraining paradigm unleashes the potential of massive action-free videos for policy learning, while retaining robot-specific action grounding, leading to improved success rates across diverse benchmarks.
Figure 2: Overall framework of DeFI. Stage I (Decoupled pretraining): (a) A general visual forward dynamics model is pretrained with human and robot videos via a video generation objective, predicting future frames from current observations and instructions. (b) In parallel, a general inverse dynamics model is pretrained in a self-supervised manner to map pairs of observations (ot,ot+n) into latent actions, capturing inverse dynamics knowledge without explicit action labels. Stage II (Coupled finetuning): The forward and inverse models are coupled, and a diffusion-based adapter is used to generate executable robot action sequences. This two-stage framework unleashes the rich priors of human videos while grounding them in robot data for scalable policy learning.
Figure 3: Experiments setup on CALVIN ABC-D, SimplerEnv Google Robot and real-world Franka Robot. We evaluate DeFI across 3 simulation environments.
View
Method
1
2
3
4
5
Avg. Len. ↑
Third View
SuSIE ( Black et al., 2023 )
87.0
69.0
49.0
38.0
26.0
2.69
CLOVER ( Bu et al., 2024b )
96.0
83.5
70.8
57.5
45.4
3.53
UniVLA ( Bu et al., 2025b )
95.5
85.8
75.4
66.9
56.5
3.80
DeFI
92.9
87.2
81.2
75.0
68.4
4.05
Multi-View
GR-1 ( Wu et al., 2024 )
85.4
71.2
59.6
49.7
40.1
3.06
OpenVLA ( Kim et al., 2024 )
91.3
77.8
62.0
52.1
43.5
3.27
Table 1: CALVIN ABC-D results. We present the average success computed over 1000 rollouts for each task and the average number of completed tasks to solve 5 instructions consecutively (Avg. Len.). DeFI shows significant superiority over baselines. The best results are bolded . *We reproduced results of π0.5 , GR00T N1 and OpenVLA-OFT on CALVIN.
Figure 5
SimplerEnv on Google Robot Tasks
Model
Visual Matching
Variant Aggregation
Pick Coke Can
Move Near
Open/Close Drawer
Avg.
Pick Coke Can
Move Near
Open/Close Drawer
Avg.
Octo-Base ( Team et al., 2024 )
17.0%
4.2%
22.7%
16.8%
0.6%
3.1%
1.1%
1.1%
TraceVLA ( Zheng et al., 2024 )
28.0%
53.7%
57.0%
42.0%
60.0%
56.4%
31.0%
45.0%
OpenVLA ( Kim et al., 2024 )
16.3%
46.2%
35.6%
27.7%
54.5%
47.7%
17.7%
39.8%
DeFI
54.2 %
60.7%
38.6%
51.2%
53.9%
58.2%
24.0%
45.4%
Table 2: Evaluation results across different policies on SimplerEnv . We evaluate DeFI on 3 tasks on the Google Robot in SimplerEnv.
Method
Success Rate (%)
Place
Open
Close
Cut
Stack Bowl
Stack Cube
Stack Bottle
Pour Water
Average
Diffusion Policy ( Chi et al., 2023 )
70.0
40.0
70.0
50.0
45.0
35.0
40.0
35.0
48.2
Octo-Base ( Team et al., 2024 )
55.0
35.0
60.0
20.0
30.0
25.0
30.0
20.0
34.4
OpenVLA ( Kim et al., 2024 )
50.0
40.0
65.0
40.0
30.0
35.0
45.0
45.0
43.8
DeFI
90.0
75.0
100.0
80.0
80.0
70.0
80.0
75.0
81.3
Table 3: Real-world evaluation with the Franka Robot across eight tasks.
Figure 6: Real-world robot setup.
Table 9Table 10Table 11
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Train parameter
Value
GPU
NVIDIA H100
Number of GPUs
8
Pretraining time of GFDM
3 days
Pretraining time of GIDM
1.5 days
Finetuning time on CALVIN
0.5 days
Training memory on CALVIN
64G
Appendix
Table 10: Training and model parameters used in our DeFI.
Category
Training dataset mixture
Proportion
Robot Videos
Fractal ( Brohan et al., 2023 )
30%
Bridge ( Ebert et al., 2021 ; Walke et al., 2023 )
10%
CALVIN-ABC ( Mees et al., 2022b )
30%
Human Videos
Something-Something-v2 Goyal et al. (2017)
15%
Ego4D Grauman et al. (2022)
15%
Appendix
Table 11: The general forward dynamics model of our DeFI is trained on a mixture of data from the Open X-Embodiment ( O’Neill et al., 2023 ) and CALVIN ( Mees et al., 2022b ) robot video datasets, as well as the Ego4D ( Grauman et al., 2022 ) and Something-Something-v2 ( Goyal et al., 2017 ) human video datasets. The proportions are normalized to sum to 100%.
Category
Training dataset mixture
Proportion
Robot Video
Fractal ( Brohan et al., 2023 )
16.3%
Kuka ( Kalashnikov et al., 2018 )
7.4%
Bridge ( Ebert et al., 2021 ; Walke et al., 2023 )
8.0%
Taco Play ( Mees et al., 2022a )
4.1%
Jaco Play ( Dass et al., 2023 )
0.7%
Berkeley Cable Routing ( Luo et al., 2024 )
0.4%
Appendix
Table 12: The general inverse dynamics model of our DeFI is trained on a mixture of data from the Open X-Embodiment ( O’Neill et al., 2023 ) robot video dataset and the Ego4D ( Grauman et al., 2022 ) human video dataset. The proportions are normalized to sum to 100%.
Figure 15
Model part
Inference time
General Forward Dynamics Model
86.1ms
General Inverse Dynamics Model
42.9ms
Action Adapter
24.3ms
Appendix
Table 13: The inference time of our DeFI is measured on an NVIDIA GeForce RTX 4090 GPU, averaged over five runs.
Figure 9: Qualitative results of different denosing step for the forward dynamics model.
Figure 10: Qualitative results of failure cases.
Figure 11: Qualitative results of failure cases.
Figure 12: Qualitative attention heatmap results of the general inverse dynamics model on robot and human videos.
Figure 13: Qualitative results of the CALVIN long-horizon task.
Figure 14: Qualitative results of the CALVIN long-horizon task.
Figure 15: Qualitative results of SimplerEnv evaluation on Google Robot.
Figure 16: Qualitative results of SimplerEnv evaluation on Google Robot.
Figure 17: Qualitative results of real-world experiments.