Vision-Language-Action (VLA) models have emerged as a promising paradigm for robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1 robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that video generation can serve as an effective scaling proxy for robot policy learning. We release code and models at https://dit4dit.github.io/.
Figures & tables
Figure 1: Proxy objectives for scalable robot policy learning. Left: Comparison of three representative training paradigms: Grounding (object-level semantic alignment), FLARE-style ( Zheng et al., 2025 ) latent modeling (VLM-to-future-frame feature prediction), and Video generation (learning physically plausible future dynamics). Right: Video generation serves as the strongest scaling proxy, yielding higher sample efficiency (up to >10× ), faster convergence (up to 7× ), and more favorable scaling trends across data regimes, with consistently better downstream manipulation success than semantic-centric baselines. All results are reported as the average success rate over 24 tasks in the RoboCasa-GR1 tabletop benchmark ( Nasiriany et al., 2024 ; Bjorck et al., 2025 ) .
Figure 2: Overview of the proposed DiT4DiT framework. Top : Given the current observation and language goal, the video DiT predicts future dynamics and exposes intermediate generative features at the specific flow timestep; these features condition the action DiT to infer control trajectories. The two models are jointly optimized with a dual flow-matching objective for video generation and action prediction. Below : Generated visual plans via video DiT (More examples are shown in Fig. 10 ).
Figure 3: Asymmetric tri-timestep design. We decouple the diffusion timesteps to optimize joint video-action generation. The video module uses uniform sampling ( τv ) to capture the full denoising trajectory, while the action module uses Beta sampling ( τa ) to focus on critical control phases. Meanwhile, stable visual conditions are extracted at a fixed deterministic timestep ( τf ) from the evolving hidden states ( ht1→ht0 ).
Figure 4: Real-world evaluation suite on the Unitree G1 humanoid robot. The selected tasks evaluate distinct dimensions of robotic proficiency, ranging from high-precision spatial manipulation (e.g., stack up the cups, insert plate into the rack, arrange the flower ) to complex, extended-horizon execution (e.g., box packing, drawer interaction) .
Spatial SR (%)
Object SR (%)
Goal SR (%)
Long SR (%)
Average SR (%)
Diffusion Policy ( Chi et al., 2023 )
78.3
92.5
68.3
50.5
72.4
Dita ( Hou et al., 2025 )
97.4
94.8
93.2
83.6
92.3
π0 ( Black et al., 2024 )
96.8
98.8
95.8
85.2
94.2
UniVLA ( Bu et al., 2025 )
96.5
96.8
95.6
92.0
95.2
π0.5 ( Intelligence et al., 2025b )
98.8
98.2
98.0
92.4
96.9
OpenVLA-OFT ( Kim et al., 2025 )
97.6
98.4
97.9
94.5
97.1
Table 1: Success rates (%) on the four evaluation suites of the LIBERO simulation benchmark. Bold numbers indicate the highest performance in each category. In this context, from scratch implies that the model was trained without using any action data outside of the current benchmark.
Task
GR00T-N1.5
GR00T-N1.6
Qwen3DiT
DiT4DiT
( Bjorck et al., 2025 )
( NVIDIA et al., 2025b )
( from scratch )
( from scratch )
BottleToCabinetClose
40.0
36.0
50.0
48.0
CanToDrawerClose
56.0
28.0
48.0
74.0
CupToDrawerClose
50.0
12.0
42.0
52.0
MilkToMicrowaveClose
52.0
20.0
38.0
50.0
PotatoToMicrowaveClose
22.0
28.0
18.0
36.0
Table 2: RoboCasa-GR1 tabletop tasks evaluation results (success rate (%)). Bold numbers indicate the highest performance in each category. from scratch implies that the model was trained without using any action data outside of the current benchmark. While the GR00T models are fine-tuned from their pre-trained weights, they are trained for the exact same number of steps as Qwen3DiT and DiT4DiT, which are trained from scratch.
Figure 5: Real-world evaluation results on the Unitree G1 robot . Success rates are reported across seven diverse household tasks. DiT4DiT comprehensively outperforms both the pre-trained GR00T-N1.5 ( Bjorck et al., 2025 ) and the parameter-matched Qwen3DiT baseline, highlighting the efficiency and efficacy of our framework.
Figure 6: Qualitative rollouts of real-world generalization tasks. We evaluate the policy’s zero-shot robustness against severe out-of-distribution physical variations. These include semantic and geometric shifts in Category (unseen cups and vases), complete object substitution in Object (packing corn instead of an eggplant), and scene clutter in Number (stacking four cups instead of three).
Figure 7: Quantitative results of zero-shot generalization. ( Left ) Success rates in the simulated RoboCasa-GR1 environment when evaluated on entirely unseen objects. ( Right ) Success rates on the real-world Unitree G1 robot across four challenging out-of-distribution scenarios, testing category variations, novel object substitutions, and distractor quantities. DiT4DiT demonstrates superior robustness and physical abstraction compared to both the parameter-matched Qwen3DiT baseline and the pre-trained GR00T-N1.5 ( Bjorck et al., 2025 ) .
Figure 8: Ablation studies on the DiT4DiT architecture. (a) Feature extraction layer: Success rate across different hidden layers of the video backbone, with performance peaking at layer 18. (b) Denoising steps: Impact of the number of iterative denoising steps used for action conditioning. A single forward step yields the highest success rate, preventing over-commitment to pixel-level reconstruction. (c) Representation learning: t-SNE visualization of latent features colored by the execution phase (Early, Middle, Late). Supported by a roughly twofold increase in the silhouette score ( Rousseeuw, 1987 ) , our joint training objective successfully induces smooth temporal flows within each task cluster, transitioning fluidly from the Early (blue) through the Middle (yellow) to the Late (red) phases.
Trainable Param Cnt
Deploy Frequency
GR00T-N1.5
2.7B
13Hz
Qwen3DiT
2.3B
9Hz
DiT4DiT
2.2B
6Hz
Table 3: Deployment efficiency comparison . We report the trainable parameter count and real-world deployment frequency (Hz) for DiT4DiT and the primary baselines. While the video backbone introduces a computational trade-off yielding a 6Hz control rate (tested on a single NVIDIA A100 GPU), DiT4DiT remains the most parameter-efficient model and comfortably supports real-time closed-loop physical execution.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Video DiT
Base VGM
Cosmos-Predict2.5-2B ( Ali et al., 2025 )
Attention Implementation
flash_attention_2
Hidden Feature Dim
2048
Extract Layer
18
Action DiT
Appendix
Table 4: Model & training configurations
Dataset
Episode Cnt
Embodiment
DoF
Fourier_GR1_Unified_1K ( Bjorck et al., 2025 )
24,000
GR1 humanoid
29
Fourier_GR1_Pretrain_10K ( Bjorck et al., 2025 )
241,450
GR1 humanoid
29
LIBERO ( Liu et al., 2024 )
1,693
Franka Emika Panda
7
Real Robot
1,400
G1 humanoid
16
Appendix
Table 5: Details of the used datasets . We report the episode count, embodiment type, and degrees of freedom.
Figure 9: Visualization of the robot system setups. The experimental platform consists of a Unitree G1 humanoid robot equipped with dual ALOHA 2 ( Aldaco et al., 2024 ) grippers and an Intel RealSense D435i camera. A human operator utilizes a PICO VR headset and the XRoboToolkit ( Zhao et al., 2025 ) framework to perform teleoperated demonstrations for high-quality data collection.
Figure 10: Future video rollouts generated by DiT4DiT.
Many recent robot policies pursue stronger control by using large pretrained vision-language models (VLMs) as the action backbone. We introduce WorldDiT, a unified diffusion transformer architecture that couples action generation with visual world modeling and achieves strong performance without a large pretrained VLM action backbone. During training, a single diffusion transformer generates continuous action chunks and predicts normalized RGB patch targets from future camera frames. Across four LIBERO simulation suites, WorldDiT lies on the reported Pareto frontier for total model parameters and mean success among methods reporting all four suites. These results provide a strong sub-billion-parameter baseline for future scaling studies.
Vision-language-action (VLA) models have shown great potential in building generalist robots, but still face a dilemma-misalignment of 2D image forecasting and 3D action prediction. Besides, such a vision-action entangled training manner limits model learning from large-scale, action-free web video data. To address these issues, we propose DeFI, a novel framework that Decouples visual Forward and Inverse dynamics pretraining to exploit respective data sources, wherein video generation and action prediction are disentangled. We introduce the General Forward Dynamics Model (GFDM), pretrained on diverse human and robot videos for future prediction, and the General Inverse Dynamics Model (GIDM), trained via self-supervised learning to infer latent actions from unlabeled video transitions. These models are then integrated into a unified architecture for end-to-end finetuning on downstream tasks. In this manner, GFDM and GIDM first shine separately and then cooperate for mutual benefit. Extensive experiments on CALVIN ABC-D and SimplerEnv demonstrate state-of-the-art performance, with DeFI achieving an average task length of 4.51 for CALVIN, 51.2% success rate on SimplerEnv-Fractal benchmark and 81.3% success rate in real-world deployment, significantly outperforming prior methods.
Wenyao Zhang, Bozhou Zhang, Zekun Qi +3
Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · Fudan University +2
Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained video diffusion models, offering a promising path toward generalizable robot manipulation. However, RGB-only video rollouts are not directly actionable: they leave metric 3D motion, contact geometry, and fine-grained spatial constraints under-specified, making action grounding ambiguous. Meanwhile, scaling action supervision across diverse tasks and embodiments remains costly. We present PointAction, a framework that bridges video predictions to robot actions through explicit point-based 4D modeling. PointAction fine-tunes a foundation video generation model to jointly predict future RGB frames and dynamic 3D pointmaps, producing temporally consistent 3D motion of task-relevant scene geometry. These point dynamics serve as a structured, embodiment-agnostic action interface, which a diffusion-based action decoder maps to executable robot actions. By using metric 3D point dynamics as the interface between video prediction and control, PointAction reduces the ambiguity of RGB-only action grounding and supports transfer across tasks and embodiments with limited action supervision. Experiments show that PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing baselines in simulation, and generalizes to two real robot arms unseen during pretraining.
Mutian Tong, Han Jiang, Qiao Feng +2
University of Pennsylvania · Work done during internship at University of Pennsylvania.