cs.ROMar 11, 2026
SaveDiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
Organizations: Mondo Robotics · HKUST(GZ) · HKUST
Abstract
Vision-Language-Action (VLA) models have emerged as a promising paradigm for robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1 robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that video generation can serve as an effective scaling proxy for robot policy learning. We release code and models at https://dit4dit.github.io/.
Figures & tables
Figure 1: Proxy objectives for scalable robot policy learning. Left: Comparison of three representative training paradigms: Grounding (object-level semantic alignment), FLARE-style ( Zheng et al., 2025 ) latent modeling (VLM-to-future-frame feature prediction), and Video generation (learning physically plausible future dynamics). Right: Video generation serves as the strongest scaling proxy, yielding higher sample efficiency (up to ), faster convergence (up to ), and more favorable scaling trends across data regimes, with consistently better downstream manipulation success than semantic-centric baselines. All results are reported as the average success rate over 24 tasks in the RoboCasa-GR1 tabletop benchmark ( Nasiriany et al., 2024 ; Bjorck et al., 2025 ) .
Figure 2: Overview of the proposed DiT4DiT framework. Top : Given the current observation and language goal, the video DiT predicts future dynamics and exposes intermediate generative features at the specific flow timestep; these features condition the action DiT to infer control trajectories. The two models are jointly optimized with a dual flow-matching objective for video generation and action prediction. Below : Generated visual plans via video DiT (More examples are shown in Fig. 10 ).
Figure 3: Asymmetric tri-timestep design. We decouple the diffusion timesteps to optimize joint video-action generation. The video module uses uniform sampling ( ) to capture the full denoising trajectory, while the action module uses Beta sampling ( ) to focus on critical control phases. Meanwhile, stable visual conditions are extracted at a fixed deterministic timestep ( ) from the evolving hidden states ( ).
Figure 4: Real-world evaluation suite on the Unitree G1 humanoid robot. The selected tasks evaluate distinct dimensions of robotic proficiency, ranging from high-precision spatial manipulation (e.g., stack up the cups, insert plate into the rack, arrange the flower ) to complex, extended-horizon execution (e.g., box packing, drawer interaction) .
| Spatial SR (%) | Object SR (%) | Goal SR (%) | Long SR (%) | Average SR (%) | |
| Diffusion Policy ( Chi et al., 2023 ) | 78.3 | 92.5 | 68.3 | 50.5 | 72.4 |
| Dita ( Hou et al., 2025 ) | 97.4 | 94.8 | 93.2 | 83.6 | 92.3 |
| ( Black et al., 2024 ) | 96.8 | 98.8 | 95.8 | 85.2 | 94.2 |
| UniVLA ( Bu et al., 2025 ) | 96.5 | 96.8 | 95.6 | 92.0 | 95.2 |
| ( Intelligence et al., 2025b ) | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 |
| OpenVLA-OFT ( Kim et al., 2025 ) | 97.6 | 98.4 | 97.9 | 94.5 | 97.1 |
Table 1: Success rates (%) on the four evaluation suites of the LIBERO simulation benchmark. Bold numbers indicate the highest performance in each category. In this context, from scratch implies that the model was trained without using any action data outside of the current benchmark.
| Task | GR00T-N1.5 | GR00T-N1.6 | Qwen3DiT | DiT4DiT |
|---|---|---|---|---|
| ( Bjorck et al., 2025 ) | ( NVIDIA et al., 2025b ) | ( from scratch ) | ( from scratch ) | |
| BottleToCabinetClose | 40.0 | 36.0 | 50.0 | 48.0 |
| CanToDrawerClose | 56.0 | 28.0 | 48.0 | 74.0 |
| CupToDrawerClose | 50.0 | 12.0 | 42.0 | 52.0 |
| MilkToMicrowaveClose | 52.0 | 20.0 | 38.0 | 50.0 |
| PotatoToMicrowaveClose | 22.0 | 28.0 | 18.0 | 36.0 |
Table 2: RoboCasa-GR1 tabletop tasks evaluation results (success rate (%)). Bold numbers indicate the highest performance in each category. from scratch implies that the model was trained without using any action data outside of the current benchmark. While the GR00T models are fine-tuned from their pre-trained weights, they are trained for the exact same number of steps as Qwen3DiT and DiT4DiT, which are trained from scratch.
Figure 5: Real-world evaluation results on the Unitree G1 robot . Success rates are reported across seven diverse household tasks. DiT4DiT comprehensively outperforms both the pre-trained GR00T-N1.5 ( Bjorck et al., 2025 ) and the parameter-matched Qwen3DiT baseline, highlighting the efficiency and efficacy of our framework.
Figure 6: Qualitative rollouts of real-world generalization tasks. We evaluate the policy’s zero-shot robustness against severe out-of-distribution physical variations. These include semantic and geometric shifts in Category (unseen cups and vases), complete object substitution in Object (packing corn instead of an eggplant), and scene clutter in Number (stacking four cups instead of three).
Figure 7: Quantitative results of zero-shot generalization. ( Left ) Success rates in the simulated RoboCasa-GR1 environment when evaluated on entirely unseen objects. ( Right ) Success rates on the real-world Unitree G1 robot across four challenging out-of-distribution scenarios, testing category variations, novel object substitutions, and distractor quantities. DiT4DiT demonstrates superior robustness and physical abstraction compared to both the parameter-matched Qwen3DiT baseline and the pre-trained GR00T-N1.5 ( Bjorck et al., 2025 ) .
Figure 8: Ablation studies on the DiT4DiT architecture. (a) Feature extraction layer: Success rate across different hidden layers of the video backbone, with performance peaking at layer 18. (b) Denoising steps: Impact of the number of iterative denoising steps used for action conditioning. A single forward step yields the highest success rate, preventing over-commitment to pixel-level reconstruction. (c) Representation learning: t-SNE visualization of latent features colored by the execution phase (Early, Middle, Late). Supported by a roughly twofold increase in the silhouette score ( Rousseeuw, 1987 ) , our joint training objective successfully induces smooth temporal flows within each task cluster, transitioning fluidly from the Early (blue) through the Middle (yellow) to the Late (red) phases.
| Trainable Param Cnt | Deploy Frequency | |
|---|---|---|
| GR00T-N1.5 | 2.7B | 13Hz |
| Qwen3DiT | 2.3B | 9Hz |
| DiT4DiT | 2.2B | 6Hz |
Table 3: Deployment efficiency comparison . We report the trainable parameter count and real-world deployment frequency (Hz) for DiT4DiT and the primary baselines. While the video backbone introduces a computational trade-off yielding a 6Hz control rate (tested on a single NVIDIA A100 GPU), DiT4DiT remains the most parameter-efficient model and comfortably supports real-time closed-loop physical execution.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Video DiT | |
| Base VGM | Cosmos-Predict2.5-2B ( Ali et al., 2025 ) |
| Attention Implementation | flash_attention_2 |
| Hidden Feature Dim | 2048 |
| Extract Layer | 18 |
| Action DiT | |
Table 4: Model & training configurations
| Dataset | Episode Cnt | Embodiment | DoF |
|---|---|---|---|
| Fourier_GR1_Unified_1K ( Bjorck et al., 2025 ) | 24,000 | GR1 humanoid | 29 |
| Fourier_GR1_Pretrain_10K ( Bjorck et al., 2025 ) | 241,450 | GR1 humanoid | 29 |
| LIBERO ( Liu et al., 2024 ) | 1,693 | Franka Emika Panda | 7 |
| Real Robot | 1,400 | G1 humanoid | 16 |
Table 5: Details of the used datasets . We report the episode count, embodiment type, and degrees of freedom.
Figure 9: Visualization of the robot system setups. The experimental platform consists of a Unitree G1 humanoid robot equipped with dual ALOHA 2 ( Aldaco et al., 2024 ) grippers and an Intel RealSense D435i camera. A human operator utilizes a PICO VR headset and the XRoboToolkit ( Zhao et al., 2025 ) framework to perform teleoperated demonstrations for high-quality data collection.
Figure 10: Future video rollouts generated by DiT4DiT.