Organizations: Sun Yat-sen University · X-Era AI Lab · Guangdong University of Technology · Guangdong Key Laboratory of Big Data Analysis and Processing
The scarcity of large-scale robotic data has motivated the repurposing of foundation models from other modalities for policy learning. In this work, we introduce PhysGen (Learning Physics from Pretrained Video Generation Models), a scalable continuous and sequential world interaction framework that leverages autoregressive video generation to solve robotic manipulation tasks. By treating the pretrained video model as a proxy for a physics simulator, PhysGen models the dynamic interplay between the external environment and robot actions. We introduce a multimodal continuous representation that unifies video and action into shared physical tokens, bridging the gap between discrete video generation and continuous robotic control. This approach enables the seamless transfer of implicit physical knowledge-such as object permanence and dynamics-from video pretraining to downstream manipulation.To ensure efficient convergence, we incorporate causal masking, inverse kinematics, Lookahead Multi-Token Prediction (L-MTP), and key-value (KV) caching. Experimental results on the Libero and ManiSkill benchmarks demonstrate that PhysGen consistently outperforms robust baselines, surpassing OpenVLA and WorldVLA by margins of 13.8% and 8.8%, respectively. Notably, in real-world scenarios, PhysGen matches the performance of large-scale action-pretrained models like π0 without requiring prior action-specific pretraining, demonstrating superior capability in physically complex tasks such as grasping transparent objects. These findings validate the potential of extracting physical intuition from pretrained video generators to facilitate generalizable robotic manipulation.
Figures & tables
Figure 1. The PhysGen Framework : Repurposing Video Generation as a World Simulator. Our approach unifies perception and control through Physical Autoregression . The model operates on a sequence of continuous physical tokens (red), which fuse visual context (orange) and embodiment actions (black) to predict their joint evolution. Acting as a predictive world model, PhysGen runs in synchronization with the physical environment (blue): at each step, predicted tokens are decoded into executed actions and anticipated video frames. The resulting environmental feedback is re-encoded into the stream, enabling a seamless continuous loop of planning and interaction.
Figure 2. The model architecture of PhysGen. Starting from text tokens, PhysGen leverages a Causal Transformer to autoregressively predict physical tokens. Notably, we apply a frame diffusion and an action diffusion to estimate the conditional distributions of visual and action signals in continuous space. The structure of the action diffusion network is shown on the right column, where the predicted token serves as the conditional vector and is injected through cross-attention.
Figure 3. The causal attention mask. Frame tokens use chunk-wise full attention; action tokens use temporal causal attention; action tokens unidirectionally attend to frame tokens for visually conditioned control.
Method
Parameters
Spatial
Object
Goal
Long
Average
Training from Scratch Baseline
DP ( 2023 ) (w/o act. pretraining)
–
78%
93%
68%
51%
72%
Vision-Language-Pretrained Baseline
Octo ( 2024 )
93M
79%
86%
85%
51%
75%
TraceVLA ( 2024a )
7B
85%
85%
75%
54
75%
OpenVLA ( 2024 )
7B
85%
88%
79%
54%
77%
Table 1. The success rate comparison in the Libero Benchmark. The best are highlighted by bold . ‘No action pretraining’ means that the model’s backbone is not pretrained on any large-scale manipulation datasets.
Method
PushCube
PickCube
StackCube
Avg.
ACT [ ( 2023 ) ]
76%
20%
30%
42%
BC-T [ ( 2021 ) ]
98%
4%
14%
39%
DP [ ( 2023 ) ]
88%
40%
80 %
69%
ICRT [ ( 2025 ) ]
77%
78 %
30%
62%
RDT [ ( 2024b ) ]
100 %
77%
74%
84 %
Pi0 [ ( 2024 ) ]
100 %
60%
48%
69%
Table 2. The success rate comparison in the ManiSkill Benchmark. The best are highlighted by bold .
Method
Pick Cube
Press Button
Stack Cube
Pick Transparency
Avg
ACT ( Zhao et al., 2023 )
40%
40%
30%
10%
30%
OpenVLA ( Kim et al., 2024 )
30%
25%
10%
0%
16.3%
Pi0 ( Black et al., 2024 )
85 %
85 %
60 %
70%
75 %
PhysGen (Ours)
80%
85 %
60 %
75 %
75 %
Table 3. Real world performance . We report the task success rates on four tabletop tasks. PhysGen achieves an average success rate comparable to Pi0.
Method
Pretrain
Token
L-MTP
Backbone
SR
PhysGen-Zero
no
cont.
+
AR
86.4%
PhysGen-Discrete
NOVA
discrete
+
AR
94.2%
PhysGen-NoAR
NOVA
cont.
-
NoAR
95.0%
PhysGen-STP
NOVA
cont.
-
AR
96.8%
PhysGen-Full
NOVA
cont.
+
AR
99.6%
Table 4. The ablation study on the Libero-Object task of the Libero Benchmark. SR denotes Success Rate; higher is better.
Figure 4. Video predictions and actual action executions. Each row shows PhysGen’s predicted video alongside the corresponding execution video for five different tasks. The strong visual similarity between predicted and actual action videos highlights the effectiveness of our approach in transferring knowledge from video pretraining.
Figure 5. Visualization of real world manipulation. We deploy our method on a Franka Panda robot across four real-world tasks: Pick Transparency , Pick Cube , Press Button , and Stack Cube . The robot executes stable and coherent manipulations across diverse settings, demonstrating effective knowledge transfer from video generation to real-world manipulation.
Figure 6. The attention map. The top row shows the token-level attention map, indicating how the predicted action attends to previous frame and action tokens. The bottom row shows the pixel-level attention map, indicating how the predicted action attends to different spatial regions of the frame.
Video generation models have emerged as a promising paradigm for embodied world simulation. However, both general-domain video generators and robot-specific data fine-tuned models can still produce physically implausible manipulations, including discontinuous motion trajectories and inconsistent robot-object interactions, which limits their reliability as world simulators. Through extensive experiments, we find that such physical instability mainly arises from two factors: deformation of moving objects and implausible spatio-temporal correlations among interacting entities, particularly during contact. Building on this observation, we propose PhysisForcing, a scalable training framework that strengthens physical consistency by focusing supervision on physics-informative regions through joint optimization of pixel-level and semantic-level features. The framework consists of a pixel-level trajectory alignment loss, which supervises DiT features using reference point trajectories, and a semantic-level relational alignment loss, which aligns DiT features with inter-region relations extracted from a frozen video understanding encoder. Extensive experiments on R-Bench, PAI-Bench, and EZS-Bench show that PhysisForcing consistently improves embodied video generation over strong baselines, improving the Wan2.2-I2V-A14B and Cosmos3-Nano base models on R-Bench by 22.3% and 9.2% (7.1% and 3.7% over vanilla finetuning), with the Cosmos3-Nano variant attaining the best overall score. Beyond generation, as a world model under the WorldArena action-planner protocol it raises the closed-loop success rate from 16.0% to 24.0% and further improves downstream policy success, indicating that physically aligned video models yield stronger representations for robotic manipulation.
Video-based world models offer a powerful paradigm for embodied simulation and planning, yet state-of-the-art models often generate physically implausible manipulations - such as object penetration and anti-gravity motion - due to training on generic visual data and likelihood-based objectives that ignore physical laws. We present ABot-PhysWorld, a 14B Diffusion Transformer model that generates visually realistic, physically plausible, and action-controllable videos. Built on a curated dataset of three million manipulation clips with physics-aware annotation, it uses a novel DPO-based post-training framework with decoupled discriminators to suppress unphysical behaviors while preserving visual quality. A parallel context block enables precise spatial action injection for cross-embodiment control. To better evaluate generalization, we introduce EZSbench, the first training-independent embodied zero-shot benchmark combining real and synthetic unseen robot-task-scene combinations. It employs a decoupled protocol to separately assess physical realism and action alignment. ABot-PhysWorld achieves new state-of-the-art performance on PBench and EZSbench, surpassing Veo 3.1 and Sora v2 Pro in physical plausibility and trajectory consistency. We will release EZSbench to promote standardized evaluation in embodied video generation.
Robotic manipulation videos are increasingly used as visual plans for embodied agents, but optimizing purely for visual plausibility often fails to capture the fragile physical manifold of real-world interactions. Even minor physics-violating errors at the interaction boundary, such as interpenetration or premature object motion, can completely invalidate the inferred timing and pose needed for downstream execution. Because standard supervised fine-tuning lacks the direct pressure to penalize these localized failures, we introduce AgiBot-PhysPref. This rigorously curated 10,000-sample preference dataset isolates condition-matched physics violations, turning the generator's own failure distribution into a foundational signal for physical consistency. Building upon this, we propose RobotAPO, an adversarial physics preference optimization framework operating in the continuous flow-matching denoising space. To prevent the policy from merely memorizing static curated failures, RobotAPO employs a lightweight adversarial counterfactual proposer that learns a condition-dependent, physical-failure-biased direction in denoising space. This encourages the model to explore and better respect the physical interaction boundary, all while maintaining a pure prompt-and-reference inference interface without requiring external structural conditioning. Comprehensive evaluations demonstrate that explicitly correcting these localized physics violations improves downstream robot execution from generated videos. On held-out AgiBot conditions, RobotAPO outperforms the strongest controlled internal baseline in physical consistency by 6.8% hard score and 10.0% soft score. Crucially, in real-robot replay, it translates these physical-consistency gains into a 37.4% relative improvement in task success over the strongest controlled internal baseline.
Kerui Li, Zhe Jing, Chenyi Huang +4
Institute of Automation, Chinese Academy of Sciences · GigaAI · Beijing Institute of Technology