RobotAPO: Adversarial Physics Preference Optimization for Robotic Manipulation Video Generation

Kerui Li; Zhe Jing; Chenyi Huang; Xiaofeng Wang; Zheng Zhu; Haoming Cui; Huaibo Huang

cs.ROOct 7, 2026

RobotAPO: Adversarial Physics Preference Optimization for Robotic Manipulation Video Generation

Authors: Kerui Li, Zhe Jing, Chenyi Huang, Xiaofeng Wang, Zheng Zhu, Haoming Cui, Huaibo Huang

Organizations: Institute of Automation, Chinese Academy of Sciences · GigaAI · Beijing Institute of Technology

Abstract

Robotic manipulation videos are increasingly used as visual plans for embodied agents, but optimizing purely for visual plausibility often fails to capture the fragile physical manifold of real-world interactions. Even minor physics-violating errors at the interaction boundary, such as interpenetration or premature object motion, can completely invalidate the inferred timing and pose needed for downstream execution. Because standard supervised fine-tuning lacks the direct pressure to penalize these localized failures, we introduce AgiBot-PhysPref. This rigorously curated 10,000-sample preference dataset isolates condition-matched physics violations, turning the generator's own failure distribution into a foundational signal for physical consistency. Building upon this, we propose RobotAPO, an adversarial physics preference optimization framework operating in the continuous flow-matching denoising space. To prevent the policy from merely memorizing static curated failures, RobotAPO employs a lightweight adversarial counterfactual proposer that learns a condition-dependent, physical-failure-biased direction in denoising space. This encourages the model to explore and better respect the physical interaction boundary, all while maintaining a pure prompt-and-reference inference interface without requiring external structural conditioning. Comprehensive evaluations demonstrate that explicitly correcting these localized physics violations improves downstream robot execution from generated videos. On held-out AgiBot conditions, RobotAPO outperforms the strongest controlled internal baseline in physical consistency by 6.8% hard score and 10.0% soft score. Crucially, in real-robot replay, it translates these physical-consistency gains into a 37.4% relative improvement in task success over the strongest controlled internal baseline.

Figures & tables

\toprule Property	Value
\midrule Generated candidates	$40{,}000$
Human-reviewed candidates	$12{,}935$
Final curated preference samples	$10{,}000$
Task IDs	$200$ dual-arm AgiBot scenarios
Screening model	Qwen3-VL-30B-Instruct
Final label source	Human double annotation with disagreement review

Table 1: \texorpdfstring AgiBot-PhysPref AgiBot-PhysPref dataset statistics. The curated preference set isolates local physics violations via condition-matched pairs, human verification, and Qwen3-VL screening.

\toprule Method	Size	Visual-H $\uparrow$	Align-H $\uparrow$	Phys-H $\uparrow$	Visual-S $\uparrow$	Align-S $\uparrow$	Phys-S $\uparrow$
\midrule \textcolor grayLVP ( Chen et al., 2025 )	\textcolor gray14B	\textcolor gray3.647	\textcolor gray3.787	\textcolor gray3.473	\textcolor gray3.030	\textcolor gray3.252	\textcolor gray3.363
\midrule Wan2.2-TI2V	5B	3.287	3.493	2.967	2.812	3.064	2.891
SFT baseline	5B	3.347	3.767	3.193	2.984	3.410	3.041
Diffusion-DPO ( Wallace et al., 2024 )	5B	3.340	3.760	3.253	3.120	3.484	3.172
RealDPO ( Cheng et al., 2025 )	5B	3.347	3.760	3.173	3.047	3.435	3.008
\midrule \texorpdfstring RobotAPO RobotAPO (Ours)	5B	3.553	3.927	3.473	3.564	3.939	3.489

Table 2: Quantitative comparison on held-out AgiBot conditions. All internal methods are controlled 5B-parameter models. \texorpdfstring RobotAPO RobotAPO outperforms all controlled internal baselines; the external 14B LVP reference is shown in gray to contextualize absolute performance across a different model and training setup. Bold marks the best internal method.

\toprule Method	Size	Grasp Success (%) $\uparrow$	Cond. Place Success (%) $\uparrow$	Task Success (%) $\uparrow$	Task Success 95% CI
\midrule \textcolor grayLVP	\textcolor gray14B	\textcolor gray72.5	\textcolor gray86.5	\textcolor gray62.7	\textcolor gray[49.0, 74.7]
\midrule SFT baseline	5B	56.9	58.6	33.3	[22.0, 47.0]
Diffusion-DPO	5B	64.7	69.7	45.1	[32.3, 58.6]
RealDPO	5B	66.7	70.6	47.1	[34.1, 60.5]
\texorpdfstring RobotAPO RobotAPO (Ours)	5B	78.4	82.5	64.7	[51.0, 76.4]

Table 3: Real-robot pick-and-place execution success rates via AnyPos action extraction. \texorpdfstring RobotAPO RobotAPO improves downstream robot execution from generated videos, raising absolute task success by

31.4

percentage points over the SFT baseline. LVP is an external, unmatched reference. (Task success 95% CI uses the Wilson score interval over

51

trials).

\toprule Method	Visual-S $\uparrow$	Align-S $\uparrow$	Phys-S $\uparrow$	Grasp Success (%) $\uparrow$	Task Success (%) $\uparrow$
\midrule SFT baseline	2.984	3.410	3.041	56.9	33.3
\texorpdfstring RobotAPO RobotAPO w/o CF/Adv	3.194	3.614	3.134	68.6	49.0
\texorpdfstring RobotAPO RobotAPO w/o Anchor/Reg	3.000	3.326	3.079	60.8	35.3
\texorpdfstring RobotAPO RobotAPO (Ours)	3.564	3.939	3.489	78.4	64.7

Table 4: Module ablation of \texorpdfstring RobotAPO RobotAPO. Removing the adversarial counterfactual proposer (w/o CF/Adv) reduces real-robot task success from

64.7\%

49.0\%

, supporting the contribution of dynamic boundary exploration in this setting.

\toprule $\lambda_{\mathrm{adv}}$	Visual-H $\uparrow$	Align-H $\uparrow$	Phys-H $\uparrow$	Visual-S $\uparrow$	Align-S $\uparrow$	Phys-S $\uparrow$
\midrule $0.00$	3.407	3.793	3.300	3.284	3.704	3.270
$0.02$	3.460	3.853	3.373	3.392	3.784	3.351
$0.05^{\dagger}$	3.553	3.927	3.473	3.564	3.939	3.489
$0.10$	3.433	3.827	3.420	3.365	3.758	3.421

Table 5: Sensitivity study of the adversarial coupling coefficient

\lambda_{\mathrm{adv}}

. This coefficient controls the strength of the adversarial counterfactual term that perturbs the local physical interaction boundary during training. The dagger

\dagger

marks the default setting used in \texorpdfstring RobotAPO RobotAPO.

\toprule Method	Visual-H $\uparrow$	Visual-S $\uparrow$	Phys-H $\uparrow$	Phys-S $\uparrow$
\midrule SFT	3.347	2.984	3.193	3.041
Diffusion-DPO	3.240	3.211	3.185	3.061
RealDPO	3.225	3.212	3.180	3.092
\texorpdfstring RobotAPO RobotAPO (Ours)	3.355	3.345	3.310	3.293
\bottomrule

Table 6: Source-matched G+G control. Both sides of every training preference pair are generated by the same SFT model under the same condition.

\toprule Diagnostic	Adversarial	Gaussian	Paired effect [95% CI]
\midrule Condition-preserving local physical failure	37.5%	20.3%	$+17.2$ pp $[3.1,31.3]$
Global corruption / condition change	9.4%	6.3%	$+3.1$ pp $[-6.3,12.5]$
Unusable	3.1%	1.6%	$+1.6$ pp $[-3.1,6.3]$
\bottomrule

Table 7: Decoded adversarial counterfactuals compared with equal-norm Gaussian controls over 64 paired held-out conditions.

\toprule Method	Visual-S	Phys-H	Phys-S	Gain [95% CI]
\midrule SFT	$3.039\pm 0.199$	$2.980\pm 0.407$	$2.775\pm 0.626$	$+0.193$ $[0.088,0.316]$
Diffusion-DPO	$3.009\pm 0.104$	$3.010\pm 0.415$	$2.824\pm 0.547$	$+0.144$ $[0.058,0.245]$
RealDPO	$3.039\pm 0.198$	$2.970\pm 0.399$	$2.786\pm 0.607$	$+0.183$ $[0.081,0.299]$
\texorpdfstring RobotAPO RobotAPO	$3.008\pm 0.176$	$3.020\pm 0.410$	$2.968\pm 0.294$	–
\bottomrule

Table 8: CogVideoX-5B evaluation. Confidence intervals are condition-clustered paired bootstrap intervals for the RobotAPO Phys-S gain over each baseline.

\toprule Failure Type	Definition	Typical Evidence
\midrule Object interpenetration	Robot, object, or scene geometry passes through another entity.	Gripper penetrates object, or object sinks into table.
\addlinespace [2pt] Entity consistency error	Robot or object identity changes in appearance, shape, or existence over time.	Object flickers, disappears, changes shape, or changes identity across frames.
\addlinespace [2pt] Contact-causality error	Object motion or state transition is not causally supported by valid contact or manipulation, including premature motion before effective contact and action/state changes without effective contact.	Object moves before contact, state changes without effective contact, or gripper closes while the object slides without being grasped.
\addlinespace [2pt] Unsupported floating after release	An object remains suspended after losing support.	Released object stays in mid-air or does not settle naturally.
\addlinespace [2pt] Invalid interaction / other	No valid manipulation event under the task condition, or other visible failure not covered above.	Robot grasps air, never interacts with the target, or executes a misaligned motion that breaks the task.
\bottomrule

Table 9: Unified five-category failure taxonomy used for \texorpdfstring AgiBot-PhysPref AgiBot-PhysPref curation, training metadata, and failure-focused evaluation. The contact-causality category covers both premature object motion and action/state changes without effective contact.

Explore similar work

Jun 26, 2026cs.CV

PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation

Video generation models have emerged as a promising paradigm for embodied world simulation. However, both general-domain video generators and robot-specific data fine-tuned models can still produce physically implausible manipulations, including discontinuous motion trajectories and inconsistent robot-object interactions, which limits their reliability as world simulators. Through extensive experiments, we find that such physical instability mainly arises from two factors: deformation of moving objects and implausible spatio-temporal correlations among interacting entities, particularly during contact. Building on this observation, we propose PhysisForcing, a scalable training framework that strengthens physical consistency by focusing supervision on physics-informative regions through joint optimization of pixel-level and semantic-level features. The framework consists of a pixel-level trajectory alignment loss, which supervises DiT features using reference point trajectories, and a semantic-level relational alignment loss, which aligns DiT features with inter-region relations extracted from a frozen video understanding encoder. Extensive experiments on R-Bench, PAI-Bench, and EZS-Bench show that PhysisForcing consistently improves embodied video generation over strong baselines, improving the Wan2.2-I2V-A14B and Cosmos3-Nano base models on R-Bench by 22.3% and 9.2% (7.1% and 3.7% over vanilla finetuning), with the Cosmos3-Nano variant attaining the best overall score. Beyond generation, as a world model under the WorldArena action-planner protocol it raises the closed-loop success rate from 16.0% to 24.0% and further improves downstream policy success, indicating that physically aligned video models yield stronger representations for robotic manipulation.

Peiwen Zhang, Yufan Deng, Shangkun Sun +11

Peking University, NVIDIA

Physics Simulation Sim-To-Real Reinforcement Learning Human Motion Generation Robotic Manipulation Video Generation

Ask PDF

Jun 3, 2026cs.CV

Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?

Video generation models have made impressive strides in synthesizing visually compelling content, yet their outputs remain confined to the virtual domain. A natural question follows: how well do these models reflect the physical world when their generated videos leave the screen and enter reality? We propose robotic manipulation as a concrete, measurable window onto this question: if a model has truly internalized physical laws, the motion it depicts should translate into executable robot behavior. We introduce Dream

.

exe, an evaluation framework that operationalizes this criterion through a video-to-execution pipeline. Given a scene image and a task description, Dream

.

exe synthesizes a manipulation video, converts the generated motion into robot trajectories, and executes them in a physics simulator, yielding a grounding signal that purely visual metrics cannot offer. Using this pipeline, we evaluate 8 models spanning frontier closed-source generators, open-source generators, and robot-specific models. Our benchmark covers 101 manually curated manipulation tasks at three levels of physical complexity, measured across visual quality, trajectory fidelity, and execution success. Encouragingly, several models achieve measurable execution success, suggesting that generative priors learned from internet-scale data already encode meaningful physical knowledge. Yet visual quality proves a poor predictor of executability, exposing a dimension of model capability that standard visual evaluations do not capture. Dream

.

exe will be open-sourced at https://github.com/showlab/Dream.exe.

Rui Zhao, Kaiming Yang, Jifeng Zhu +6

Show Lab, National University of Singapore · University of Oxford · Tencent.

Generative Video Models Human Motion Generation Video Generation Robot Systems

Ask PDF

Jul 17, 2026cs.CV

PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation

Recent advances in physics-grounded video generation leverage physics simulation as a physical prior to guide video synthesis toward physically plausible outcomes. The simulation process is controlled by physical specifications, which are typically generated by a vision-language model in a single pass. Such one-shot prediction often fails to accurately translate user intent into executable simulations, particularly for fine-grained object dynamics, complex motion trajectories, and temporally structured interactions. In this paper, we propose PhysAgent, a reflective agentic framework that closes the loop among physical program generation, physics simulation, stage-specific verification, and targeted program repair. Beyond improving the control of coupled physical parameters, our framework enables the agent to progressively realize complex trajectories, multi-stage interactions, and precise event outcomes by treating each physical program as an executable hypothesis. In addition, we design a set of physics-control APIs to support more stable and complex motion behaviors. Extensive experiments demonstrate that PhysAgent produces more physically plausible videos, achieves better prompt alignment, and generalizes more effectively across diverse physical scenarios.

Qirui Li, Jinkun Hao, Yibo Li +3

Shanghai Jiao Tong University · Cardiff University

Physics Simulation Agentic Control Human Motion Generation Motion

Ask PDF