VPTwin: Real-Sim-Real Video Prediction for Robotic Manipulation Planning

Zhenghao Xiao; Minting Pan; Nantian He; Dongzhan Zhou; Yunbo Wang

cs.ROSep 27, 2026

VPTwin: Real-Sim-Real Video Prediction for Robotic Manipulation Planning

Authors: Zhenghao Xiao, Minting Pan, Nantian He, Dongzhan Zhou, Yunbo Wang

Organizations: Carnegie Mellon University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University

Abstract

While action-conditioned video prediction provides an intuitive world model for robotics, purely data-driven predictors often suffer from compounding errors and physically implausible hallucinations in long-horizon rollouts, severely undermining downstream action planning. We propose VPTwin, a Real-Sim-Real video prediction framework that anchors real-world future prediction using real-synchronized simulation twins. For a target manipulation task, a VLM reconstructs an executable digital twin from a real demonstration episode. To accommodate the ill-posed estimation of unobserved physical properties, Isaac Sim simulates multiple forward dynamic rollouts across randomized physical configurations under candidate action trajectories. Using these rollouts as in-context references, VPTwin harmonizes both domains, using simulation dynamics to enforce physical plausibility while capturing unmodeled contact interactions from real video. Furthermore, we establish a predictive planning loop using VPTwin to visually verify VLM-proposed actions and guide reliable real-world execution. Evaluations show substantial reductions in physical hallucinations during video prediction and marked improvements in manipulation planning performance.

Figures & tables

Figure 1: Overview of VPTwin. (A) A VLM reconstructs executable digital-twin assets to drive randomized dynamic rollouts in Isaac Sim. (B) Conditioned on time-aligned simulation rollouts, VPTwin-Core synthesizes physically grounded real-domain future video under candidate actions. (C) Staged action planning evaluates and refines candidate actions via joint simulation and visual verification before real-world deployment.

Figure 2: Architecture of VPTwin-Core. Two-stage training and autoregressive inference conditioned on candidate actions and time-aligned simulation rollouts.

Measured-catalog (548), 6 families
Model	MSE $\downarrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	UIQI $\uparrow$
IWS ( 2026 )	$0.016\pm 0.008$	$0.329\pm 0.071$	$276.0\pm 32.1$	$2095.0\pm 384.4$	$25.58\pm 3.00$	$0.770\pm 0.069$	$0.249\pm 0.049$
VPTwin	$\mathbf{0.007\pm 0.002}$	$\mathbf{0.284\pm 0.056}$	$\mathbf{265.6\pm 36.9}$	$\mathbf{2014.5\pm 264.7}$	$\mathbf{28.57\pm 1.92}$	$\mathbf{0.798\pm 0.058}$	$\mathbf{0.283\pm 0.040}$
VLM-authored (277), 4 families
IWS ( 2026 )	$0.016\pm 0.009$	$0.310\pm 0.067$	$277.7\pm 45.3$	$2450.0\pm 701.8$	$25.64\pm 3.35$	$0.780\pm 0.073$	$0.262\pm 0.048$
VPTwin	$\mathbf{0.006\pm 0.002}$	$\mathbf{0.259\pm 0.048}$	$\mathbf{259.1\pm 42.3}$	$\mathbf{2131.0\pm 340.6}$	$\mathbf{28.96\pm 1.83}$	$\mathbf{0.809\pm 0.060}$	$\mathbf{0.294\pm 0.039}$

Table 1: Full-episode video prediction results (mean

\pm

std across tasks after averaging each family’s scores over five training seeds) Corresponding results for each task are presented in Table 14 .

Figure 3: Full-episode video prediction results at $0\%$ , $50\%$ , and $100\%$ of an execution timeline. Orange and green circles highlight baseline failure artifacts and our physically consistent predictions, respectively. $\Delta\mathrm{PSNR}$ denotes the quantitative gain of VPTwin over IWS ( Wang et al., 2026 ) .

	Grasp Cup				Open and Close Drawer
Method	Success	Precision	Recall	F1-score	Success	Precision	Recall	F1-score
$\pi_{0.5}$	0/7	na	na	na	0/5	na	na	na
SIMPACT	2/7	0.33	1.00	0.50	2/5	0.67	1.00	0.80
VPTwin	3/7	1.00	1.00	1.00	5/5	1.00	1.00	1.00

Table 2: Real-robot task success and critic evaluation metrics. Precision, Recall, and F1 assess critic verification against true physical outcomes: Precision denotes physical success rates among accepted plans, while Recall reflects the proportion of successful executions accepted. Rejected plans are additionally executed for evaluation.

Figure 4: Real-robot deployment trajectories. Columns represent sequential task phases. Orange and green circles highlight failure and success modes, respectively.

Figure 5: Visual prediction rollouts on failure episodes. Frame sequences across simulation ground truth, the success-only model, and the failure-aware co-trained model ( Failmix ) at $0\%$ , $50\%$ , and $100\%$ completion for (A) unachieved drawer closing and (B) unachieved drawer opening.

		30 frames		Full-episode
Start	Task	Failmix	Success-only	Failmix	Success-only
First frame	Grasp Cup	23.38	23.30	23.21	23.13
First frame	Open and Close Drawer	22.46	22.39	21.22	20.25
Perturbed frame	Grasp Cup	23.36	23.23	23.20	23.11
Perturbed frame	Open and Close Drawer	21.32	21.03	20.76	19.90

Table 3: Prediction results of failure episodes in PSNR (dB). Failmix incorporates synthetic failure co-training, while Success-Only relies solely on the

277

filtered success demonstrations. Scores are evaluated on

30

-frame prefixes and full rollouts.

Figure 6: Sensitivity to context quantity and outcome mismatch during inference. Columns distinguish digital-twin settings: (a, c) measured-catalog and (b, d) VLM-authored. (a, b) Full-test

\mathrm{PSNR}

as a function of the number of distinct simulation rollouts

k

cyclically repeated across the 10 conditioning slots (

k=0

corresponds to zero simulation context); values for

k\in\{1,3,5,7,9\}

report averaged results across three evaluation seeds. (c, d) Relative prediction gain under counterfactual, failure-only simulation context. Open markers denote sparse sample regimes (

n\leq 2

Train $\rightarrow$ Test	WipeTable	GraspCup	PickCup/Cloth	RemoveCloth	Open/CloseDrawer	PlaceCube	Mean
$3\rightarrow 3$	27.04	28.03	31.89	27.12	25.72	27.14	27.82
$10\rightarrow 3$	29.04	28.33	32.08	28.65	26.58	27.43	28.69

Table 4: Full-rollout PSNR (dB) with matched test context. Both models use seeds

0

–

2

, repeated to fill ten conditioning slots for the ten-rollout model. Rows denote training

\rightarrow

test rollout counts.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Real-robot experimental setup. (A) Bimanual platform, camera, and five-point placement grid. (B) Workspace dimensions and base-center coordinates. Each slot is marked by a $2.5$ cm-diameter red sticker; objects cover the marker without requiring perfect center alignment.

Task	Placement	Catalog reference	VLM suggested
Wipe Table	Folded blue towel on one of the five markers.	Pack $15\times 15\times 2$ cm.	Pack $16\times 16\times 1.8$ cm.
Grasp Cup	Upright cup on one of the five markers. Catalog episodes use a green or blue cup; VLM episodes use the green cup.	Height $10$ cm; bottom / top radius $2.5$ / $3.5$ cm; $40$ g.	Height $10$ cm; bottom / top radius $2.5$ / $3.48$ cm; $45$ g.
Pickup Cup and Cloth	Cloth covers the cup. Both sit on the left, center, or right marker.	Sheet $50\times 53$ cm, $1.0$ mm thick, $4$ g.	Sheet $52.8\times 50.4$ cm, $0.8$ mm thick, $4$ g.
Remove Cloth	Cloth covers the cup. Both sit on the left, center, or right marker.	Same sheet as Pickup Cup and Cloth.	Same VLM sheet and VLM cup. This family is catalog-only.
Open and Close Drawer	Fixed on the center marker.	$20\times 20\times 7$ cm; travel $15$ cm.	$20\times 20\times 7.3$ cm; cavity depth $6.0$ cm; travel $15.5$ cm.
Place Cube into Drawer	Fixed at the center; blue or red cube on the left or right marker.	Cube $5$ cm, $50$ g.	Cube $5$ cm, $75$ g. This family is catalog-only.

Appendix

Table 5: Placement and object size for the six task families. The five markers are the table center and four axial offsets of

20

cm. Catalog object dimensions were human-checked; listed masses are nominal coding-agent settings, not physical measurements. Only Grasp Cup and Open and Close Drawer are executed in the real-robot comparison. The VLM-authored set is Wipe Table, green-cup Grasp Cup, Pickup Cup and Cloth, and Open and Close Drawer.

Parameter	Mild	Mid	Harsh
Towel mass (g)	$30$ – $50$	$22$ – $65$	$\{20,80\}$
Towel friction	$0.8$ – $1.2$	$0.55$ – $1.4$	$\{0.40,1.60\}$
Cup mass (g)	$25$ – $48$	$15$ – $55$	$\{10,60\}$
Cup friction	$0.9$ – $1.5$	$0.6$ – $1.8$	$\{0.40,2.00\}$
Cloth mass (g)	$3$ – $6$	$2.5$ – $8$	$\{2,10\}$
Cloth friction	$1.4$ – $2.0$	$1.0$ – $2.1$	$\{0.60,2.20\}$

Appendix

Table 6: Measured-catalog parameter randomization. Severity levels correspond to seeds

1

–

3

4

–

6

, and

7

–

9

. Friction coefficients are dimensionless. Ranges suggested by coding agent.

Parameter	Suggested	Suggested range
Towel width $x$ (cm)	$16$	$14$ – $18.5$
Towel depth $y$ (cm)	$16$	$14$ – $19$
Towel pack height (cm)	$1.8$	$1.2$ – $2.6$
Towel density (kg/m 3 )	$90$	$10$ – $210$
Towel mass (g)	$41.47$	$1.47$ – $101.47$
Towel friction	$1.05$	$0.65$ – $1.25$

Appendix

Table 7: VLM-suggested parameter ranges. Each interval is

[\mathrm{suggested}-\mathrm{minus},\,\mathrm{suggested}+\mathrm{plus}]

from the VLM range file. The green and blue cups share one range set; the red and blue cubes share another.

Figure 8: VLM scene authoring and simulation-data construction. (A) A VLM authors digital-twin assets and parameter ranges from real demonstrations. (B) VLM-derived success criteria retain 277 seed- $0$ rollouts. (C) Retained episodes are randomized, time-aligned, and paired with real RGB and actions for video-predictor training.

	Measured-catalog			VLM-authored
Task	Train	Val	Test	Train	Val	Test
Wipe Table	86	10	10	62	8	8
Grasp Cup	103	13	13	24	5	3
Pickup Cup and Cloth	85	10	10	68	5	8
Remove Cloth	35	5	5	–	–	–
Open and Close Drawer	84	10	10	70	8	8

Appendix

Table 8: Episode-level train/validation/test splits. The VLM-authored set inherits the measured-catalog assignments.

Task	Instruction	Success Condition
Remove Cloth	Remove a white cloth from over a cup while leaving the cup standing on the table. The arm grips and gathers the fabric beside the covered form, then pulls or lifts the cloth clear rather than carrying away the covered object.	This kind is successful when the arm removes the covering cloth completely, exposing the green cup and leaving it standing on its base near the previously covered location. The cloth must no longer cover or hang over the cup at the end, and the cup must remain on the table rather than leave with the cloth. The final view should show the uncovered cup, the surrounding bare tabletop, and the unchanged red markers after the arm withdraws.
Place Cube into Drawer	Open a white drawer using its yellow front knob, pick up the cube from the table, place it inside the open compartment, and close the drawer.	This kind is successful when the drawer is pulled open far enough to receive the cube, the arm transfers the cube from the tabletop into the compartment, and the drawer is then pushed fully closed. The cube should no longer be at its original outside location or sitting on the unit’s top. The final view should show the closed drawer unit standing in place, with the cube enclosed following the visible placement sequence and the surrounding table markers undisturbed.
Open and Close Drawer	Pull the initially closed drawer outward to expose its empty compartment, then push it closed again without transferring another object.	This kind is successful when the arm produces a visible opening of the drawer, revealing the empty compartment, and subsequently returns the drawer front to its closed position against the housing. No object needs to be added or removed. At the end, the unit should remain standing in approximately its original place, the drawer should no longer be extended, and the surrounding tabletop should be unchanged.
Pickup Cup and Cloth	Grasp the raised portion of a white cloth-covered bundle and lift the entire bundle away, leaving no uncovered object behind. Its concealed contents are not separately revealed in these photographs.	This kind is successful when the arm takes hold of the cloth-covered raised portion and removes the whole bundle from the tabletop, including the trailing fabric. The former covered area should be clear at the end, with no cup or other loose object left standing there and no cloth remaining on the table. The visible endpoint is a bare tabletop apart from its stationary red markers; identifying the concealed contents is not necessary to judge that outcome.
Wipe Table	Press a blue cloth against the table with the gripper held nearly horizontal and slide it through short local strokes, then lift the arm away while leaving the cloth on the table.	This kind is successful when the arm maintains contact that presses the blue cloth against the tabletop and moves the cloth through a visible short wiping stroke or back-and-forth motion. It should then withdraw, leaving the cloth resting on the table in roughly the same local area rather than carrying it away. Small shifts, changes in outline, and rearranged folded edges are consistent with the photographs, while the fixed markers remain undisturbed. The supported success criterion is the contact-and-sliding action, not a claim that unseen dirt has been removed.
Grasp Cup	Grasp a standing cup, lift it off the table, and set it down near its original position.	This kind is successful when the gripper takes hold of the cup, lifts it clear of the tabletop, and subsequently returns it to a stable standing position near where it began. The cup may tilt while held or temporarily leave the image, but it should finish upright on its base after the gripper releases and withdraws. The final table arrangement should remain essentially the starting arrangement, apart from a small placement difference, with the red markers unchanged.

Appendix

Table 9: Definition of tasks. For each manipulation task, we list the corresponding instruction and success criteria. Instruction and success conditions are GPT-6 Astra text authored from multi-time real-video stills. The Grasp Cup row describes the return-near-start variant retained in the VLM-authored set; the measured-catalog set also includes cup-relocation variants.

Measured-catalog setting
Task family	$n$	$N_{\mathrm{frame}}$	$N_{\mathrm{clip}}$
Wipe Table	10	1369	166
Grasp Cup	13	3346	412
Pickup Cup and Cloth	10	1207	146
Remove Cloth	5	713	87
Open and Close Drawer	10	2463	302

Appendix

Table 10: Per-family sample counts for FID and FVD.

n

denotes test episodes,

N_{\mathrm{frame}}

the number of frames used for FID, and

N_{\mathrm{clip}}

the number of clips used for FVD. Counts are identical for Baseline, VPTwin, and ground truth.

Measured-catalog (548), 6 families
Metric	$\mathbf{k{=}0}$	$\mathbf{k{=}1}$	$\mathbf{k{=}5}$	$\mathbf{k{=}10}$
MSE $\downarrow$	$0.101\pm 0.048$	$0.007\pm 0.002$	$0.007\pm 0.002$	$0.007\pm 0.002$
LPIPS $\downarrow$	$0.579\pm 0.030$	$0.289\pm 0.055$	$0.286\pm 0.055$	$0.284\pm 0.056$
FID $\downarrow$	$363.3\pm 18.6$	$267.8\pm 36.5$	$266.6\pm 37.5$	$265.6\pm 36.9$
PSNR $\uparrow$	$16.71\pm 2.74$	$28.36\pm 1.91$	$28.51\pm 1.89$	$28.57\pm 1.92$
SSIM $\uparrow$	$0.547\pm 0.110$	$0.796\pm 0.057$	$0.797\pm 0.056$	$0.798\pm 0.058$

Appendix

Table 11: Dest-count ablation, family-averaged (mean

\pm

std over task families). The input retains ten conditioning slots. For

k>0

the

k

sampled dests are cycled into every slot (no black frames);

k{=}0

is all zeros.

k\in\{1,5\}

averages three evaluation seeds, while

k=10

uses all ten rollouts.

		$\mathbf{k{=}1}$			$\mathbf{k{=}3}$
Family	Metric	B	C	$\bm{\Delta}$	B	C	$\bm{\Delta}$
Pickup Cup and Cloth $n{=}6$	MSE $\downarrow$	$0.0040$	$\mathbf{0.0036}$	$-0.0004$	$0.0042$	$\mathbf{0.0036}$	$-0.0006$
	LPIPS $\downarrow$	$\mathbf{0.260}$	$0.277$	$+0.017$	$\mathbf{0.281}$	$0.293$	$+0.012$
	FID $\downarrow$	$\mathbf{242.5}$	$242.8$	$+0.3$	$260.2$	$\mathbf{254.4}$	$-5.8$
	PSNR $\uparrow$	$30.28$	$\mathbf{30.68}$	$+0.40$	$30.05$	$\mathbf{30.82}$	$+0.77$
	SSIM $\uparrow$	$\mathbf{0.865}$	$0.862$	$-0.003$	$0.864$	$0.864$	$0.000$

Appendix

Table 12: Fail-dest ablation on the measured catalog (548). Concat receives the

k

lowest-index failed dests, cycled into all 10 slots. Baseline is the action-only model on the same episodes.

k{=}3

is marked ∗ when

n\leq 2

. Bold is the better of baseline and concat.

\Delta

is concat minus baseline.

n

is the

k{=}1

episode count.

		$\mathbf{k{=}1}$			$\mathbf{k{=}3}$
Family	Metric	B	C	$\bm{\Delta}$	B	C	$\bm{\Delta}$
Pickup Cup and Cloth $n{=}5$	MSE $\downarrow$	$\mathbf{0.0044}$	$0.0052$	$+0.0008$	$0.0059^{*}$	$\mathbf{0.0054}^{*}$	$-0.0005^{*}$
	LPIPS $\downarrow$	$\mathbf{0.252}$	$0.339$	$+0.087$	$\mathbf{0.264}^{*}$	$0.307^{*}$	$+0.043^{*}$
	FID $\downarrow$	$\mathbf{228.5}$	$253.1$	$+24.6$	$278.5^{*}$	$\mathbf{271.5}^{*}$	$-7.0^{*}$
	PSNR $\uparrow$	$\mathbf{29.82}$	$29.00$	$-0.81$	$28.29^{*}$	$\mathbf{28.69}^{*}$	$+0.41^{*}$
	SSIM $\uparrow$	$\mathbf{0.857}$	$0.846$	$-0.011$	$0.830^{*}$	$\mathbf{0.831}^{*}$	$+0.001^{*}$

Appendix

Table 13: Fail-dest ablation on the VLM-authored set (277). Same protocol as Table 12 . Grasp Cup is a single episode at

k{=}1

. Wipe Table has no episode with three failures. Every

k{=}3

cell that exists is one episode ( ∗ ).

Measured-catalog (548), six task families
Task family	$\bm{n}$	MSE $\downarrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	UIQI $\uparrow$
Wipe Table	10	$0.0137\pm 0.0153$ / $\mathbf{0.0063\pm 0.0050}$	$0.252\pm 0.037$ / $\mathbf{0.213\pm 0.014}$	$250.3$ / $\mathbf{230.6}$	$2332.2$ / $\mathbf{2194.8}$	$26.25\pm 3.46$ / $\mathbf{28.93\pm 2.80}$	$0.812\pm 0.051$ / $\mathbf{0.847\pm 0.026}$	$0.296\pm 0.033$ / $\mathbf{0.336\pm 0.016}$
Grasp Cup	13	$0.0239\pm 0.0204$ / $\mathbf{0.0067\pm 0.0032}$	$0.349\pm 0.051$ / $\mathbf{0.277\pm 0.021}$	$278.5$ / $\mathbf{254.2}$	$2644.5$ / $\mathbf{2396.9}$	$23.54\pm 3.66$ / $\mathbf{28.26\pm 2.33}$	$0.747\pm 0.048$ / $\mathbf{0.794\pm 0.019}$	$0.216\pm 0.029$ / $\mathbf{0.270\pm 0.013}$
Pickup Cup and Cloth	10	$0.0036\pm 0.0012$ / $\mathbf{0.0026\pm 0.0010}$	$0.237\pm 0.049$ / $\mathbf{0.223\pm 0.032}$	$225.0$ / $\mathbf{216.5}$	$\mathbf{1689.7}$ / $1700.6$	$30.73\pm 1.40$ / $\mathbf{32.11\pm 1.56}$	$0.872\pm 0.019$ / $\mathbf{0.876\pm 0.021}$	$0.324\pm 0.038$ / $\mathbf{0.329\pm 0.034}$
Remove Cloth	5	$0.0132\pm 0.0162$ / $\mathbf{0.0080\pm 0.0066}$	$0.337\pm 0.076$ / $\mathbf{0.310\pm 0.038}$	$306.0$ / $\mathbf{295.4}$	$\mathbf{1633.7}$ / $1748.7$	$26.68\pm 4.06$ / $\mathbf{28.20\pm 3.65}$	$0.793\pm 0.061$ / $\mathbf{0.802\pm 0.049}$	$0.234\pm 0.046$ / $\mathbf{0.240\pm 0.039}$
Open and Close Drawer	10	$0.0243\pm 0.0092$ / $\mathbf{0.0090\pm 0.0016}$	$0.394\pm 0.028$ / $\mathbf{0.337\pm 0.010}$	$\mathbf{299.0}$ / $300.9$	$2147.3$ / $\mathbf{1988.5}$	$22.48\pm 1.78$ / $\mathbf{26.56\pm 0.75}$	$0.687\pm 0.021$ / $\mathbf{0.728\pm 0.010}$	$0.206\pm 0.019$ / $\mathbf{0.255\pm 0.009}$

Appendix

Table 14: Per-family full-episode scores. Each cell is Baseline /- Ours. Baseline is action-only VPTwin; Ours is simulation-conditioned VPTwin (10 dests). Pixel metrics are mean

\pm

std over test episodes. Higher PSNR, SSIM, and UIQI are better (

\uparrow

); lower MSE, LPIPS, FID, and FVD are better (

\downarrow

). The better value is bolded. These family scores are the entries averaged in Table 1 .

		30 frames		Full rollout
Start	Task	Failmix	Success-only	Failmix	Success-only
First frame	Grasp Cup	0.714	0.704	0.707	0.700
First frame	Open and Close Drawer	0.674	0.637	0.624	0.579
Perturbed frame	Grasp Cup	0.713	0.703	0.707	0.700
Perturbed frame	Open and Close Drawer	0.620	0.590	0.605	0.558

Appendix

Table 15: Failure-rollout SSIM on the same 11 test pairs as Table 3 . Failmix uses success–failure co-training; Success-only uses the 277 successful VLM-filtered episodes. Scores use a 30-frame prefix or the full rollout from the indicated start frame. Higher is better.

	MSE $\downarrow$		SSIM $\uparrow$		LPIPS $\downarrow$		UIQI $\uparrow$
Task	3 $\rightarrow$ 3	10 $\rightarrow$ 3	3 $\rightarrow$ 3	10 $\rightarrow$ 3	3 $\rightarrow$ 3	10 $\rightarrow$ 3	3 $\rightarrow$ 3	10 $\rightarrow$ 3
Wipe Table	0.0124	0.0060	0.823	0.847	0.197	0.173	0.312	0.335
Grasp Cup	0.0073	0.0067	0.792	0.794	0.244	0.241	0.266	0.269
Pickup Cup and Cloth	0.0028	0.0026	0.876	0.876	0.173	0.174	0.329	0.329
Remove Cloth	0.0117	0.0068	0.800	0.806	0.261	0.252	0.248	0.240
Open and Close Drawer	0.0112	0.0090	0.723	0.730	0.314	0.304	0.247	0.255

Appendix

Table 16: Full-rollout MSE, SSIM, LPIPS, and UIQI for Table 4 . Same episodes and the same two settings.

	FID $\downarrow$		FVD $\downarrow$
Task	3 $\rightarrow$ 3	10 $\rightarrow$ 3	3 $\rightarrow$ 3	10 $\rightarrow$ 3
Wipe Table	231.90	230.12	2182.2	2136.6
Grasp Cup	255.10	255.43	2431.9	2391.0
Pickup Cup and Cloth	220.18	217.52	1709.4	1693.5
Remove Cloth	296.15	292.55	1653.2	1781.9
Open and Close Drawer	309.82	307.23	2087.1	2011.7

Appendix

Table 17: Pooled FID and FVD for Table 4 . FID uses every predicted frame (Inception-2048). FVD uses 16-frame clips with stride 8. Both are one score per task, not a mean of per-episode scores. Lower is better.

Action-Conditioned Video Generation

Action-conditioned video generation creates videos whose motion and scene changes are guided by controls, actions, or interaction signals.

See all 111 papers

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
Sep 29, 2026Shenghe Zheng, Wenbo Li, Jiyao Zhang +4Action-Conditioned Video Generation World Models for Robotics
The Hong Kong University of Science and Technology · Joy Future Academy · Peking University +1
Video Prediction Policy 2: Predict Better, Act Better
Oct 7, 2026Yanjiang Guo, Haodong Yan, Zhide Zhong +15Visuomotor Policy Learning World Action Models
Robotera · Tsinghua University · HKUST (GZ) +2
PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation
Oct 12, 2026Xinyu Guo, Chao Jing, Li Yi +2Robotic Manipulation Vision-Language-Action Models
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhongguancun Academy +2

Sep 29, 2026J/K move · Enter open · S save

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

Shenghe Zheng, Wenbo Li, Jiyao Zhang, Bin Xia+3

The Hong Kong University of Science and Technology · Joy Future Academy · Peking University+1

Action-Conditioned Video Generation World Models for Robotics Robotics Simulation Robot Dynamics Learning Robotic Policy Evaluation Cross-Embodiment Robot Learning+1

Open paper AskPDF arXiv

Real-world robot learning is constrained by the cost of collecting experience and evaluating candidate behaviors. Video generation models offer a scalable foundation for visual simulators that predict action outcomes before physical execution. Yet they often favor visual plausibility over accurate action following and coherent robot--object dynamics, while action-conditioned simulators depend on scarce, embodiment-specific data that are difficult to share across incompatible control spaces. We introduce WorldLine, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding. WorldLine learns manipulation dynamics from more than 10,000 hours of action-free robot videos and grounds them using over 2,000 hours of action trajectories across more than ten embodiments. An image-space action representation provides a shared control interface across embodiments, while multi-view and failure-enriched training with relational regularization improves interaction-sensitive prediction. Robot-focused few-step distillation enables efficient causal rollout while preserving action-critical motion. Across held-out and out-of-domain settings, WorldLine maintains strong visual quality and robot-motion agreement; on failed trajectories, it improves robot-mask IoU by 0.1626 over the strongest baseline. It predicts trajectory success with 74% mean accuracy across RoboTwin and AgiBot, one percentage point above the strongest baseline. Without RoboTwin training or adaptation, its rollouts improve task success by up to 21.4 percentage points over direct policy execution. Together, these capabilities make WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning. More results are available at project page.

Abstract

Action-Conditioned Video Generation

Explore similar work

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

Video Prediction Policy 2: Predict Better, Act Better

PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation