CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization

Yinlong Dai; Homanga Bharadhwaj; Dylan P. Losey

cs.ROOct 8, 2026

CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization

Authors: Yinlong Dai, Homanga Bharadhwaj, Dylan P. Losey

Organizations: Collab, Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA. · B3, Dept. of Computer Science, Johns Hopkins University, Baltimore, MD.

Abstract

Prior research on learning from humans often requires real human demonstrations. But advances in video generation models offer a new paradigm: we can now prompt these generators to make synthetic videos of the task. On the plus side, these synthetic videos take place in the robot's current environment, and do not require actual human teaching. However, while the generated videos appear to show plausible motions, they lack both physical consistency and robot grounding, and thus the robot cannot directly roll out the trajectory shown by the video. In this paper we seek to formalize the process of going from synthetic video generation to real-world robot trajectories for zero-shot manipulation tasks. We recognize that this problem is an instance of constrained optimization. The object trajectories extracted from the synthetic video provide an initial guess at the correct motion, and the robot must refine these trajectories (for instance, within a digital twin) while enforcing physical feasibility and semantic consistency. Our optimization-based formalism provides insights into the video generation properties necessary for success. Specifically, we show that the video generation must probabilistically seed the search within a semantic basin of attraction where the trajectory can be locally modified without losing its task alignment. Our analysis further reveals that state-of-the-art approaches can be viewed as instances of our unified formalism with key simplifications. In our experiments we remove these simplifications to more closely match the underlying optimization, leading to more robust zero-shot trajectories from synthetic videos and improving average task success by 34.1% compared to the best-performing baseline. See videos at our project website: https://collab.me.vt.edu/CASTER

Figures & tables

Figure 1: Using video generation models at inference time to guide real-world robot trajectories. The generated video looks plausible, but may make mistakes that prevent directly transferring this video to the real robot’s motion. We bridge the gap between generation and deployment by developing an optimization-based formalism. Under this formalism the robot uses multiple synthetic videos to seed its search and identify task-critical features. By matching these features while enforcing semantic consistency and physical feasibility, we can transfer erroneous videos to successful zero-shot trajectories.

Figure 2: Overview of CASTER . (Top Left) Given an initial RGB-D observation and a task prompt, we (Top) generate synthetic videos of a human performing that task in the current environment and (Bottom Left) build a digital twin of the scene. From the synthetic videos we extract the object trajectories, which then serve as initial guesses for trajectory optimization within the digital twin. (Bottom) The trajectories extracted from synthetic videos do not need to be perfect: instead, they must lie within a semantic basin of attraction where the optimization can locally modify the trajectory shape without removing its semantic meaning. (Bottom Right) We perform constrained optimization so that the robot’s actions in the digital twin cause the objects to move similarly to their video trajectories, while enforcing kinematic constraints so that the output can be directly executed by the real robot.

Figure 3: Examples of our real-world tasks. (Left) Generated videos. (Middle) Digital twin rollouts. (Right) Real-world execution. The rows show four tasks: placing a banana in a pan, stacking bowls, pouring from a kettle, and toasting bread. See our supplemental video for more examples.

Task	Retarget	Mimic	CASTER
1. Move banana in pan	$0.0$	$66.7\pm 14.5$	$100.0\pm 0.0$
2. Pour cup	$42.9$	$52.4\pm 16.0$	$95.2\pm 4.8$
3. Place block in basket	$28.6$	$76.2\pm 14.0$	$100.0\pm 0.0$
4. Pour kettle	$57.1$	$66.7\pm 12.6$	$61.9\pm 8.7$
5. Plate orange	$57.1$	$61.9\pm 15.3$	$100.0\pm 0.0$
6. Pile up blocks	$0.0$	$9.5\pm 6.1$	$57.1\pm 20.2$

Table I: Zero-shot success (%) when performing real-world manipulation tasks. Mimic and CASTER report mean

\pm

standard error across seven video initialized CEM runs per task; Retarget reports success rate only.

Model	Valid	Success
HunyuanVideo-1.5 [ 35 ]	$55/80$	$79.8\pm 6.5$
Wan 2.2 I2V-A14B [ 37 ]	$71/80$	$76.2\pm 6.1$
LTX-2.3 22B Distilled [ 38 ]	$52/80$	$47.6\pm 8.6$
CogVideoX1.5 5B-I2V [ 39 ]	$1/80$	$0.0\pm 0.0$

Table II: Evaluating CASTER with different video generation models. Valid reports the number of task-aligned videos out of 80 sampled videos across four tasks. Success reports mean task success (%) across seven sampled videos per task.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Sampling steps	12
Guidance scale	1.0
Flow shift	7.0
Scheduler	Native flow matching, Euler solver
Resolution	480p, native image-aspect-ratio buckets
Video length / frame rate	121 frames / 24 fps

Appendix

Table III: HunyuanVideo inference settings.

Setting	Value
Physics / control frequency	120 Hz / 60 Hz
Physics steps per control command	2
Parallel environments	1050
Object mass	0.08 kg per object
Static / dynamic object friction	4.0 / 3.0
Object restitution	0

Appendix

Table IV: Simulation parameters.

Zero-Shot Robotic Manipulation

Zero-shot robotic manipulation enables robots to perform unfamiliar tasks and handle unseen objects, scenes, or embodiments without task-specific training.

See all 103 papers

Mar 5, 2026

EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation

Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout videos that capture rich object dynamics, offering a compelling foundation for zero-shot robotic manipulation. However, VGMs often produce physically implausible rollouts, and converting their pixel-space motion into robot actions through geometric retargeting further introduces cumulative errors from imperfect depth estimation and keypoint tracking. To address these challenges, we present EmboAlign, a data-free framework that aligns VGM outputs with compositional constraints generated by vision-language models (VLMs) at inference time. The key insight is that VLMs offer a capability complementary to VGMs: structured spatial reasoning that can identify the physical constraints critical to the success and safety of manipulation execution. Given a language instruction, EmboAlign uses a VLM to automatically extract a set of compositional constraints capturing task-specific requirements, which are then applied at two stages: (1) constraint-guided rollout selection, which scores and filters a batch of VGM rollouts to retain the most physically plausible candidate, and (2) constraint-based trajectory optimization, which uses the selected rollout as initialization and refines the robot trajectory under the same constraint set to correct retargeting errors. We evaluate EmboAlign on six real-robot manipulation tasks requiring precise, constraint-sensitive execution, improving the overall success rate by 43.3 percentage points over the strongest baseline without any task-specific training data.

Gehao Zhang, Zhenyang Ni, Payal Mohapatra +3

Northwestern University · Stanford University

Trajectory Optimization Physical Consistency in Video Generation Zero-Shot Robotic Manipulation Robotic Manipulation

Ask PDF

Jun 3, 2026

Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?

Video generation models have made impressive strides in synthesizing visually compelling content, yet their outputs remain confined to the virtual domain. A natural question follows: how well do these models reflect the physical world when their generated videos leave the screen and enter reality? We propose robotic manipulation as a concrete, measurable window onto this question: if a model has truly internalized physical laws, the motion it depicts should translate into executable robot behavior. We introduce Dream

.

exe, an evaluation framework that operationalizes this criterion through a video-to-execution pipeline. Given a scene image and a task description, Dream

.

exe synthesizes a manipulation video, converts the generated motion into robot trajectories, and executes them in a physics simulator, yielding a grounding signal that purely visual metrics cannot offer. Using this pipeline, we evaluate 8 models spanning frontier closed-source generators, open-source generators, and robot-specific models. Our benchmark covers 101 manually curated manipulation tasks at three levels of physical complexity, measured across visual quality, trajectory fidelity, and execution success. Encouragingly, several models achieve measurable execution success, suggesting that generative priors learned from internet-scale data already encode meaningful physical knowledge. Yet visual quality proves a poor predictor of executability, exposing a dimension of model capability that standard visual evaluations do not capture. Dream

.

exe will be open-sourced at https://github.com/showlab/Dream.exe.

Rui Zhao, Kaiming Yang, Jifeng Zhu +6

Show Lab, National University of Singapore · University of Oxford · Tencent.

Physical Consistency in Video Generation Robotic Manipulation Robot Manipulation Benchmarks Video Generation Evaluation

Ask PDF

Oct 7, 2026

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.

Huang Huang, Wensi Ai, Ziyu Chen +6

Stanford University · Nvidia · Microsoft Research

Sim-to-Real Transfer Synthetic Data Generation for Robotics Controllable Video Generation Zero-Shot Robotic Manipulation Robot Learning from Video

Ask PDF

Abstract

Zero-Shot Robotic Manipulation

Explore similar work

EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation

Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer