Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.
Figures & tables
Figure 1: Tapestry overview. The model is trained across video, joint video/action, and action-only diffusion. At deployment, it bypasses heavy video decoding to predict actions in a single step, delivering the speed and memory efficiency required for real-time robotic control.
Figure 2: The video-pretrained backbone has the wrong frequency prior for actions. A backbone trained to synthesize high-frequency video detail at low t keeps doing so when repurposed for action denoising, where the target distribution carries no such content. (a) Cause: spectral mismatch in the training distributions. f95 (vertical dashes) is the frequency below which 95% of a signal’s energy sits; action chunks ( f95≈0.35 Hz) carry far less high-frequency content than the natural video the backbone was pretrained on ( f95≈2.0 Hz). (b) Effect: high-frequency jitter on the action output. Per- t jitter rises as t→0 even under teacher-forcing with in-distribution noisy inputs (blue), and iterative DDIM (orange) compounds the per-step residuals into a much larger error. Single-step inference at t=T sidesteps the regime entirely (§ 3 ).
Table 1: LIBERO task generalization. Success rates (%) on four held-out LIBERO suites. The model is trained with action labels only from LIBERO-90 and video data from all suites until convergence. Each suite is evaluated over 100 rollouts.
Method
Toolbox
Oven
Corn
Total
Diffusion Policy [ 6 ]
3/10
4/10
8/10
15/30
Tapestry w/o video generation objective
5/10
9/10
5/10
19/30
Tapestry w/o large-scale video co-training
4/10
7/10
9/10
20/30
Tapestry (ours)
9/10
9/10
10/10
28/30
Table 2: Real-world in-distribution success. Cumulative success counts on the three trained tasks, 10 rollouts per task.
Method
Croissant on transparent door
Pour golf ball out of bowl
Lego brick on boomerang
Total
Partial
Full
Partial
Full
Partial
Full
Diffusion Policy [ 6 ]
0/5
0/5
1/5
0/5
1/5
0/5
0/15
DreamZero [ 31 ]
0/5
0/5
5/5
4/5
1/5
0/5
4/15
Tapestry w/o video generation objective
0/5
0/5
0/5
0/5
4/5
0/5
0/15
Tapestry w/o large-scale video co-training
0/5
0/5
2/5
0/5
3/5
0/5
0/15
Tapestry (ours)
3/5
1/5
5/5
4/5
5/5
5/5
10/15
Table 3: Real-world out-of-distribution generalization with partial and full success rates. Three OOD conditions probe novel object placement (croissant on transparent oven door), motion primitive (pour an unseen golf ball out of an unseen bowl), and object composition (place an unseen yellow Lego brick on an unseen wooden boomerang). Tasks additionally include unseen distractor objects in the scene and perturbed initial states. Partial credits the rollout for clearing the first subtask of each task (e.g., grasping the target object); Full requires the rollout to complete the entire task. Each condition is evaluated over 5 rollouts.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Subtask
BC
w/o lg-scale video
w/o video gen. obj.
Ours
Toolbox
Pick screwdriver
6
10
6
10
Place screwdriver
3
4
5
9
Oven
Open oven
10
9
9
10
Grasp croissant
6
7
9
9
Place croissant
4
7
9
9
Corn
Grab corn
8
9
5
10
Appendix
Table 4: Per-subtask cumulative success counts, in-distribution (out of 10 rollouts per cell). A failure at one subtask propagates as a failure for all subsequent subtasks in the same rollout.
Figure 3: Real-world bimanual setup: a 20-DoF bimanual YAM platform with wrist and third-person cameras, used for the experiments in § 4.2 .
Figure 4: Out-of-distribution evaluation scenes. Rollout snapshots for each of the three OOD conditions. Left: novel object placement: croissant on the transparent glass oven door rather than inside the oven. Center: novel motion primitive: pour an unseen golf ball from an unseen bowl. Right: novel object composition: place an unseen yellow Lego brick on an unseen wooden boomerang.
Condition
Subtask
BC
w/o lg-scale video
w/o video gen. obj.
Ours
Croissant on door
Pick up croissant
0
0
0
3
Place croissant on plate
0
0
0
1
Pour golf ball
Pick up bowl
1
2
0
5
Pour golf ball out
0
0
0
4
Lego on boomerang
Pick up Lego
1
3
4
5
Place Lego on boomerang
0
0
0
5
Appendix
Table 5: Per-subtask cumulative success counts, out-of-distribution (out of 5 rollouts per cell). Conditions: novel object placement (croissant on transparent oven door), novel motion primitive (pour golf ball out of bowl), novel object composition (place yellow Lego brick on a wooden boomerang).
Variant
Inference
Avg Success Rate
Pretraining strategy
From scratch (no pretrained CogVideoX init)
action
13
Pretrained init, action-only fine-tuning
action
0
Pretrained init, separate video and action objectives
action
36
Diffusion objective composition
Action-only training (no video loss)
action
0
Appendix
Table 6: Ablations on LIBERO task generalization. Average held-out success rate (%) over 10 rollouts per task ( 100 per suite). All variants trained for 5 epochs and share the LIBERO-90 action data and the evaluation protocol of Table 1 . At this budget the full recipe reaches 58% and the from-scratch variant 13% ; at convergence they reach 76% and 58% (Table 1 ). Inference : whether the evaluated model denoises actions alone or generates video jointly with actions.
Optimizer
AdamW
Learning rate
1×10−4 pretrain, 2×10−5 finetune (constant)
β1,β2
0.9, 0.95
Weight decay
1×10−4
Gradient clipping (max norm)
1
Batch
Per-GPU batch size
4
Appendix
Table 7: Tapestry training hyperparameters (CogVideoX-2B backbone).
Source
Type
Real hrs
Native FPS
Effective FPS
Training hrs
Batch weight
Cross-embodiment video pretraining (video objective only)
DROID [ 16 ]
robot
131.8
15
15
131.8
0.027
Ego4D [ 11 ]
human
32.1
30
30
32.1
0.020
Epic Kitchens [ 8 ]
human
6.1
60
60
6.1
0.014
SSv2 [ 10 ]
human
108.5
12
12
108.5
0.020
Panda-70M [ 5 ]
web
165.4
30
30
165.4
0.127
Appendix
Table 8: Training data mixture. Per-source wall-clock hours of footage (“real hrs”), source-native FPS, model-effective FPS, hours seen at training time after FPS adjustment, and per-batch sampling weight. Cross-embodiment video sources dominate raw hours but are throttled by sampling weight so that in-domain data drives each gradient step.
Figure 5: Variance of vθ(xt,t,c) across ϵ samples ( N=64 ). Per-coordinate mean variance for both vθ and the implied x^0 , plotted on a log scale across 10 scheduled t values. Reference bands show the spread of the empirical action distribution under matched conditioning. The variance at t=T (regime A) is ∼ 5 orders below the spread of the conditional action distribution; for t<T the model output is ϵ -coupled (regime B).
Figure 6: Teacher-forced vs. iterative jitter at each t , N=50 val chunks. Teacher-forced (blue): single forward pass at each t on in-distribution xt from GT. Iterative diamonds (orange): x^0(k) from a K=10 DDIM trajectory at t∈{999,499,99} . Gray band: GT chunk jitter ±1σ . Teacher-forced jitter rises monotonically but stays within ∼2× GT; iterative jitter grows to ∼8× GT, isolating exposure-bias compounding on top of the per- t output bias.
DDIM step k
t
TF ∥x^0jit∥ (mean ± std, 50 chunks)
Iterative ∥x^0(k)jit∥ (1 trajectory)
0
999
0.152±0.011
0.237
1
899
0.153±0.011
–
2
799
0.156±0.012
–
3
699
0.161±0.012
–
4
599
0.172±0.013
–
5
499
0.190±0.014
0.595
Appendix
Table 9: Teacher-forced sweep and iterative trajectory jitter at each scheduled t . Jitter is the per-chunk RMS of the second finite difference of x^0 across the 16 -step horizon. GT band: 0.166±0.027 .
Figure 7: Parameterization specificity at the noise terminus. 64-seed traces of vθ (top) and the implied x^0 (bottom) for the highest-variance action dim, with all other inputs held fixed. Left (blue): Tapestry (CogVideoX, v -pred + ZTS) at t=T ( αˉT=0 ): Varϵvθ=3.5×10−6 , traces collapse onto a single curve. Right (orange): Wan rectified flow at σmax=1 : Varϵvθ=1.17 , traces span the action range. Five-and-a-half orders of magnitude separate the two parameterizations on this same metric.
Figure 8: Wan rectified-flow analog of Fig. 6 . Teacher-forced single-step jitter across σ (blue) collapses with σ as x^0=xσ−σvθ . The iterative trajectory (orange diamonds) stays elevated throughout and converges to ∼ 59 × the GT band at σmin . Unlike CogVideoX, Wan has no σ at which the regression target structurally decouples from ϵ , so the corresponding gate around the mis-tuned low- σ frequency prior is unavailable.
Quantity
Action chunks
Natural video pixels
f50 (chunk horizon, 16 steps)
0.00 Hz
1.875 Hz
f95 (chunk horizon, 16 steps)
1.875 Hz
7.50 Hz
Frac. above 3.75 Hz (chunk horizon)
0.014
0.122
Frac. above 7.5 Hz (chunk horizon)
0.008
0.048
f95 (long-form, 256 steps)
0.352 Hz
1.992 Hz
Frac. in [0.75,1.5] Hz (long-form)
0.011
0.080
Appendix
Table 10: Spectral energy distribution: natural video (pixels) vs. action chunks. Unit-energy-normalized power, averaged per modality. “Frac. above f ” is the fraction of total energy above the listed frequency. Long-form bands from a 256 -step Hann window FFT; chunk-horizon f50 / f95 from a 16 -step rectangular FFT.
Vision-Language-Action (VLA) models have emerged as a promising paradigm for robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1 robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that video generation can serve as an effective scaling proxy for robot policy learning. We release code and models at https://dit4dit.github.io/.
Video generative models have emerged as a promising robotics backbone, capable of generating videos that depict the completion of complex tasks across embodiments and environments. Recent work proposes robot foundation models that jointly predict future observations and actions by finetuning video models with action-labeled data. In this paper, we test the limits of an alternative approach: leave the video planner as-is while training an embodiment-specific inverse dynamics model (IDM). This decoupling offers several natural benefits: the video planner remains embodiment-agnostic, different video models can be interchanged easily without re-training the IDM, and the IDM can be independently trained with readily available self-play data. We present a closed-loop, video-to-action policy that combines an action-free video world model with a carefully-designed IDM based on the robot embodiment Jacobian. We demonstrate that our IDM design is both data-efficient and scalable to high-dimensional action spaces. Our policy, which we coin the Video-to-Embodied Robot Action Model (VERA), achieves strong performance across simulated and real-world benchmarks, including zero-shot Panda arm manipulation and 16-DoF Allegro-hand dexterous cube re-orientation. The same video planner can be used across multiple embodiments by pairing it with different embodiment-specific IDMs. Our results show that decoupled video planning plus faithful video-to-action translation is a viable alternative route towards zero-shot, cross-embodiment, and generalizable robot control. More results are available on our project website: https://vera.csail.mit.edu.
World action models (WAMs) have emerged as a promising direction for robot policy learning, as they can leverage powerful video backbones to model the future states. However, existing approaches often rely on separate action modules, or use action representations that are not pixel-grounded, making it difficult to fully exploit the pretrained knowledge of video models and limiting transfer across viewpoints and environments. In this work, we present Action Images, a unified world action model that formulates policy learning as multiview video generation. Instead of encoding control as low-dimensional tokens, we translate 7-DoF robot actions into interpretable action images: multi-view action videos that are grounded in 2D pixels and explicitly track robot-arm motion. This pixel-grounded action representation allows the video backbone itself to act as a zero-shot policy, without a separate policy head or action module. Beyond control, the same unified model supports video-action joint generation, action-conditioned video generation, and action labeling under a shared representation. On RLBench and real-world evaluations, our model achieves the strongest zero-shot success rates and improves video-action joint generation quality over prior video-space world models, suggesting that interpretable action images are a promising route to policy learning.