TAPESTRY: A Single Backbone for Video Generation and Robot Control

Jeremy A. Collins; Seungjae Lee; Rachanon Wachakorn; Krishnan Srinivasan; Animesh Garg; Vitor Campagnolo Guizilini; Paarth Shah

cs.ROOct 27, 2026

TAPESTRY: A Single Backbone for Video Generation and Robot Control

Authors: Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn, Krishnan Srinivasan, Animesh Garg, Vitor Campagnolo Guizilini, Paarth Shah

Organizations: Georgia Institute of Technology. · Toyota Research Institute. · University of Maryland.

Abstract

Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.

Figures & tables

Figure 1: Tapestry overview. The model is trained across video, joint video/action, and action-only diffusion. At deployment, it bypasses heavy video decoding to predict actions in a single step, delivering the speed and memory efficiency required for real-time robotic control.

Panel of Figure 2 — Figure 2: The video-pretrained backbone has the wrong frequency prior for actions. A backbone trained to synthesize high-frequency video detail at low $t$ keeps doing so when repurposed for action denoising, where the target distribution carries no such content. (a) Cause: spectral mismatch in the training distributions. $f_{95}$ (vertical dashes) is the frequency below which $95\%$ of a signal’s energy sits; action chunks ( $f_{95}\approx 0.35$ Hz) carry far less high-frequency content than the natural video the backbone was pretrained on ( $f_{95}\approx 2.0$ Hz). (b) Effect: high-frequency jitter on the action output. Per- $t$ jitter rises as $t\to 0$ even under teacher-forcing with in-distribution noisy inputs (blue), and iterative DDIM (orange) compounds the per-step residuals into a much larger error. Single-step inference at $t=T$ sidesteps the regime entirely (§ 3 ).

Method	LIBERO-10	Object	Spatial	Goal	Avg
Behavior-cloning baselines (LIBERO-90 action labels only)
QueST [ 22 ]	7	0	1	1	2
BAKU [ 12 ]	6	0	0	0	2
Diffusion Policy [ 6 ]	0	0	0	0	0
Learning-from-video methods (LIBERO-90 actions + held-out videos)
AMPLIFY [ 7 ]	52	80	69	41	61

Table 1: LIBERO task generalization. Success rates (%) on four held-out LIBERO suites. The model is trained with action labels only from LIBERO-90 and video data from all suites until convergence. Each suite is evaluated over

100

rollouts.

Method	Toolbox	Oven	Corn	Total
Diffusion Policy [ 6 ]	3/10	4/10	8/10	15/30
Tapestry w/o video generation objective	5/10	9/10	5/10	19/30
Tapestry w/o large-scale video co-training	4/10	7/10	9/10	20/30
Tapestry (ours)	9/10	9/10	10/10	28/30

Table 2: Real-world in-distribution success. Cumulative success counts on the three trained tasks, 10 rollouts per task.

Method	Croissant on transparent door		Pour golf ball out of bowl		Lego brick on boomerang		Total
	Partial	Full	Partial	Full	Partial	Full
Diffusion Policy [ 6 ]	0/5	0/5	1/5	0/5	1/5	0/5	0/15
DreamZero [ 31 ]	0/5	0/5	5/5	4/5	1/5	0/5	4/15
Tapestry w/o video generation objective	0/5	0/5	0/5	0/5	4/5	0/5	0/15
Tapestry w/o large-scale video co-training	0/5	0/5	2/5	0/5	3/5	0/5	0/15
Tapestry (ours)	3/5	1/5	5/5	4/5	5/5	5/5	10/15

Table 3: Real-world out-of-distribution generalization with partial and full success rates. Three OOD conditions probe novel object placement (croissant on transparent oven door), motion primitive (pour an unseen golf ball out of an unseen bowl), and object composition (place an unseen yellow Lego brick on an unseen wooden boomerang). Tasks additionally include unseen distractor objects in the scene and perturbed initial states. Partial credits the rollout for clearing the first subtask of each task (e.g., grasping the target object); Full requires the rollout to complete the entire task. Each condition is evaluated over

5

rollouts.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Subtask	BC	w/o lg-scale video	w/o video gen. obj.	Ours
Toolbox	Pick screwdriver	6	10	6	10
	Place screwdriver	3	4	5	9
Oven	Open oven	10	9	9	10
	Grasp croissant	6	7	9	9
	Place croissant	4	7	9	9
Corn	Grab corn	8	9	5	10

Appendix

Table 4: Per-subtask cumulative success counts, in-distribution (out of 10 rollouts per cell). A failure at one subtask propagates as a failure for all subsequent subtasks in the same rollout.

Figure 3: Real-world bimanual setup: a 20-DoF bimanual YAM platform with wrist and third-person cameras, used for the experiments in § 4.2 .

Figure 4: Out-of-distribution evaluation scenes. Rollout snapshots for each of the three OOD conditions. Left: novel object placement: croissant on the transparent glass oven door rather than inside the oven. Center: novel motion primitive: pour an unseen golf ball from an unseen bowl. Right: novel object composition: place an unseen yellow Lego brick on an unseen wooden boomerang.

Condition	Subtask	BC	w/o lg-scale video	w/o video gen. obj.	Ours
Croissant on door	Pick up croissant	0	0	0	3
	Place croissant on plate	0	0	0	1
Pour golf ball	Pick up bowl	1	2	0	5
	Pour golf ball out	0	0	0	4
Lego on boomerang	Pick up Lego	1	3	4	5
	Place Lego on boomerang	0	0	0	5

Appendix

Table 5: Per-subtask cumulative success counts, out-of-distribution (out of 5 rollouts per cell). Conditions: novel object placement (croissant on transparent oven door), novel motion primitive (pour golf ball out of bowl), novel object composition (place yellow Lego brick on a wooden boomerang).

Variant	Inference	Avg Success Rate
Pretraining strategy
From scratch (no pretrained CogVideoX init)	action	13
Pretrained init, action-only fine-tuning	action	0
Pretrained init, separate video and action objectives	action	36
Diffusion objective composition
Action-only training (no video loss)	action	0

Appendix

Table 6: Ablations on LIBERO task generalization. Average held-out success rate (%) over 10 rollouts per task (

100

per suite). All variants trained for 5 epochs and share the LIBERO-90 action data and the evaluation protocol of Table 1 . At this budget the full recipe reaches

58\%

and the from-scratch variant

13\%

; at convergence they reach

76\%

and

58\%

(Table 1 ). Inference : whether the evaluated model denoises actions alone or generates video jointly with actions.

Optimizer	AdamW
Learning rate	$1\times 10^{-4}$ pretrain, $2\times 10^{-5}$ finetune (constant)
$\beta_{1},\beta_{2}$	0.9, 0.95
Weight decay	$1\times 10^{-4}$
Gradient clipping (max norm)	1
Batch
Per-GPU batch size	4

Appendix

Table 7: Tapestry training hyperparameters (CogVideoX-2B backbone).

Source	Type	Real hrs	Native FPS	Effective FPS	Training hrs	Batch weight
Cross-embodiment video pretraining (video objective only)
DROID [ 16 ]	robot	131.8	15	15	131.8	0.027
Ego4D [ 11 ]	human	32.1	30	30	32.1	0.020
Epic Kitchens [ 8 ]	human	6.1	60	60	6.1	0.014
SSv2 [ 10 ]	human	108.5	12	12	108.5	0.020
Panda-70M [ 5 ]	web	165.4	30	30	165.4	0.127

Appendix

Table 8: Training data mixture. Per-source wall-clock hours of footage (“real hrs”), source-native FPS, model-effective FPS, hours seen at training time after FPS adjustment, and per-batch sampling weight. Cross-embodiment video sources dominate raw hours but are throttled by sampling weight so that in-domain data drives each gradient step.

Figure 5: Variance of

\mathbf{v}_{\theta}(\mathbf{x}_{t},t,\mathbf{c})

across

\bm{\epsilon}

samples (

N=64

). Per-coordinate mean variance for both

\mathbf{v}_{\theta}

and the implied

\hat{\mathbf{x}}_{0}

, plotted on a log scale across

10

scheduled

t

values. Reference bands show the spread of the empirical action distribution under matched conditioning. The variance at

t=T

(regime A) is

\sim

5 orders below the spread of the conditional action distribution; for

t<T

the model output is

\bm{\epsilon}

-coupled (regime B).

Figure 6: Teacher-forced vs. iterative jitter at each

t

N=50

val chunks. Teacher-forced (blue): single forward pass at each

t

on in-distribution

\mathbf{x}_{t}

from GT. Iterative diamonds (orange):

\hat{\mathbf{x}}_{0}^{(k)}

from a

K=10

DDIM trajectory at

t\in\{999,499,99\}

. Gray band: GT chunk jitter

\pm 1\sigma

. Teacher-forced jitter rises monotonically but stays within

\sim\!2\times

GT; iterative jitter grows to

\sim\!8\times

GT, isolating exposure-bias compounding on top of the per-

t

output bias.

DDIM step $k$	$t$	TF $\\|\hat{x}_{0}\,\mathrm{jit}\\|$ (mean $\pm$ std, 50 chunks)	Iterative $\\|\hat{x}_{0}^{(k)}\,\mathrm{jit}\\|$ (1 trajectory)
0	999	$0.152\pm 0.011$	0.237
1	899	$0.153\pm 0.011$	–
2	799	$0.156\pm 0.012$	–
3	699	$0.161\pm 0.012$	–
4	599	$0.172\pm 0.013$	–
5	499	$0.190\pm 0.014$	0.595

Appendix

Table 9: Teacher-forced sweep and iterative trajectory jitter at each scheduled

t

. Jitter is the per-chunk RMS of the second finite difference of

\hat{\mathbf{x}}_{0}

across the

16

-step horizon. GT band:

0.166\pm 0.027

Figure 7: Parameterization specificity at the noise terminus. 64-seed traces of

\mathbf{v}_{\theta}

(top) and the implied

\hat{\mathbf{x}}_{0}

(bottom) for the highest-variance action dim, with all other inputs held fixed. Left (blue): Tapestry (CogVideoX,

v

-pred + ZTS) at

t=T

(

\bar{\alpha}_{T}=0

\mathrm{Var}_{\bm{\epsilon}}\,\mathbf{v}_{\theta}=3.5{\times}10^{-6}

, traces collapse onto a single curve. Right (orange): Wan rectified flow at

\sigma_{\max}=1

\mathrm{Var}_{\bm{\epsilon}}\,\mathbf{v}_{\theta}=1.17

, traces span the action range. Five-and-a-half orders of magnitude separate the two parameterizations on this same metric.

Figure 8: Wan rectified-flow analog of Fig. 6 . Teacher-forced single-step jitter across

\sigma

(blue) collapses with

\sigma

\hat{\mathbf{x}}_{0}=\mathbf{x}_{\sigma}-\sigma\,\mathbf{v}_{\theta}

. The iterative trajectory (orange diamonds) stays elevated throughout and converges to

\sim

\times

the GT band at

\sigma_{\min}

. Unlike CogVideoX, Wan has no

\sigma

at which the regression target structurally decouples from

\bm{\epsilon}

, so the corresponding gate around the mis-tuned low-

\sigma

frequency prior is unavailable.

Quantity	Action chunks	Natural video pixels
$f_{50}$ (chunk horizon, 16 steps)	0.00 Hz	1.875 Hz
$f_{95}$ (chunk horizon, 16 steps)	1.875 Hz	7.50 Hz
Frac. above 3.75 Hz (chunk horizon)	0.014	0.122
Frac. above 7.5 Hz (chunk horizon)	0.008	0.048
$f_{95}$ (long-form, 256 steps)	0.352 Hz	1.992 Hz
Frac. in $[0.75,1.5]$ Hz (long-form)	0.011	0.080

Appendix

Table 10: Spectral energy distribution: natural video (pixels) vs. action chunks. Unit-energy-normalized power, averaged per modality. “Frac. above

f

” is the fraction of total energy above the listed frequency. Long-form bands from a

256

-step Hann window FFT; chunk-horizon

f_{50}

f_{95}

from a

16

-step rectangular FFT.

Vision-Based Robot Control

Vision-based robot control uses camera images to guide robots’ navigation and manipulation through closed-loop perception and action.

See all 221 papers

DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
Mar 11, 2026Teli Ma, Jia Zheng, Zifan Wang +4Robotic Control Vision-Based Robot Control
Mondo Robotics · HKUST(GZ) · HKUST
Turning Video Models into Generalist Robot Policies
May 27, 2026Sizhe Lester Li, Evan Kim, Xingjian Bai +4Robot Policy Learning Cross-Embodiment Robot Learning
1MIT · 2Independent Researcher · 3CMU +1
Action Images: End-to-End Policy Learning via Multiview Video Generation
Apr 7, 2026Haoyu Zhen, Zixian Gao, Qiao Sun +7Action Representation Learning World Action Models
UMass Amherst · NVIDIA · Harvard University +1

Mar 11, 2026J/K move · Enter open · S save

DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control

Teli Ma, Jia Zheng, Zifan Wang, Chunli Jiang+3

Mondo Robotics · HKUST(GZ) · HKUST

Robotic Control Vision-Based Robot Control Data-Efficient Robot Learning Diffusion Policy Video Diffusion Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have emerged as a promising paradigm for robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1 robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that video generation can serve as an effective scaling proxy for robot policy learning. We release code and models at https://dit4dit.github.io/.

Abstract

Vision-Based Robot Control

Explore similar work

DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control

Turning Video Models into Generalist Robot Policies

Action Images: End-to-End Policy Learning via Multiview Video Generation