Frontier video generation models work in a VAE latent space that compresses video in both space and time. This compression made training tractable, but now limits high-resolution, controllable, and interactive generation -- doubling resolution quadruples token count, conditions apply to groups of frames rather than individual ones, and each response must wait for the rest of its multi-frame block. We present PixVid, a simple yet effective method that lifts a pretrained latent video model to directly generate every frame in pixel space. PixVid keeps the diffusion transformer and replaces the VAE decoder with a convolutional head that renders its outputs into pixels. We find that predicting clean video rather than velocity, together with a larger noise shift, is key to quality. Finetuned from Wan2.1-14B, PixVid is competitive with the latent baseline within ∼1.3M training clips. Working framewise in pixels, it generates at 2× higher resolution with the same per-frame token count, keeps depth-conditioned video editing aligned under fast motion, and responds to each action at the next frame. Project page: https://chenshuo20.github.io/PixVid/
Figures & tables
Figure 2: Lifting a pretrained latent video DiT to pixel space. Top: the pretrained latent model. A linear pixel-latent alignment, fitted offline, is fused with the pretrained patch embedding to obtain a pixel patch embedding. Bottom: PixVid. A timestep calibration rescales noisy pixel input to the signal-to-noise ratio the pretrained DiT was trained on. The DiT is initialized from the latent model, and a convolutional detailer head replaces the linear head and VAE decoder to predict the clean video directly.
Figure 3: Patch-size adaptation. To obtain 1×2p×2p patches, the new patch embedding weight is obtained by fusing 2×2 average downsampling with the pretrained 1×p×p patch embedding. A new up block L+1 is prepended to the detailer head and initialized as nearest neighbor upsampling. With token count unchanged, we train at 2× resolution at the same DiT cost.
Overall
Quality dimensions
Semantic dimensions
Model
Size
Total
Quality
Semantic
Subject consist.
Backgr. consist.
Temporal flicker.
Motion smooth.
Dynamic degree
Aesthetic quality
Imaging quality
Object class
Multiple objects
Human action
Color
Spatial relation.
Scene
Appear. style
Temporal style
Overall consist.
Wan2.1-T2V [ 85 ]
14B
84.05
84.91
80.63
97.22
98.20
99.21
98.28
60.83
66.60
67.77
95.24
86.65
97.00
84.73
78.42
52.18
22.52
24.57
26.89
PixelUMM [ 88 ]
8B MoT
83.72
84.19
81.86
97.27
96.75
99.25
98.69
58.33
62.56
68.72
96.79
89.31
96.20
90.24
77.38
55.35
21.86
24.84
27.23
PixVid 480p
14B
84.78
85.44
82.15
96.65
97.30
98.94
96.92
73.89
67.30
71.25
95.66
88.87
96.20
81.59
84.74
57.75
23.23
24.32
26.91
PixVid 960p
14B
85.09
86.19
80.68
97.00
98.51
99.59
98.31
65.00
67.49
71.83
97.59
88.16
96.40
74.54
84.14
56.22
21.91
24.46
26.58
Table 1: VBench comparison. PixVid is finetuned from the latent model Wan2.1-T2V-14B; PixelUMM is a concurrent pixel-space model. Each model generates five videos per GPT-enhanced prompt, and we score the first 1.3 s, the length of our videos (App. B.2 ). PixVid has a higher total score than both baselines at both resolutions, so lifting generation from latents to pixels does not degrade quality; its largest gains over Wan2.1 are in dynamic degree, spatial relationship, scene, and imaging quality, and its largest loss is in color. Bold : best; underlined : second best.
Figure 4: Human evaluation. Pairwise preferences between PixVid (480p) and baselines on 100 Movie Gen [ 69 ] prompts, judged by 10 evaluators. Error bars indicate 95% confidence intervals on PixVid’s share of votes. PixVid wins in all comparisons.
Alignment
Temporal
Quality
Model
Motion ↓
Kept ↑
Depth ↓
tLP ↓
FVMD ↓
FVD ↓
Zero-shot
VACE-1.3B [ 43 ]
30.4
32%
0.270
0.075
3119
199
VACE-14B [ 43 ]
24.7
63%
0.268
0.055
2469
250
Wan2.1-Fun-1.3B [ 1 ]
27.3
56%
0.275
0.057
2453
343
Wan2.2-Fun-5B [ 1 ]
32.4
46%
0.344
0.057
2921
239
Table 2: Precise video editing. Motion error in pixels, motion kept, depth error, and temporal LPIPS (tLP) are medians over the 250 fast clips, with all motion bins in Tab. 7 ; FVMD and FVD are computed over all clips (App. B.4 ).
Figure 5: Depth-conditioned Editing. Our pixel model enables precise video editing under high motion, while the temporally compressed latent model morphs and blurs fast-moving objects.
Figure 6: Control-to-screen latency. Control inputs, e.g . turn right ( → ), turn left ( ← ), and look down ( ↓ ), arrive over time (top row). Colors indicate which control each generated frame reflects. Latent model: controls are reflected in the next block. With block size N this means a wait of up to N−1 frames. We use N=4 for this visualization. Pixel model (ours): generates one frame per control, so controls are reflected with zero wait, making it better suited for real-time interactive generation.
Latency and speed
Quality
Camera
Model
Block
Wait ↓
Meas. ↓
FPS ↑
FVD ↓
Rot. ↓
Tr. ↓
Turn ↑
SolarWM [ 37 ]
12
11
495 ms
72.0
53.8
4.24 ∘
0.231
0.70
PixVid
1
0
34 ms
29.4
51.1
5.44 ∘
0.312
0.95
Table 3: Low-latency autoregressive generation. Wait and Meas. report the maximum control wait in frames and median control-to-screen latency on one B200; Rot., Tr., and Turn measure camera rotation error, translation error, and motion correlation on abrupt turns (App. B.5 ). Both models exceed their native 16 fps, but only PixVid responds at the next frame, with 14× lower control-to-screen latency.
Figure 7: Responsiveness. SolarWM receives one pose per four-frame latent, resulting in inaccurate motion when the direction changes; here continuing to look up when it should look down. PixVid handles direction changes natively.
Figure 8: Impact of output head. Under x -prediction, a linear head maps each DiT token to its pixel patch independently, causing visible grid-like artifacts, while our convolutional head removes most seams. Metric details are in App. B.6 .
Figure 9: Impact of prediction target. With the same convolutional head, v -prediction produces grainy frames at 80× the estimated noise of x -prediction, while x -prediction produces clean ones. Metric details are in App. B.6 .
Noise shift s
1
5
15
30
60
Effective shift s/c
0.26
1.32
3.97
7.94
15.87
VBench total ↑
81.85
84.27
84.40
84.09
83.42
VBench quality ↑
83.87
85.71
85.64
85.98
85.06
VBench semantic ↑
73.79
78.51
79.46
76.49
76.88
Table 4: Noise shift. Wan2.1-T2V-1.3B with our convolutional head and x -prediction, trained and sampled with shift s ; VBench uses all 946 prompts with one video each. The timestep calibration acts as a shift by 1/c ( c=3.78 ), so the DiT sees the effective shift s/c . The total and semantic scores peak at s=15 .
Figure 10: Sampler-state precision. We use the same prompt and initial noise in every column. With a bf16 sampler state, UniPC drains pixel-space sample color (red), while Euler keeps it. Color kept is mean OKLab chroma relative to an fp32 sampler state.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
480 p
960 p
Patch size p
1×16×16
1×32×32
Tokens per frame
30×52
30×52
Up blocks L
4
5
Widths CL,…,C1
1024,256,64,64
4096,1024,256,64,64
Head parameters
337 M
942 M
Timestep shift
11
22
Appendix
Table 5: Detailer head at 480 p vs. 960 p.
Figure 11: Detailer head architecture (patch size p=16 ). Top: the DiT’s pre-head tokens s , combined with the timestep embedding as SiLU(et+s) , form a feature map on the token grid. A bottleneck convolution and L=log2p=4 up blocks upsample it 2× per level to full resolution, and an output convolution predicts x^0 . Feature sizes are per frame, where D is the DiT width and Cℓ the width of up block ℓ . Bottom left: each up block is a 2× transposed convolution followed by a causal 3×3×3 convolution and SiLU. Bottom right: all convolutions are causal in time ( kt=3 ): output frame t only reads frames t−2 to t , so in frame-by-frame generation the head keeps the previous two frames as a streaming cache.
Overall
Quality dimensions
Semantic dimensions
Model
Size
Total
Quality
Semantic
Subject consist.
Backgr. consist.
Temporal flicker.
Motion smooth.
Dynamic degree
Aesthetic quality
Imaging quality
Object class
Multiple objects
Human action
Color
Spatial relation.
Scene
Appear. style
Temporal style
Overall consist.
Wan2.1-T2V [ 85 ] , first 21 frames
14B
84.05
84.91
80.63
97.22
98.20
99.21
98.28
60.83
66.60
67.77
95.24
86.65
97.00
84.73
78.42
52.18
22.52
24.57
26.89
Wan2.1-T2V [ 85 ] , all 81 frames
14B
84.07
84.84
81.01
94.96
97.64
99.18
98.18
66.39
67.15
67.81
95.27
86.69
97.60
85.36
78.86
52.66
22.43
24.83
27.13
PixelUMM [ 88 ] , first 31 frames
8B MoT
83.72
84.19
81.86
97.27
96.75
99.25
98.69
58.33
62.56
68.72
96.79
89.31
96.20
90.24
77.38
55.35
21.86
24.84
27.23
PixelUMM [ 88 ] , all 96 frames
8B MoT
83.82
84.21
82.25
95.46
95.52
99.31
98.71
61.94
64.38
68.82
96.93
89.33
97.20
90.59
78.97
55.73
21.65
24.98
27.34
PixVid 480p
14B
84.78
85.44
82.15
96.65
97.30
98.94
96.92
73.89
67.30
71.25
95.66
88.87
96.20
81.59
84.74
57.75
23.23
24.32
26.91
Appendix
Table 6: VBench with full-length baseline videos. Rows marked “first” repeat Tab. 1 , which scores the first 1.3 s of every video, the length of our videos. Rows marked “all” score the same baseline videos over their full length: 81 frames at 16 fps (5.1 s) for Wan2.1 and 96 frames at 24 fps (4.0 s) for PixelUMM. Over the full videos, the baselines’ total scores change by at most 0.10 (84.07 for Wan2.1, 83.82 for PixelUMM) and stay below PixVid’s: longer clips lower subject consistency but raise dynamic degree, and the two changes largely cancel. Best and second best are marked over all six rows. Bold : best; underlined : second best.
Source motion
Model
Slow
Moderate
Medium
Fast
All
Motion error (px) ↓
VACE-1.3B [ 43 ]
0.6
3.4
9.1
30.4
5.6
VACE-14B [ 43 ]
0.4
2.4
6.7
24.7
4.1
Wan2.1-Fun-1.3B [ 1 ]
0.5
2.8
7.6
27.3
4.6
Wan2.2-Fun-5B [ 1 ]
0.8
3.8
10.2
32.4
5.9
Appendix
Table 7: Precise video editing by source motion. Medians over the 250 held-out clips in each motion bin (App. B.4 ) and over all 1,000 clips; Cosmos-Transfer2.5 completed 962 clips. Motion kept is the generated video’s flow magnitude as a percentage of the source’s. The first five models are zero-shot; the last two are finetuned on our data with the same control branch. Set-level FVMD and FVD are in Tab. 2 . Bold: best per block.
Metric
SolarWM [ 37 ]
PixVid
Ground truth
FID ↓
12.65
12.70
–
FVD ↓
53.8
52.4
–
Subject consistency ↑
87.8
88.8
91.2
Background consistency ↑
93.2
92.6
94.4
Motion smoothness ↑
97.6
98.5
98.2
Dynamic degree ↑
97.0
97.7
93.2
Appendix
Table 8: Video quality for autoregressive generation on the 807 camera paths (App. B.5 ). FID and FVD compare with the ground-truth videos; the VBench-style scores [ 39 ] are in percent, with ground-truth videos shown for reference. Bold: better of the two models.
Rollout
Model
FVD ↓
Rot. ↓
Tr. ↓
21 frames (1.3 s)
SolarWM [ 37 ]
58.2
1.53 ∘
0.255
PixVid
54.0
1.60 ∘
0.273
41 frames (2.5 s)
SolarWM [ 37 ]
50.2
2.52 ∘
0.236
PixVid
46.1
2.76 ∘
0.268
81 frames (5.1 s)
SolarWM [ 37 ]
53.8
4.24 ∘
0.231
PixVid
51.1
5.44 ∘
0.312
Appendix
Table 9: Camera-controlled autoregressive generation across rollout lengths. We computed over the first 21 , 41 , 81 generated frames, following the same metrics as in Tab. 3 .
Model
Code
Block
Tokens
TFLOPs
FPS
Latency
SolarWM [ 37 ]
official
12
1,215
5.6
16.1
1,091 ms
SolarWM
ours
12
1,215
5.6
72.0
495 ms
PixVid
ours
1
405
16.4
29.4
34 ms
Appendix
Table 10: Cost of autoregressive generation on one B200 at 480×864 with 4 denoising steps. PixVid trades throughput for single-frame generation, cutting median control-to-screen latency from 495 ms to 34 ms while still streaming faster than real time.
Figure 12: Comparison with One-latent SolarWM. SolarWM and PixVid show relatively small motion errors and produce moderately realistic videos. In contrast, One-latent SolarWM follows the controls less closely, here severely overestimating the motion, and looks less realistic.
Latency and speed
Quality
Camera
Model
Block
Wait ↓
Meas. ↓
FPS ↑
FVD ↓
Rot. ↓
Tr. ↓
Turn ↑
SolarWM [ 37 ]
12
11
495 ms
72.0
53.8
4.24 ∘
0.231
0.70
SolarWM (1-latent)
4
3
164 ms
50.4
118.6
25.58 ∘
0.410
0.11
PixVid
1
0
34 ms
29.4
51.1
5.44 ∘
0.312
0.95
Appendix
Table 11: One-latent ( 4 -frame) SolarWM ablations compared with 12 -frame SolarWM and frame-wise PixVid on all metrics of Tab. 3 .
Model
Frames
Tokens
Tok./s
PFLOPs
Time (s)
Mem. (GB)
480×832
Wan2.1 [ 85 ]
81
32,760
6,471
168
142
31.0
Wan2.1
21
9,360
7,131
30
26
30.8
PixVid
21
32,760
24,960
174
151
33.4
960×1664
Latent †
21
37,440
28,526
206
176
41.9
Appendix
Table 12: Text-to-video inference cost of the 14B models per clip on one B200, with 50 sampling steps and classifier-free guidance. While removing temporal compression increases per-clip cost at 480×832 , doubling the patch size at 960×1664 keeps token count and runtime comparable to a Wan2.1-VAE latent model.
Figure 13: Inference cost vs. resolution for a 21-frame clip with the 14B DiT (100 passes; Tab. 12 ). Doubling the resolution gives a latent model 4× the tokens and raises its FLOPs 6.9× , while PixVid keeps 30×52 tokens per frame, so only its detailer grows and its FLOPs rise by 17%.
Figure 14: Text-to-video comparison on prompts from Movie Gen Video Bench [ 69 ] . We show frames 0, 10, and 20 of each 21-frame video from Wan2.1-T2V-14B, PixelUMM, and PixVid at 480×832 .
Figure 15: High-Resolution Video Generation. Left: text-to-video samples generated by our pixel model at a resolution of 1664×960 . Right: native-resolution crops of the boxed regions. Our method preserves fine pixel-level details such as fur, whiskers and architectural details because the model directly generates pixels without a VAE that compresses and reconstructs them.
Figure 16: Depth-to-video comparison on fast motion. PixVid can generate high-quality outputs under sharp and fast motion.
Figure 17: Camera-controlled autoregressive generation. We show 8 consecutive generated frames at 16 fps for simplicity.
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
Latent diffusion and flow matching models are ubiquitous in state-of-the-art generative video models. While highly effective, relying on pretrained autoencoders to compress the data into a lower-dimensional latent space also comes with drawbacks. First, training proceeds in two stages: autoencoder pretraining, with compression and reconstruction objectives, followed by latent generative model training, precluding fully end-to-end training. Second, autoencoders also lead to an inherent loss of signal, and the trade-off between reconstruction quality and generative performance in the latent space is still an active area of research. In this paper we explore the viability of generative video models directly in pixel space. We analyze recent approaches in pixel-space generative modeling of images that rely on basic transformer designs, and identify several shortcomings that limit their effectiveness for the high-dimensional pixel-space video generation. To overcome these limitations, we propose a UNet-like transformer architecture along with improvements to the training recipe and sampling strategy. We conduct controlled experiments, training latent and pixel-space models from scratch on Kinetics-600. For the standard frames-to-video task at 128 resolution, our U-JiT model improves generative FVD by 27% over prior state-of-the-art results, and by 14% against the JiT pixel-space baseline. Scaling to higher 256 resolution, we find the improvement over pixel-space and latent-based models grows as we increase the spatiotemporal compression rate up to 64×64×8. Finally, we find that U-JiT improves per-frame FID by 27% over latent models and 61% over pixel-space models on class-to-video generation, with respectively 9% and 56% reduction in FVD.
We present Paris 2.0, the first video generation model pre-trained through decentralized computation. Its training recipe builds upon Paris 1.0 (arXiv:2510.03434), the first ever open-weight Decentralized Diffusion Model (DDM), which showed that image generation can be trained without a monolithic GPU cluster. However, temporally coherent video generation had remained an open problem under decentralized training, and Paris 2.0 closes it. In low-resolution text-to-video training, against a monolithic model trained on the same data under a matched total compute budget, Paris 2.0 cuts Frechet Video Distance (FVD) from 561.04 to 279.01, a ~2.0x improvement, and lifts CLIP text-video similarity and aesthetic score.
Video generation, while capable of generating realistic videos, is computationally expensive and slow, prohibiting real-time applications. In this paper, we observe that video latents encoded via an autoencoder under the Latent Diffusion Model (LDM) framework contain redundancy along the temporal axis. Analogous to how traditional video compression algorithms avoid transmitting redundant frame data, we propose the Latent Inter-frame Pruning framework to prune (skip the re-computation of) duplicated latent patches, thereby reducing computational burden and increasing throughput. However, direct pruning results in visual artifacts due to the discrepancy between full-sequence training and pruned inference. To resolve these artifacts, we propose an Attention Recovery mechanism to bridge the train-inference gap. With our proposed method, we increase video editing throughput by 1.44×, achieving 12.44 FPS on an NVIDIA RTX 6000 while maintaining video quality. We hope our work inspires further research into integrating traditional video compression methods with modern video generation pipelines. This work is a preliminary work on Training-free Latent Inter-Frame Pruning with Attention Recovery.
Dennis Menn, Chih-Hsien Chou
The University of Texas at Austin · Futurewei Technologies, Inc.