Back to Pixels: Lifting Latent Video Diffusion into Pixel Space

Yanke Song; Shuo Chen; Chris Rockwell; Guande He; Nan Liu; Desai Xie; Tianpei Gu; Yong-Hyun Park

cs.CVNov 12, 2026

Back to Pixels: Lifting Latent Video Diffusion into Pixel Space

Authors: Yanke Song, Shuo Chen, Chris Rockwell, Guande He, Nan Liu, Desai Xie, Tianpei Gu, Yong-Hyun Park,

Organizations: Roblox · University of California, Merced · University of Pennsylvania · University of Texas at Austin

Abstract

Frontier video generation models work in a VAE latent space that compresses video in both space and time. This compression made training tractable, but now limits high-resolution, controllable, and interactive generation -- doubling resolution quadruples token count, conditions apply to groups of frames rather than individual ones, and each response must wait for the rest of its multi-frame block. We present PixVid, a simple yet effective method that lifts a pretrained latent video model to directly generate every frame in pixel space. PixVid keeps the diffusion transformer and replaces the VAE decoder with a convolutional head that renders its outputs into pixels. We find that predicting clean video rather than velocity, together with a larger noise shift, is key to quality. Finetuned from Wan2.1-14B, PixVid is competitive with the latent baseline within $\sim 1.3$ M training clips. Working framewise in pixels, it generates at $2\times$ higher resolution with the same per-frame token count, keeps depth-conditioned video editing aligned under fast motion, and responds to each action at the next frame. Project page: https://chenshuo20.github.io/PixVid/

Figures & tables

Figure 2: Lifting a pretrained latent video DiT to pixel space. Top: the pretrained latent model. A linear pixel-latent alignment, fitted offline, is fused with the pretrained patch embedding to obtain a pixel patch embedding. Bottom: PixVid. A timestep calibration rescales noisy pixel input to the signal-to-noise ratio the pretrained DiT was trained on. The DiT is initialized from the latent model, and a convolutional detailer head replaces the linear head and VAE decoder to predict the clean video directly.

Figure 3: Patch-size adaptation. To obtain $1{\times}2p{\times}2p$ patches, the new patch embedding weight is obtained by fusing $2{\times}2$ average downsampling with the pretrained $1{\times}p{\times}p$ patch embedding. A new up block $L{+}1$ is prepended to the detailer head and initialized as nearest neighbor upsampling. With token count unchanged, we train at $2{\times}$ resolution at the same DiT cost.

		Overall			Quality dimensions							Semantic dimensions
Model	Size	Total	Quality	Semantic	Subject consist.	Backgr. consist.	Temporal flicker.	Motion smooth.	Dynamic degree	Aesthetic quality	Imaging quality	Object class	Multiple objects	Human action	Color	Spatial relation.	Scene	Appear. style	Temporal style	Overall consist.
Wan2.1-T2V [ 85 ]	14B	84.05	84.91	80.63	97.22	98.20	99.21	98.28	60.83	66.60	67.77	95.24	86.65	97.00	84.73	78.42	52.18	22.52	24.57	26.89
PixelUMM [ 88 ]	8B MoT	83.72	84.19	81.86	97.27	96.75	99.25	98.69	58.33	62.56	68.72	96.79	89.31	96.20	90.24	77.38	55.35	21.86	24.84	27.23
PixVid 480p	14B	84.78	85.44	82.15	96.65	97.30	98.94	96.92	73.89	67.30	71.25	95.66	88.87	96.20	81.59	84.74	57.75	23.23	24.32	26.91
PixVid 960p	14B	85.09	86.19	80.68	97.00	98.51	99.59	98.31	65.00	67.49	71.83	97.59	88.16	96.40	74.54	84.14	56.22	21.91	24.46	26.58

Table 1: VBench comparison. PixVid is finetuned from the latent model Wan2.1-T2V-14B; PixelUMM is a concurrent pixel-space model. Each model generates five videos per GPT-enhanced prompt, and we score the first

1.3

s, the length of our videos (App. B.2 ). PixVid has a higher total score than both baselines at both resolutions, so lifting generation from latents to pixels does not degrade quality; its largest gains over Wan2.1 are in dynamic degree, spatial relationship, scene, and imaging quality, and its largest loss is in color. Bold : best; underlined : second best.

Figure 4: Human evaluation. Pairwise preferences between PixVid (480p) and baselines on

100

Movie Gen [ 69 ] prompts, judged by

10

evaluators. Error bars indicate

95\%

confidence intervals on PixVid’s share of votes. PixVid wins in all comparisons.

	Alignment			Temporal		Quality
Model	Motion $\downarrow$	Kept $\uparrow$	Depth $\downarrow$	tLP $\downarrow$	FVMD $\downarrow$	FVD $\downarrow$
Zero-shot
VACE-1.3B [ 43 ]	30.4	32%	0.270	0.075	3119	199
VACE-14B [ 43 ]	24.7	63%	0.268	0.055	2469	250
Wan2.1-Fun-1.3B [ 1 ]	27.3	56%	0.275	0.057	2453	343
Wan2.2-Fun-5B [ 1 ]	32.4	46%	0.344	0.057	2921	239

Table 2: Precise video editing. Motion error in pixels, motion kept, depth error, and temporal LPIPS (tLP) are medians over the

250

fast clips, with all motion bins in Tab. 7 ; FVMD and FVD are computed over all clips (App. B.4 ).

Figure 5: Depth-conditioned Editing. Our pixel model enables precise video editing under high motion, while the temporally compressed latent model morphs and blurs fast-moving objects.

Figure 6: Control-to-screen latency. Control inputs, e.g . turn right ( $\rightarrow$ ), turn left ( $\leftarrow$ ), and look down ( $\downarrow$ ), arrive over time (top row). Colors indicate which control each generated frame reflects. Latent model: controls are reflected in the next block. With block size $N$ this means a wait of up to $N-1$ frames. We use $N=4$ for this visualization. Pixel model (ours): generates one frame per control, so controls are reflected with zero wait, making it better suited for real-time interactive generation.

Model	Block	Wait $\downarrow$	Meas. $\downarrow$	FPS $\uparrow$	FVD $\downarrow$	Rot. $\downarrow$	Tr. $\downarrow$	Turn $\uparrow$
	Latency and speed				Quality	Camera
SolarWM [ 37 ]	12	11	495 ms	72.0	53.8	4.24 ∘	0.231	0.70
PixVid	1	0	34 ms	29.4	51.1	5.44 ∘	0.312	0.95

Table 3: Low-latency autoregressive generation. Wait and Meas. report the maximum control wait in frames and median control-to-screen latency on one B200; Rot., Tr., and Turn measure camera rotation error, translation error, and motion correlation on abrupt turns (App. B.5 ). Both models exceed their native

16

fps, but only PixVid responds at the next frame, with

14\times

lower control-to-screen latency.

Figure 7: Responsiveness. SolarWM receives one pose per four-frame latent, resulting in inaccurate motion when the direction changes; here continuing to look up when it should look down. PixVid handles direction changes natively.

Panel of Figure 8 — Figure 8: Impact of output head. Under $\bm{x}$ -prediction, a linear head maps each DiT token to its pixel patch independently, causing visible grid-like artifacts, while our convolutional head removes most seams. Metric details are in App. B.6 .

Panel of Figure 9 — Figure 9: Impact of prediction target. With the same convolutional head, $\bm{v}$ -prediction produces grainy frames at $80\times$ the estimated noise of $\bm{x}$ -prediction, while $\bm{x}$ -prediction produces clean ones. Metric details are in App. B.6 .

Noise shift $s$	1	5	15	30	60
Effective shift $s/c$	0.26	1.32	3.97	7.94	15.87
VBench total $\uparrow$	81.85	84.27	84.40	84.09	83.42
VBench quality $\uparrow$	83.87	85.71	85.64	85.98	85.06
VBench semantic $\uparrow$	73.79	78.51	79.46	76.49	76.88

Table 4: Noise shift. Wan2.1-T2V-1.3B with our convolutional head and

\bm{x}

-prediction, trained and sampled with shift

s

; VBench uses all

946

prompts with one video each. The timestep calibration acts as a shift by

1/c

(

c=3.78

), so the DiT sees the effective shift

s/c

. The total and semantic scores peak at

s=15

Panel of Figure 10 — Figure 10: Sampler-state precision. We use the same prompt and initial noise in every column. With a bf16 sampler state, UniPC drains pixel-space sample color (red), while Euler keeps it. Color kept is mean OKLab chroma relative to an fp32 sampler state.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	$480$ p	$960$ p
Patch size $p$	$1\times 16\times 16$	$1\times 32\times 32$
Tokens per frame	$30\times 52$	$30\times 52$
Up blocks $L$	$4$	$5$
Widths $C_{L},\dots,C_{1}$	$1024,256,64,64$	$4096,1024,256,64,64$
Head parameters	$337$ M	$942$ M
Timestep shift	$11$	$22$

Appendix

Table 5: Detailer head at

480

p vs.

960

Figure 11: Detailer head architecture (patch size $p{=}16$ ). Top: the DiT’s pre-head tokens $s$ , combined with the timestep embedding as $\mathrm{SiLU}(e_{t}+s)$ , form a feature map on the token grid. A bottleneck convolution and $L{=}\log_{2}p{=}4$ up blocks upsample it $2{\times}$ per level to full resolution, and an output convolution predicts $\hat{x}_{0}$ . Feature sizes are per frame, where $D$ is the DiT width and $C_{\ell}$ the width of up block $\ell$ . Bottom left: each up block is a $2{\times}$ transposed convolution followed by a causal $3{\times}3{\times}3$ convolution and SiLU. Bottom right: all convolutions are causal in time ( $k_{t}{=}3$ ): output frame $t$ only reads frames $t{-}2$ to $t$ , so in frame-by-frame generation the head keeps the previous two frames as a streaming cache.

		Overall			Quality dimensions							Semantic dimensions
Model	Size	Total	Quality	Semantic	Subject consist.	Backgr. consist.	Temporal flicker.	Motion smooth.	Dynamic degree	Aesthetic quality	Imaging quality	Object class	Multiple objects	Human action	Color	Spatial relation.	Scene	Appear. style	Temporal style	Overall consist.
Wan2.1-T2V [ 85 ] , first 21 frames	14B	84.05	84.91	80.63	97.22	98.20	99.21	98.28	60.83	66.60	67.77	95.24	86.65	97.00	84.73	78.42	52.18	22.52	24.57	26.89
Wan2.1-T2V [ 85 ] , all 81 frames	14B	84.07	84.84	81.01	94.96	97.64	99.18	98.18	66.39	67.15	67.81	95.27	86.69	97.60	85.36	78.86	52.66	22.43	24.83	27.13
PixelUMM [ 88 ] , first 31 frames	8B MoT	83.72	84.19	81.86	97.27	96.75	99.25	98.69	58.33	62.56	68.72	96.79	89.31	96.20	90.24	77.38	55.35	21.86	24.84	27.23
PixelUMM [ 88 ] , all 96 frames	8B MoT	83.82	84.21	82.25	95.46	95.52	99.31	98.71	61.94	64.38	68.82	96.93	89.33	97.20	90.59	78.97	55.73	21.65	24.98	27.34
PixVid 480p	14B	84.78	85.44	82.15	96.65	97.30	98.94	96.92	73.89	67.30	71.25	95.66	88.87	96.20	81.59	84.74	57.75	23.23	24.32	26.91

Appendix

Table 6: VBench with full-length baseline videos. Rows marked “first” repeat Tab. 1 , which scores the first 1.3 s of every video, the length of our videos. Rows marked “all” score the same baseline videos over their full length: 81 frames at 16 fps (5.1 s) for Wan2.1 and 96 frames at 24 fps (4.0 s) for PixelUMM. Over the full videos, the baselines’ total scores change by at most 0.10 (84.07 for Wan2.1, 83.82 for PixelUMM) and stay below PixVid’s: longer clips lower subject consistency but raise dynamic degree, and the two changes largely cancel. Best and second best are marked over all six rows. Bold : best; underlined : second best.

Motion error (px) $\downarrow$
	Source motion
Model	Slow	Moderate	Medium	Fast	All
VACE-1.3B [ 43 ]	0.6	3.4	9.1	30.4	5.6
VACE-14B [ 43 ]	0.4	2.4	6.7	24.7	4.1
Wan2.1-Fun-1.3B [ 1 ]	0.5	2.8	7.6	27.3	4.6
Wan2.2-Fun-5B [ 1 ]	0.8	3.8	10.2	32.4	5.9

Appendix

Table 7: Precise video editing by source motion. Medians over the 250 held-out clips in each motion bin (App. B.4 ) and over all 1,000 clips; Cosmos-Transfer2.5 completed 962 clips. Motion kept is the generated video’s flow magnitude as a percentage of the source’s. The first five models are zero-shot; the last two are finetuned on our data with the same control branch. Set-level FVMD and FVD are in Tab. 2 . Bold: best per block.

Metric	SolarWM [ 37 ]	PixVid	Ground truth
FID $\downarrow$	12.65	12.70	–
FVD $\downarrow$	53.8	52.4	–
Subject consistency $\uparrow$	87.8	88.8	91.2
Background consistency $\uparrow$	93.2	92.6	94.4
Motion smoothness $\uparrow$	97.6	98.5	98.2
Dynamic degree $\uparrow$	97.0	97.7	93.2

Appendix

Table 8: Video quality for autoregressive generation on the 807 camera paths (App. B.5 ). FID and FVD compare with the ground-truth videos; the VBench-style scores [ 39 ] are in percent, with ground-truth videos shown for reference. Bold: better of the two models.

Rollout	Model	FVD $\downarrow$	Rot. $\downarrow$	Tr. $\downarrow$
21 frames (1.3 s)	SolarWM [ 37 ]	58.2	1.53 ∘	0.255
21 frames (1.3 s)	PixVid	54.0	1.60 ∘	0.273
41 frames (2.5 s)	SolarWM [ 37 ]	50.2	2.52 ∘	0.236
41 frames (2.5 s)	PixVid	46.1	2.76 ∘	0.268
81 frames (5.1 s)	SolarWM [ 37 ]	53.8	4.24 ∘	0.231
81 frames (5.1 s)	PixVid	51.1	5.44 ∘	0.312

Appendix

Table 9: Camera-controlled autoregressive generation across rollout lengths. We computed over the first

21

41

81

generated frames, following the same metrics as in Tab. 3 .

Model	Code	Block	Tokens	TFLOPs	FPS	Latency
SolarWM [ 37 ]	official	12	1,215	5.6	16.1	1,091 ms
SolarWM	ours	12	1,215	5.6	72.0	495 ms
PixVid	ours	1	405	16.4	29.4	34 ms

Appendix

Table 10: Cost of autoregressive generation on one B200 at

480\times 864

with

4

denoising steps. PixVid trades throughput for single-frame generation, cutting median control-to-screen latency from

495

ms to

34

ms while still streaming faster than real time.

Figure 12: Comparison with One-latent SolarWM. SolarWM and PixVid show relatively small motion errors and produce moderately realistic videos. In contrast, One-latent SolarWM follows the controls less closely, here severely overestimating the motion, and looks less realistic.

Model	Block	Wait $\downarrow$	Meas. $\downarrow$	FPS $\uparrow$	FVD $\downarrow$	Rot. $\downarrow$	Tr. $\downarrow$	Turn $\uparrow$
	Latency and speed				Quality	Camera
SolarWM [ 37 ]	12	11	495 ms	72.0	53.8	4.24 ∘	0.231	0.70
SolarWM (1-latent)	4	3	164 ms	50.4	118.6	25.58 ∘	0.410	0.11
PixVid	1	0	34 ms	29.4	51.1	5.44 ∘	0.312	0.95

Appendix

Table 11: One-latent (

4

-frame) SolarWM ablations compared with

12

-frame SolarWM and frame-wise PixVid on all metrics of Tab. 3 .

Model	Frames	Tokens	Tok./s	PFLOPs	Time (s)	Mem. (GB)
$480\times 832$
Wan2.1 [ 85 ]	81	32,760	6,471	168	142	31.0
Wan2.1	21	9,360	7,131	30	26	30.8
PixVid	21	32,760	24,960	174	151	33.4
$960\times 1664$
Latent †	21	37,440	28,526	206	176	41.9

Appendix

Table 12: Text-to-video inference cost of the 14B models per clip on one B200, with

50

sampling steps and classifier-free guidance. While removing temporal compression increases per-clip cost at

480\times 832

, doubling the patch size at

960\times 1664

keeps token count and runtime comparable to a Wan2.1-VAE latent model.

Figure 13: Inference cost vs. resolution for a 21-frame clip with the 14B DiT (100 passes; Tab. 12 ). Doubling the resolution gives a latent model

4\times

the tokens and raises its FLOPs

6.9\times

, while PixVid keeps

30\times 52

tokens per frame, so only its detailer grows and its FLOPs rise by 17%.

Figure 14: Text-to-video comparison on prompts from Movie Gen Video Bench [ 69 ] . We show frames 0, 10, and 20 of each 21-frame video from Wan2.1-T2V-14B, PixelUMM, and PixVid at $480\times 832$ .

Figure 15: High-Resolution Video Generation. Left: text-to-video samples generated by our pixel model at a resolution of $1664\times 960$ . Right: native-resolution crops of the boxed regions. Our method preserves fine pixel-level details such as fur, whiskers and architectural details because the model directly generates pixels without a VAE that compresses and reconstructs them.

Figure 16: Depth-to-video comparison on fast motion. PixVid can generate high-quality outputs under sharp and fast motion.

Figure 17: Camera-controlled autoregressive generation. We show $8$ consecutive generated frames at $16$ fps for simplicity.

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 780 papers

Pixel-space video generation with U-JiT
Nov 3, 2026Théophane Vallaeys, Jakob Verbeek, Matthieu CordHigh-Resolution Video Generation
FAIR, Meta · Sorbonne University, Paris
Paris 2.0: A Decentralized Diffusion Model for Video Generation
May 25, 2026Ali Rouzbayani, Bidhan Roy, Marcos Villagra +1Video Diffusion Models T2V Generation
Bagel Labs (bagel.com)
Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation
Apr 26, 2026Dennis Menn, Chih-Hsien ChouVideo Diffusion Models Token Pruning
The University of Texas at Austin · Futurewei Technologies, Inc.

Nov 3, 2026J/K move · Enter open · S save

Pixel-space video generation with U-JiT

Théophane Vallaeys, Jakob Verbeek, Matthieu Cord

FAIR, Meta · Sorbonne University, Paris

High-Resolution Video Generation

Open paper AskPDF arXiv

Latent diffusion and flow matching models are ubiquitous in state-of-the-art generative video models. While highly effective, relying on pretrained autoencoders to compress the data into a lower-dimensional latent space also comes with drawbacks. First, training proceeds in two stages: autoencoder pretraining, with compression and reconstruction objectives, followed by latent generative model training, precluding fully end-to-end training. Second, autoencoders also lead to an inherent loss of signal, and the trade-off between reconstruction quality and generative performance in the latent space is still an active area of research. In this paper we explore the viability of generative video models directly in pixel space. We analyze recent approaches in pixel-space generative modeling of images that rely on basic transformer designs, and identify several shortcomings that limit their effectiveness for the high-dimensional pixel-space video generation. To overcome these limitations, we propose a UNet-like transformer architecture along with improvements to the training recipe and sampling strategy. We conduct controlled experiments, training latent and pixel-space models from scratch on Kinetics-600. For the standard frames-to-video task at 128 resolution, our U-JiT model improves generative FVD by 27% over prior state-of-the-art results, and by 14% against the JiT pixel-space baseline. Scaling to higher 256 resolution, we find the improvement over pixel-space and latent-based models grows as we increase the spatiotemporal compression rate up to

64 \times 64 \times 8

. Finally, we find that U-JiT improves per-frame FID by 27% over latent models and 61% over pixel-space models on class-to-video generation, with respectively 9% and 56% reduction in FVD.

Abstract

Video Diffusion Models

Explore similar work

Pixel-space video generation with U-JiT

Paris 2.0: A Decentralized Diffusion Model for Video Generation

Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation