GRACE: Generation-aware latent compression for efficient video generation

Jiyoung Kim; Paul Hyunbin Cho; Jisu Nam; Donghoon Lee; Hyunsung Go; Yeonkyeong Lee; Hansaem Kim; Seungryong Kim

Figure 1: Teaser. We present GRACE , a novel latent compression technique that fine-tunes pretrained video generation models, such as Wan2.1-14B ( Wan et al., 2025 ) , to generate from substantially fewer latent tokens (e.g., nearly

8\times

fewer). (Top) Text-to-video (T2V) samples at 736p from Wan2.1-14B before compression and from GRACE, using the same prompt. (Bottom left) The same comparison at 480p. (Bottom right) On VBench-T2V ( Huang et al., 2023 ; Huang et al., 2024 ) , GRACE preserves the generation quality of Wan2.1-14B before compression and achieves higher scores than existing high-compression autoencoders ( HaCohen et al., 2024 ; Zheng et al., 2026 ; He et al., 2026 ) , while generating

11.1\times

faster at 480p and

15.5\times

faster at 736p. All models in the plot are evaluated at matched resolutions with 50 sampling steps. The plot reports text-to-video, and the speedups above are measured on image-to-video.

Figure 2: Overall architecture. Stage 1 trains the autoencoder. The frozen encoder

\mathcal{E}

maps the downsampled input to

\mathbf{z}_{\text{base}}

, the residual encoder

\mathcal{E}_{\text{res}}

maps the full-resolution input to

\mathbf{z}_{\text{res}}

, and

\hat{\mathcal{D}}

decodes both, while

\mathcal{L}_{\text{align}}

matches the compressed latent to the pretrained latent inside the frozen DiT. The dashed box on the right of Stage 1 shows a simplified view of the blocks added to

\mathcal{E}_{\text{res}}

and

\hat{\mathcal{D}}

. Stage 2 adapts the DiT to the compressed latent with LoRA, denoising

\mathbf{z}_{\text{base}}

at a lower noise level than

\mathbf{z}_{\text{res}}

at every step, so the base is denoised first.

Figure 3: Effect of the dual latent and

\mathcal{L}_{\text{align}}

. (a) single latent, (b) dual-latent representation without

\mathcal{L}_{\text{align}}

, and (c) ours, all generated at

480{\times}832{\times}81

. (Left) latent distribution, projected with t-SNE and colored by kernel density, with its uniformity metrics ( Yao et al., 2025 ) and the total VBench ( Huang et al., 2023 ; Huang et al., 2024 ) score below, followed by T2V samples generated after Stage 2. (Right) spatio-temporal structure of the latent, shown as its principal components mapped to RGB under the input frames. More samples are in Figs. I.11 and I.12 .

Figure 4: Denoising order. Each row shows the sampling steps of

\mathbf{z}_{\text{base}}

and

\mathbf{z}_{\text{res}}

( Left ) and the generated frames ( Right ), with

\tau=1

pure noise and

\tau=0

clean. (Top) Both parts are denoised at the same noise level. (Bottom)

\mathbf{z}_{\text{base}}

stays

\delta

ahead of

\mathbf{z}_{\text{res}}

in schedule time

u

(Eq. 11 ) at every step.

Autoencoder	Config	AE Training	Latent Tokens	Reconstruction				VBench-I2V
Autoencoder	Config	AE Training	Latent Tokens	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	Total $\uparrow$
Wan2.1-VAE ( Wan et al., 2025 )	f8t4c16p2	scratch †	32.8k	35.15	0.958	0.016	1.13	87.92
Step-Video-VAE ( Ma et al., 2025 )	f16t8c64p1	scratch ‡	17.2k	33.88	0.950	0.029	3.16	84.05
Video DC-AE ( Zheng et al., 2026 )	f32t4c128p1	scratch	8.2k	34.61	0.956	0.024	3.70	84.94
LTX-VAE ( HaCohen et al., 2024 )	f32t8c128p1	scratch	4.3k	31.97	0.914	0.051	19.53	87.06
Single-latent Baseline	f16t8c32p2	fine-tuned	4.3k	33.76	0.956	0.031	13.11	86.44

Table 1: Video autoencoder comparison at

256{\times}256{\times}81

reconstruction and

480{\times}832{\times}81

generation. We report the VBench-I2V ( Huang et al., 2023 ; Huang et al., 2024 ) total after adapting the same pretrained Wan2.1-I2V-14B ( Wan et al., 2025 ) to every latent under the same budget. Config lists

f

,

t

, channel count

c

, and patch size

p

, which set the latent token count. Single-latent Baseline is our baseline without the dual latent or the alignment loss. The first row is the pretrained autoencoder before compression, and our method is shaded . Bold marks the best value in each column among the last three rows (4.3k tokens). † Initialized by inflating its own 2D image autoencoder. ‡ Trained first at f8t4 and then extended to f16t8 with additional modules.

Figure 6

Figure 7

Figure 7: Convergence during DiT adaptation. With and without

\mathcal{L}_{\text{align}}

; solid lines show EMA.

Method		Components			Reconstruction			VBench-T2V			VBench-I2V
Method		$[\mathbf{z}_{\text{base}};\mathbf{z}_{\text{res}}]$	$\mathcal{L}_{\text{align}}$	$\delta>0$	PSNR $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	Quality $\uparrow$	Semantic $\uparrow$	Total $\uparrow$	I2V $\uparrow$	Quality $\uparrow$	Total $\uparrow$
(I)	Wan2.1-14B (pretrained)	–	–	–	35.15	0.016	1.13	85.24	78.70	83.93	95.82	80.01	87.92
(II)	Single-latent Baseline	$\times$	$\times$	$\times$	33.76	0.031	13.11	83.21	77.75	82.12	93.67	79.21	86.44
(III)	(II) + DiT from scratch	$\times$	$\times$	$\times$	33.76	0.031	13.11	74.20	58.40	71.04	53.06	73.94	63.50
(IV)	(II) + dual-latent	$\checkmark$	$\times$	$\times$	33.10	0.031	13.89	84.77	78.65	83.55	93.96	79.32	86.64
(V)	(IV) + $\mathcal{L}_{\text{align}}$	$\checkmark$	$\checkmark$	$\times$	32.63	0.032	13.53	85.22	82.55	84.68	95.23	79.78	87.51

Table 4: Ablation on design components. Row (III) keeps the autoencoder of (II) but trains the DiT from scratch for the same number of steps. Bold marks the best value in each column among rows (II) and (IV)–(VI).

Alignment Target	Reconstruction		VBench-T2V
Alignment Target	PSNR $\uparrow$	rFVD $\downarrow$	Quality $\uparrow$	Semantic $\uparrow$	Total $\uparrow$
None	33.10	13.89	84.77	78.65	83.55
V-JEPA 2.1 features	32.61	13.93	84.98	80.04	83.99
Pretrained DiT features ( Ours )	32.63	13.53	85.22	82.55	84.68

Table 5: Alignment target. Asymmetric denoising is disabled in all rows.

Denoising Order	Offset	VBench-I2V
Denoising Order	Offset	I2V $\uparrow$	Quality $\uparrow$	Total $\uparrow$
$\tau_{\text{base}}=\tau_{\text{res}}$	$\delta=0$	95.23	79.78	87.51
$\tau_{\text{base}}>\tau_{\text{res}}$	$\delta<0$	94.61	79.72	87.17
$\tau_{\text{base}}<\tau_{\text{res}}$ (Ours)	$\delta>0$	95.48	80.31	87.90

Table 6: Ablation on denoising order.

Figure A.1: Detailed autoencoder architecture.

\mathcal{E}_{\text{res}}

adds a downsampling stage between its middle blocks and head, consisting of residual blocks and a strided causal convolution, paired with a parameter-free shortcut that folds space and time into channels.

	Hyperparameter	Phase 1	Phase 2
Architecture	pretrained autoencoder	Wan2.1-VAE	Wan2.1-VAE
	$(C,C^{\prime})$	$(16,16)$	$(16,16)$
	$(r_{s},r_{t})$	$(2,2)$	$(2,2)$
Training setup	input shape	$256{\times}256{\times}81$	$512{\times}512{\times}81$
	trained modules	$\mathcal{E}_{\text{res}}$ , $\hat{\mathcal{D}}$	$\hat{\mathcal{D}}$
	optimizer	AdamW	AdamW

Table A.1: Stage 1 hyperparameters. All settings follow the training recipe of the pretrained autoencoder.

	Hyperparameter	I2V	T2V
Architecture	pretrained DiT	Wan2.1-I2V-14B	Wan2.1-T2V-14B
	input dim	72	32
	hidden dim	5120	5120
	blocks	40	40
	num. heads	40	40
	patch size	2	2

Table B.1: Stage 2 hyperparameters. Offset and shift are sampled from a range in training, fixed at inference.

Table D.1: Per-dimension VBench ( Huang et al., 2023 ; Huang et al., 2024 ) scores at

480{\times}832{\times}81

. Single-latent Baseline is our baseline without the dual latent or the alignment loss. The best score in each row is in bold. † Reported for reference only, as the official VBench-I2V quality score does not include this dimension.

Table D.2: Per-dimension VBench ( Huang et al., 2023 ; Huang et al., 2024 ) scores at

736{\times}1280{\times}81

, following Tab. D.1 . The best score in each row is in bold. ‡ Measured with spatial tiling in the VAE encoder to avoid running out of memory when encoding the conditioning image at this resolution.

Task	Aspect	vs. Wan2.1-14B ( Wan et al., 2025 ) (%)			vs. DC-Gen ( He et al., 2026 ) (%)
Task	Aspect	Ours	Equal	Wan2.1-14B	Ours	Equal	DC-Gen
T2V	Visual quality	49.4	15.4	35.3	71.2	11.5	17.3
	Temporal consistency	46.8	21.2	32.1	64.7	21.2	14.1
	Text alignment	52.6	21.8	25.6	62.2	23.1	14.7
I2V	Visual quality	23.7	32.7	43.6	60.3	16.7	23.1
	Temporal consistency	29.5	25.6	44.9	64.1	11.5	24.4

Table D.3: Human evaluation. Participants compared each pair of videos without knowing which model produced which. Each cell gives the percentage of votes preferring our model (Ours), rating both videos about equal (Equal), or preferring the baseline named in the column.

Figure D.1: User study interface for T2V samples.

Figure D.2: User study interface for I2V samples.

First frame	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
$\times$	31.97	0.928	0.038
$\checkmark$	32.63	0.930	0.032

Table E.1: Ablation on first-frame conditioning.

Depth	Reconstruction			VBench-I2V
Depth	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	I2V $\uparrow$	Quality $\uparrow$	Total $\uparrow$
10	32.63	0.930	0.032	95.48	80.31	87.90
40	32.21	0.929	0.038	95.70	80.19	87.94

Table E.2: Ablation on alignment depth.

Figure F.1: Reconstruction under latent noise. We noise

\mathbf{z}_{\text{base}}

or

\mathbf{z}_{\text{res}}

at level

\tau

, leave the other unchanged, and reconstruct. (a) Reconstruction quality on Panda-70M ( Chen et al., 2024 ) at

480{\times}832{\times}81

as

\tau

grows, measured in PSNR and LPIPS. (b) Reconstructed frames at two levels of

\tau

, shown below the input and the noise-free reconstruction, with the first block noising

\mathbf{z}_{\text{base}}

and the second noising

\mathbf{z}_{\text{res}}

.

Figure F.2: PCA visualization of the base and residual latents. Principal Component Analysis (PCA) of each part at f16t8p2, encoded from a

480{\times}832{\times}81

video, for (a)

\mathbf{z}_{\text{base}}

, (b)

\mathbf{z}_{\text{res}}

without

\mathcal{L}_{\text{align}}

, and (c)

\mathbf{z}_{\text{res}}

with

\mathcal{L}_{\text{align}}

. Since

\mathcal{E}

is frozen,

\mathbf{z}_{\text{base}}

is the same in both settings.

Figure F.3: Convergence of the base and the residual. Flow matching loss on the base channels (left) and the residual channels (right) during DiT adaptation, for the dual latent with and without

\mathcal{L}_{\text{align}}

. Each part is standardized with its own statistics. Faint curves show the raw loss and solid curves its EMA.

		T2V		I2V
		Wan2.1-14B	Ours	Wan2.1-14B	Ours
Training	Stage 1, full autoencoder	–	6.9	–	6.9
	Stage 1, decoder-only (+EMA)	–	1.6	–	1.6
	Stage 2	–	30	–	30
	Total (H200 GPU days)	–	38.5	–	38.5
Inference latency (s) $\downarrow$		851.5	75.8	863.2	77.7

Table G.1: Computational cost. Training in H200 GPU days, and inference at

480{\times}832{\times}81

on a single A100 with 50 sampling steps.

Figure I.1: Additional image-to-video results. Samples from VBench-I2V ( Huang et al., 2023 ; Huang et al., 2024 ) at

480{\times}832{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and DC-Gen ( He et al., 2026 ) at f32t4p1, from the same conditioning frame and prompt.

Figure I.2: Additional image-to-video results (continued). Samples from VBench-I2V ( Huang et al., 2023 ; Huang et al., 2024 ) at

480{\times}832{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and DC-Gen ( He et al., 2026 ) at f32t4p1, from the same conditioning frame and prompt.

Figure I.3: Additional text-to-video results. Samples from VBench-T2V ( Huang et al., 2023 ; Huang et al., 2024 ) at

480{\times}832{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and DC-Gen ( He et al., 2026 ) at f32t4p1, from the same prompt.

Figure I.4: Additional text-to-video results (continued). Samples from VBench-T2V ( Huang et al., 2023 ; Huang et al., 2024 ) at

480{\times}832{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and DC-Gen ( He et al., 2026 ) at f32t4p1, from the same prompt.

Figure I.5: Additional high-resolution text-to-video results. Samples at

736{\times}1280{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and DC-Gen ( He et al., 2026 ) at f32t4p1, from the same prompt.

Figure I.6: Additional high-resolution text-to-video results (continued). Samples at

736{\times}1280{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and DC-Gen ( He et al., 2026 ) at f32t4p1, from the same prompt.

Figure I.7: Additional high-resolution image-to-video results. Samples at

736{\times}1280{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and LTX-Video ( HaCohen et al., 2024 ) 0.9.7 at f32t8p1, from the same conditioning frame and prompt. The motion of LTX-Video tends to come from a global zoom or a slow camera movement over the conditioning frame, while the scene stays static.

Figure I.8: Additional high-resolution image-to-video results (continued). Samples at

736{\times}1280{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and LTX-Video ( HaCohen et al., 2024 ) 0.9.7 at f32t8p1, from the same conditioning frame and prompt. The motion of LTX-Video tends to come from a global zoom or a slow camera movement over the conditioning frame, while the scene stays static.

Figure I.9: Additional high-resolution image-to-video results (continued). Samples at

736{\times}1280{\times}81

, generated by the pretrained Wan2.1-14B ( Wan et al., 2025 ) at f8t4p2, ours at f16t8p2, and LTX-Video ( HaCohen et al., 2024 ) 0.9.7 at f32t8p1, from the same conditioning frame and prompt. The motion of LTX-Video tends to come from a global zoom or a slow camera movement over the conditioning frame, while the scene stays static.

Figure I.10: Generation in various styles. Text-to-video samples from ours at

736{\times}1280{\times}81

, generated from a single prompt with a different style suffix in each row.

Figure I.11: Additional latent PCA visualizations. Principal components of the full

C{+}C^{\prime}

latent at f16t8p2, encoded from

480{\times}832{\times}81

videos, for (a) a single encoder, (b) ours without

\mathcal{L}_{\text{align}}

, and (c) ours, with the input frames on top. The three leading components are computed per clip and mapped to RGB, so colors are not comparable across panels.

Figure I.12: Additional latent PCA visualizations (continued). Principal components of the full

C{+}C^{\prime}

latent at f16t8p2, encoded from

480{\times}832{\times}81

videos, for (a) a single encoder, (b) ours without

\mathcal{L}_{\text{align}}

, and (c) ours, with the input frames on top. The three leading components are computed per clip and mapped to RGB, so colors are not comparable across panels.

GRACE: Generation-aware latent compression for efficient video generation

Abstract

Figures & tables

Explore similar work

Generative Video Compression with Adaptive Score Distillation

PARE: Pruning and Adaptive Routing for Efficient Video Generation

Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation