A Spatiotemporal Semantic Importance-Guided Unified Compression and Editing Framework for AI-Generated Videos

Xihua Sheng; Dong Liu; Chang Wen Chen

cs.CVOct 5, 2026

A Spatiotemporal Semantic Importance-Guided Unified Compression and Editing Framework for AI-Generated Videos

Authors: Xihua Sheng, Dong Liu, Chang Wen Chen

Organizations: Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China · MOE Key Laboratory of Brain-Inspired Intelligent Perception and Cognition, University of Science and Technology of China, Hefei 230093, China

Abstract

AI-generated videos are rapidly increasing in volume, duration, and resolution, creating growing demands for efficient storage and transmission. Unlike natural videos captured from the physical world, AI-generated videos are samples from a learned generative distribution, where semantic structures are critical to content consistency, while many local textures and stochastic details can be plausibly regenerated. This distinction suggests that compression should preserve semantically important spatiotemporal information rather than reconstruct every pixel of a particular generative sample. Beyond reconstruction, AI-generated videos also create a practical need for prompt-based editing, where users expect to modify generated content while preserving its original spatiotemporal semantics. Motivated by these observations, we propose a unified compression and editing framework for AI-generated videos that incorporates a frozen video generator as a reusable generative prior. Within this framework, we design three spatiotemporal semantic importance-guided techniques that respectively address what to transmit, how much to transmit, and how to use the transmitted side information. First, an innovation selection method projects the latent discrepancy using spatiotemporal semantic importance, so that the selected innovations prioritize semantic invariants over replaceable generative variations. Second, a frame-adaptive bit allocation method estimates the nonuniform semantic demands of latent frames and allocates more innovations to frames requiring stronger semantic preservation. Third, a unified reconstruction and editing method continuously adjusts the influence of the transmitted side information, enabling the same compressed representation to provide strong guidance for faithful reconstruction or serve as a flexible semantic anchor for structure-preserving prompt-driven editing.

Figures & tables

Fig. 1: Overview of the proposed spatiotemporal semantic importance-guided unified compression and editing framework for AI-generated videos.

Fig. 2: Illustration of the spatiotemporal semantic importance-guided innovation selection method. Here, $\mathbf{y}_{0,f}$ and $\mathbf{y}_{0,f-1}$ denote the current and preceding source latent frames, respectively. $\mathbf{r}_{t,f}$ denotes the unrecovered latent discrepancy for latent frame $f$ at generation step $t$ . $\mathbf{W}_{t,f}$ is the position-wise spatiotemporal semantic importance map. $\mathbf{r}_{t,f}^{\mathrm{sem}}$ is the importance-weighted discrepancy and $\overline{\mathbf{r}}_{t,f}^{\mathrm{sem}}$ is the temporally averaged discrepancy.

Fig. 3: Illustration of the proposed spatiotemporal semantic importance-guided bit allocation method. Here,

\mathbf{y}_{0,f}

and

\mathbf{y}_{0,f-1}

denote the current and preceding source latent frames, respectively, and

\mathbf{r}_{t,f}

denotes the unrecovered latent discrepancy for latent frame

f

at generation step

t

L_{t,f}

is the frame-level semantic coding demand.

\rho_{t,f}

is the bounded allocation score mapped from

L_{t,f}

M_{t,f}

denotes the number of allocated codebook atoms.

Fig. 4: Illustration of the proposed spatiotemporal semantic importance-constrained unified reconstruction and editing. Here, $\mathbf{z}_{t,f}^{\mathcal{B}}$ is the reconstructed innovation. $i_{t,f,m}$ and $s_{t,f,m}$ denote the index and sign of the $m$ -th selected atom, respectively. The constraint strength $\eta$ controls the contribution of $\mathbf{z}_{t,f}^{\mathcal{B}}$ , and $\boldsymbol{\epsilon}_{t,f}$ denotes an independent unit-variance innovation. Their combination produces the controlled innovation $\widetilde{\mathbf{z}}_{t,f}(\eta)$ . Under the null-text condition and $\eta=1$ , the reconstructed latent $\widehat{\mathbf{y}}_{0}^{\mathrm{rec}}$ and reconstructed video $\widehat{\mathbf{x}}^{\mathrm{rec}}$ are produced. Under a user-defined prompt and a relaxed constraint, the edited latent $\widehat{\mathbf{y}}_{0}^{\mathrm{edit}}$ and edited video $\widehat{\mathbf{x}}^{\mathrm{edit}}$ are produced.

Fig. 5: Representative test videos generated by Kling, Seedance, Sora, Veo, and Wan models. The test dataset contains 75 videos with diverse content, including human subjects, natural landscapes, objects, and animations.

Fig. 6: Quantitative comparison results on Kling, Seedance, Sora, Veo, and Wan test video sequences.

Class	GVC1D Anchor				YODA Anchor
Class	DISTS	FID	LPIPS	PSNR	DISTS	FID	LPIPS	PSNR
Kling	$-75.50$	$-69.93$	$8.40$	$-50.68$	—	$-87.12$	—	$-72.74$
Seedance	$-70.29$	$-57.77$	$18.18$	$-55.09$	$-57.84$	$-80.82$	—	$-75.59$
Sora	$-51.27$	$-52.63$	$22.40$	$-52.04$	$-51.91$	$-74.97$	—	$-73.62$
Veo	$-59.74$	$-64.28$	$32.19$	$-31.83$	—	$-69.53$	—	$-54.96$
Wan	$-66.52$	$-58.31$	$-5.30$	$-54.89$	$-73.53$	$-87.68$	$-42.94$	$-79.48$

TABLE I: BD-rate (%) of our codec relative to the state-of-the-art generative video codecs GVC1D and YODA on five AI-generated video classes. A dash (—) indicates that BD-rate cannot be computed because the quality ranges do not sufficiently overlap.

Fig. 7: Qualitative reconstruction comparison results.

Fig. 8: Reconstruction and prompt-controllable editing comparison results.

SSIGIS	SSIGBA	DISTS	FID
$\times$	$\times$	0.00	0.00
$\checkmark$	$\times$	–6.89	–14.26
$\checkmark$	$\checkmark$	–12.12	–17.57

TABLE II: Ablation study of our proposed methods in terms of BD-rate (%).

Fig. 9: Visualization of the proposed spatiotemporal semantic importance-guided innovation selection. The top row shows the source video frame, spatial variation $\mathcal{N}(|\nabla_{h}\mathbf{y}_{0,f}|+|\nabla_{w}\mathbf{y}_{0,f}|)$ , and temporal variation $\mathcal{N}(|\mathbf{y}_{0,f}-\mathbf{y}_{0,f-1}|)$ . The bottom row shows the original discrepancy $|\mathbf{r}_{t,f}|$ , semantic importance map $\mathbf{W}_{t,f}$ , and weighted discrepancy $|\mathbf{r}_{t,f}^{\mathrm{sem}}|$ .

Fig. 10: Visualization of the proposed spatiotemporal semantic importance-guided bit allocation. The first row shows the semantic allocation scores and corresponding frame-wise innovation budgets, while the second row presents representative video frames. Higher budgets are assigned to frames containing temporal scene transitions or complex spatial structures, such as text.

Fig. 11: Effect of the semantic constraint strength $\eta$ . The editing prompt is “Change the color of this lady’s shirt to purple.” As $\eta$ decreases, the influence of the transmitted source spatiotemporal semantics is progressively relaxed, producing increasingly stronger changes in appearance and motion.

Explore similar work

Apr 26, 2026cs.CV

Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation

Video generation, while capable of generating realistic videos, is computationally expensive and slow, prohibiting real-time applications. In this paper, we observe that video latents encoded via an autoencoder under the Latent Diffusion Model (LDM) framework contain redundancy along the temporal axis. Analogous to how traditional video compression algorithms avoid transmitting redundant frame data, we propose the Latent Inter-frame Pruning framework to prune (skip the re-computation of) duplicated latent patches, thereby reducing computational burden and increasing throughput. However, direct pruning results in visual artifacts due to the discrepancy between full-sequence training and pruned inference. To resolve these artifacts, we propose an Attention Recovery mechanism to bridge the train-inference gap. With our proposed method, we increase video editing throughput by 1.44

\times

, achieving 12.44 FPS on an NVIDIA RTX 6000 while maintaining video quality. We hope our work inspires further research into integrating traditional video compression methods with modern video generation pipelines. This work is a preliminary work on Training-free Latent Inter-Frame Pruning with Attention Recovery.

Dennis Menn, Chih-Hsien Chou

The University of Texas at Austin · Futurewei Technologies, Inc.

Video Compression Video Latents Autoencoder Architectures

Ask PDF

Aug 4, 2026cs.CV

JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

Real-time video editing requires low-latency causal generation with bounded computational resources while preserving source fidelity and long-term temporal consistency. We present JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework for real-time, open-ended video editing without access to future frames or a predefined video duration. Our method combines chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation to reduce train--inference mismatch, preserve source fidelity during two-step generation, and mitigate accumulated temporal drift. Extensive automatic and human evaluations show that JoyAI-Video-Edit substantially outperforms existing streaming editors and remains competitive with strong offline systems on both short and long videos. The complete system achieves end-to-end 720p video editing at approximately 30 FPS on a single Nvidia B200 GPU. Code is available at https://github.com/jd-opensource/JoyAI-Video-Edit.

Yicheng Xiao, Wenxun Dai, Xinran Qin +22

Joy Future Academy, JD

Video Editing Diffusion-Based Image Editing Autoregressive Diffusion

Ask PDF

Jun 21, 2026eess.IV

ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors

Recent generative video compression methods leverage powerful generative priors to achieve perceptually pleasing reconstructions. However, most existing approaches require additional training to adapt generative models to produce realistic reconstructions from compact representations. In this paper, we propose ZeroGVC, a zero-shot generative video compression framework that leverages pretrained autoregressive diffusion priors for low-delay video reconstruction. ZeroGVC encodes the first frame of each group of pictures (GOP) with an image codec and represents subsequent P-frames through Codebook-Guided Autoregressive Latent Compression. This design is motivated by our observation that the compression scheme of denoising diffusion codebook models is effective in few-step consistency sampling. By selecting compact combinations of reproducible codebook noise vectors, ZeroGVC steers the latent denoising trajectory toward the target P-frame while allowing the decoder to reproduce the same trajectory in only a few denoising steps. In addition, we design an optional bidirectional reference mode that mitigates error propagation by leveraging the next I-frame context without introducing any additional bitrate overhead. Extensive experiments on standard video compression benchmarks demonstrate that ZeroGVC achieves superior perceptual reconstruction quality at ultra-low bitrates without any additional training.

Yixin Gao, Xiaohan Pan, Lin Liu +3

University of Science and Technology of China · 2Huawei Inc.

Video Compression Autoregressive Diffusion Perceptual Quality

Ask PDF