Current video generation models suffer from high computational latency, making real-time applications prohibitively costly. In this paper, we address this limitation by exploiting the temporal redundancy inherent in video latent patches. To this end, we propose the Latent Inter-frame Pruning with Attention Recovery (LIPAR) framework, which detects and skips recomputing duplicated latent patches. Additionally, we introduce a novel Attention Recovery mechanism that approximates the attention values of pruned tokens, thereby removing visual artifacts arising from naively applying the pruning method. Empirically, our method increases video editing throughput by 1.53×, achieving an average of 19.3 FPS on an NVIDIA RTX 4090 with the 1.3B Self-Forcing model (4-step denoising, FP16). The proposed method does not compromise generation quality and can be seamlessly integrated with the model without additional training. Our approach effectively bridges the gap between traditional compression algorithms and modern generative pipelines.
Figures & tables
Figure 2: Decoding Compressed Latents. Original: Directly decode the video latents; Compressed: Compressed (nearly) unchanged latent patches.
Figure 3: Illustration of the approximation of pruned tokens to the unpruned token sequence. Dashed circles indicate pruned tokens, where x1≈x2≈x3 and x4≈x5 .
Figure 4: LIPAR overview: The proposed method consists of three stages: 1. Pruning 2. Attention Recovery and 3. Restoration.
Figure 5: Illustration of the Attention Recovery Method. This method preserves visual quality in pruned tokens via two mechanisms: M-Degree Approximation and Noise-Aware Duplication. Pruned keys ( k ) and values ( v ) are approximated by copying temporal counterparts from the clean KV-cache ( e.g. , t−1 ) to maintain the i.i.d. noise assumption, ensuring the m closest tokens to the query remain populated. For simplicity, we only explicitly draw the Noise-Aware duplication for k .
Figure 6: Qualitative comparison with representative low latency V2V models. Our method achieves comparable results to Self-Forcing while having higher throughput, and outperforms the rest of the models. Prompt: “Three corgi puppies sharing a meal together on a kitchen floor.” We encourage readers to refer to the supplementary materials for more video comparisons.
Figure 7: Comparison of user preference and throughput against other models.
Figure 8: Visual comparison of different pruning methods. LIPAR achieves superior visual quality compared with other token pruning methods. Prompt: “Animation style of a frog dancing and performing acrobatic side somersaults.“
Figure 8
Figure 10: Inference latency on a NVIDIA A6000 GPU for generating a 4.5-second video across varying token remains.
V-Bench Quality ↑
Method
FPS ↑
Warp Error ↓
Subj.
Backg.
Motion
Img. Qual.
TTM (5b)
0.58
63.6
0.956
0.957
0.991
0.652
LIPAR (Ours)
0.87
40.4
0.961
0.962
0.993
0.665
Table 2: Quantitative comparison of performance and quality on TTM.
Figure 11: LPIPS Score vs. θ . As we increase the threshold θ for compression in Eqn. 14 , the compression rate (annotated in black) increases. Notably, high visual similarity (LPIPS ≤0.05 , dashed line) is maintained even when the compression rate rises to 46% . This quantitatively confirms that substantial temporal redundancy exists in latent space.
Figure 12: Webpage for performing human evaluation test.
Figure 13: Qualitative comparison on motion control tasks. We visualize the results of our LIPAR applied to motion control applications compared against baseline (original) methods.
Video generation, while capable of generating realistic videos, is computationally expensive and slow, prohibiting real-time applications. In this paper, we observe that video latents encoded via an autoencoder under the Latent Diffusion Model (LDM) framework contain redundancy along the temporal axis. Analogous to how traditional video compression algorithms avoid transmitting redundant frame data, we propose the Latent Inter-frame Pruning framework to prune (skip the re-computation of) duplicated latent patches, thereby reducing computational burden and increasing throughput. However, direct pruning results in visual artifacts due to the discrepancy between full-sequence training and pruned inference. To resolve these artifacts, we propose an Attention Recovery mechanism to bridge the train-inference gap. With our proposed method, we increase video editing throughput by 1.44×, achieving 12.44 FPS on an NVIDIA RTX 6000 while maintaining video quality. We hope our work inspires further research into integrating traditional video compression methods with modern video generation pipelines. This work is a preliminary work on Training-free Latent Inter-Frame Pruning with Attention Recovery.
Dennis Menn, Chih-Hsien Chou
The University of Texas at Austin · Futurewei Technologies, Inc.
Video diffusion models have recently enabled high-quality video generation with ViT-based architectures, but remain computationally intensive because generation requires attention computation over long spatiotemporal sequences. Token pruning has proven effective for ViTs and VLMs. However, most prior pruning methods are attention-based and operate per frame, failing to ensure the vital temporal coherence across frames in video generation tasks. In practice, naively adopting attention-only pruning causes noticeable degradation due to worsened background consistency, flickering, and reduced image quality. To address this, we propose TAPE, a training-free Temporal Aware Pruning for Efficient diffusion-based video generation. TAPE (i) applies temporal smoothing to align token-importance across adjacent frames and suppress selection jitter; and (ii) performs token reselection in selected layers to align token pruning with layers' diverse semantic focus and avoid error accumulation in specific areas; it also (iii) adopt a timestep-level budget scheduling that prunes aggressively at early noisy steps and relaxes pruning during fidelity-critical refinement. The experimental results show that TAPE delivers significant speedups while preserving high visual fidelity, outperforming prior token reduction approaches.
Sheng Li, Yang Sui, Junhao Ran +3
University of Pittsburgh · Rice University · Rutgers University +1
High computational costs and slow inference hinder the practical application of video generation models. While prior works accelerate the generation process through feature caching, they often suffer from notable quality degradation. In this work, we reveal that this issue arises from their inability to distinguish truly redundant features, which leads to the unintended skipping of computations on important features. To address this, we propose PreciseCache, a plug-and-play framework that precisely detects and skips truly redundant computations, thereby accelerating inference without sacrificing quality. Specifically, PreciseCache contains two components: LFCache for step-wise caching and BlockCache for block-wise caching. For LFCache, we compute the Low-Frequency Difference (LFD) between the prediction features of the current step and those from the previous cached step. Empirically, we observe that LFD serves as an effective measure of step-wise redundancy, accurately detecting highly redundant steps whose computation can be skipped through reusing cached features. To further accelerate generation within each non-skipped step, we propose BlockCache, which precisely detects and skips redundant computations at the block level within the network. Extensive experiments on various backbones demonstrate the effectiveness of our PreciseCache, such as achieving an average of 2.6× speedup on Wan2.1-14B without noticeable quality loss.
Jiangshan Wang, Kang Zhao, Jiayi Guo +5
MMLab, CUHK · Tsinghua University · Tongyi Lab, Alibaba