Current video generation models suffer from high computational latency, making real-time applications prohibitively costly. In this paper, we address this limitation by exploiting the temporal redundancy inherent in video latent patches. To this end, we propose the Latent Inter-frame Pruning with Attention Recovery (LIPAR) framework, which detects and skips recomputing duplicated latent patches. Additionally, we introduce a novel Attention Recovery mechanism that approximates the attention values of pruned tokens, thereby removing visual artifacts arising from naively applying the pruning method. Empirically, our method increases video editing throughput by 1.53×, achieving an average of 19.3 FPS on an NVIDIA RTX 4090 with the 1.3B Self-Forcing model (4-step denoising, FP16). The proposed method does not compromise generation quality and can be seamlessly integrated with the model without additional training. Our approach effectively bridges the gap between traditional compression algorithms and modern generative pipelines.
Figures & tables
Figure 2: Decoding Compressed Latents. Original: Directly decode the video latents; Compressed: Compressed (nearly) unchanged latent patches.
Figure 3: Illustration of the approximation of pruned tokens to the unpruned token sequence. Dashed circles indicate pruned tokens, where x1≈x2≈x3 and x4≈x5 .
Figure 4: LIPAR overview: The proposed method consists of three stages: 1. Pruning 2. Attention Recovery and 3. Restoration.
Figure 5: Illustration of the Attention Recovery Method. This method preserves visual quality in pruned tokens via two mechanisms: M-Degree Approximation and Noise-Aware Duplication. Pruned keys ( k ) and values ( v ) are approximated by copying temporal counterparts from the clean KV-cache ( e.g. , t−1 ) to maintain the i.i.d. noise assumption, ensuring the m closest tokens to the query remain populated. For simplicity, we only explicitly draw the Noise-Aware duplication for k .
Figure 6: Qualitative comparison with representative low latency V2V models. Our method achieves comparable results to Self-Forcing while having higher throughput, and outperforms the rest of the models. Prompt: “Three corgi puppies sharing a meal together on a kitchen floor.” We encourage readers to refer to the supplementary materials for more video comparisons.
Figure 7: Comparison of user preference and throughput against other models.
Figure 8: Visual comparison of different pruning methods. LIPAR achieves superior visual quality compared with other token pruning methods. Prompt: “Animation style of a frog dancing and performing acrobatic side somersaults.“
Figure 8
Figure 10: Inference latency on a NVIDIA A6000 GPU for generating a 4.5-second video across varying token remains.
V-Bench Quality ↑
Method
FPS ↑
Warp Error ↓
Subj.
Backg.
Motion
Img. Qual.
TTM (5b)
0.58
63.6
0.956
0.957
0.991
0.652
LIPAR (Ours)
0.87
40.4
0.961
0.962
0.993
0.665
Table 2: Quantitative comparison of performance and quality on TTM.
Figure 11: LPIPS Score vs. θ . As we increase the threshold θ for compression in Eqn. 14 , the compression rate (annotated in black) increases. Notably, high visual similarity (LPIPS ≤0.05 , dashed line) is maintained even when the compression rate rises to 46% . This quantitatively confirms that substantial temporal redundancy exists in latent space.
Figure 12: Webpage for performing human evaluation test.
Figure 13: Qualitative comparison on motion control tasks. We visualize the results of our LIPAR applied to motion control applications compared against baseline (original) methods.