Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolation

Hanshuai Cui; Zhiqing Tang; Zhi Yao; Fanshuai Meng; Weijia Jia; Wei Zhao

cs.CVApr 3, 2026

Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolation

Authors: Hanshuai Cui, Zhiqing Tang, Zhi Yao, Fanshuai Meng, Weijia Jia, Wei Zhao

Organizations: School of Artificial Intelligence, Beijing Normal University, Beijing, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai, China · Shenzhen University of Advanced Technology, Shenzhen, China

Abstract

Autoregressive (AR) video diffusion models enable long-form video generation but remain expensive due to repeated multi-step denoising. Existing training-free acceleration methods rely on binary cache-or-recompute decisions, overlooking intermediate cases where direct reuse is too coarse yet full recomputation is unnecessary. Moreover, asynchronous AR schedules assign different noise levels to co-generated frames, yet existing methods process the entire valid interval uniformly. To address these AR-specific inefficiencies, we present SCOPE, a training-free framework for efficient AR video diffusion. SCOPE introduces a tri-modal scheduler over cache, predict, and recompute, where prediction via noise-level Taylor extrapolation fills the gap between reuse and recomputation with explicit stability controls backed by error propagation analysis. It further introduces selective computation that restricts execution to the active frame interval. On MAGI-1 and SkyReels-V2, SCOPE achieves up to 4.73x speedup while maintaining quality comparable to the original output, outperforming all training-free baselines.

Figures & tables

Figure 1. SCOPE achieves large speedup while preserving near-original quality on both MAGI-1 and SkyReels-V2. A teaser figure showing SCOPE's speedup over the original pipeline on MAGI-1 and SkyReels-V2.

Panel of Figure 2 — Figure 2. At Frame 13 over steps 54–59: (a) Taylor prediction tracks the ground-truth feature more closely than reuse-only; (b) Taylor prediction also yields lower feature error. Two plots for SkyReels-V2 540P generation. The curves denote cross-prompt means and the shaded bands denote cross-prompt standard deviations. The left plot compares the feature value norm of the ground-truth feature, a reuse-only approximation, and Taylor prediction at Frame 13 over denoising steps 54 to 59, with Taylor prediction staying closer to the ground truth. The right plot compares mean absolute feature errors for reuse-only and Taylor prediction, showing lower error for Taylor prediction.

Figure 3. Conceptual step-matrix view of asynchronous autoregressive denoising. At each iteration, only part of the valid frame interval is active; dark cells mark frames whose scheduling state is still advancing. An illustration of an asynchronous autoregressive denoising schedule, highlighting the active frame interval used by selective computation.

Figure 4. Overview of SCOPE for accelerating autoregressive video diffusion by reducing redundant computation in both the spatial and temporal dimensions through selective computation and predictive extrapolation. A framework figure for SCOPE. The left part shows autoregressive video generation across multiple chunks with denoising inside each chunk. The middle top shows a frame-by-step denoising matrix where different frames are at different stages. The middle bottom shows selective computation over the active frame interval. The right part shows predictive extrapolation along the temporal denoising trajectory using cached history.

Model	Method	Efficiency				Visual Quality
Model	Method	Time (s) $\downarrow$	Speedup $\uparrow$	FLOPs (P) $\downarrow$	Speedup $\uparrow$	VBench $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
SkyReels-V2 DF-1.3B	Original	1452.41	1.00 $\times$	245.01	1.00 $\times$	81.51%	0.0000	1.0000	$\infty$
	$\Delta$ -DiT	1223.01	1.19 $\times$	201.60	1.22 $\times$	74.57%	0.6052	0.5228	14.25
	TeaCache-slow	697.60	2.08 $\times$	109.44	2.24 $\times$	79.83%	0.4436	0.5986	15.60
	TeaCache-fast	559.76	2.59 $\times$	84.68	2.89 $\times$	77.49%	0.4786	0.5595	15.59
	TaylorSeer-slow	941.41	1.54 $\times$	130.76	1.87 $\times$	75.54%	0.6838	0.4516	10.32

Table 1. Main comparison on SkyReels-V2 DF-1.3B and MAGI-1 4.5B-distill, with all results measured on a single NVIDIA A800 80GB GPU. LPIPS, SSIM, and PSNR are computed against the Original output as reference. Best and second-best accelerated results are shown in bold and underlined, respectively. Original is excluded from ranking.

Figure 5. Qualitative comparison of all methods on representative prompts from the two models. The left two groups show results on SkyReels-V2, while the right group shows results on MAGI-1. Each method row is annotated with its end-to-end speedup on SkyReels-V2 and MAGI-1, respectively. Compared with other accelerated baselines, SCOPE preserves object structure and visual fidelity more consistently while maintaining the strongest overall acceleration-quality tradeoff. Representative qualitative comparison across all methods on the two target models, with each row annotated by its speedup on SkyReels-V2 and MAGI-1.

Figure 6. Tri-modal decision timeline comparison between SkyReels-V2 (top) and MAGI-1 (bottom). Each row shows the Cache , Predict , and Recompute decisions across denoising iterations. Decision timeline visualization showing cache, predict, and recompute mode distributions across denoising iterations for both MAGI-1 and SkyReels-V2.

SkyReels-V2
Config $(\tau_{c},\tau_{p})$	Efficiency		Quality
Config $(\tau_{c},\tau_{p})$	Time (s) $\downarrow$	Speedup $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
$(0.12,0.30)$	353.11	4.11 $\times$	0.1723	0.7581	23.17
$(0.15,0.36)$	326.21	4.45 $\times$	0.1798	0.7460	22.74
$(0.18,0.42)$	307.34	4.73 $\times$	0.1900	0.7309	21.96
$(0.21,0.48)$	281.56	5.16 $\times$	0.2152	0.7018	20.69

Table 2. Sensitivity analysis across threshold settings on both models. Each configuration is denoted as

(\tau_{c},\tau_{p})

where

\tau_{c}

is the cache threshold and

\tau_{p}

is the prediction threshold.

Panel of Figure 7 — Figure 7. Bounded skip-length sensitivity analysis visualized across runtime and quality metrics for SkyReels-V2 and MAGI-1. A single-column 2 by 2 figure for the bounded skip-length sensitivity analysis. The top-left panel shows Time, the top-right panel shows 1-LPIPS, the bottom-left panel shows SSIM, and the bottom-right panel shows PSNR for SkyReels-V2 and MAGI-1.

Figure 8. Denoising time of Original versus SCOPE at varying video lengths. Left: SkyReels-V2 DF-1.3B (97–737 frames). Right: MAGI-1 4.5B-distill (120–480 frames). Annotations indicate the speedup at each frame count. Two side-by-side bar charts comparing denoising time of Original and SCOPE across different frame counts, with speedup annotations above each pair.

Figure 9. Peak GPU memory comparison under short and long video generation. Left: SkyReels-V2 at 257 and 480 frames. Right: MAGI-1 at 240 and 480 frames. A single-column memory comparison figure with two panels. The left panel shows short-video and long-video peak GPU memory for six methods on SkyReels-V2 at 257 and 480 frames, and the right panel shows the same comparison on MAGI-1 at 240 and 480 frames.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Model	Key Parameter	Value
$\Delta$ -DiT	MAGI-1	cache_interval	8
$\Delta$ -DiT	SkyReels-V2	cache_interval	3
TeaCache-slow	MAGI-1	rel_l1_thresh	0.008
TeaCache-slow	SkyReels-V2	teacache_thresh	0.10
TeaCache-fast	MAGI-1	rel_l1_thresh	0.015
TeaCache-fast	SkyReels-V2	teacache_thresh	0.15

Appendix

Table 3. Baseline configurations on both target models.

Config	Efficiency		Quality
Config	Time (s) $\downarrow$	Speedup $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
No selective	390.12	3.72 $\times$	0.1685	0.8053	24.42
M 0.55 / L 0.35	343.27	4.23 $\times$	0.2198	0.6878	21.42
M 1.00 / L 0.75	318.48	4.56 $\times$	0.1901	0.7302	22.03
M 1.00 / L 1.00	307.34	4.73 $\times$	0.1900	0.7309	21.96

Appendix

Table 4. Stage-ratio study for selective computation on SkyReels-V2.

SkyReels-V2
Predictor	Efficiency		Quality
Predictor	Time (s) $\downarrow$	Speedup $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
auto	329.67	4.41 $\times$	0.1900	0.7313	21.97
momentum	306.70	4.74 $\times$	0.1898	0.7317	21.98
taylor1	306.41	4.74 $\times$	0.1902	0.7308	21.97
taylor2	307.34	4.73 $\times$	0.1900	0.7309	21.96

Appendix

Table 5. Predictor comparison across both target models.

Max Skip	Efficiency		Quality
Max Skip	Time (s) $\downarrow$	Speedup $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
SkyReels-V2
3	473.34	3.07 $\times$	0.1724	0.7586	23.14
4	377.42	3.85 $\times$	0.1803	0.7438	22.48
5	307.34	4.73 $\times$	0.1900	0.7309	21.96
6	268.76	5.40 $\times$	0.2087	0.7124	21.08

Appendix

Table 6. Effect of max_skip_steps on runtime and quality for SCOPE.

Figure 13. Visual comparison on Infinity under the quality-first setting. Zoom in for best viewing. Single-column qualitative comparison on Infinity between the Original baseline and SCOPE.

Method	Time $\downarrow$	Speedup $\uparrow$	CLIPScore $\uparrow$	ImageReward $\uparrow$
Original	3.2511	1.00 $\times$	0.3204	1.4110
SCOPE	2.0791	1.56 $\times$	0.3202	1.3133

Appendix

Table 7. Autoregressive image generation acceleration on Infinity with synchronized timing. Time is reported in seconds per image.

Figure 14. More Visual Comparisons on SkyReels-V2. Zoom in for best viewing. Additional qualitative comparison on SkyReels-V2 across all methods.

Figure 15. More Visual Comparisons on MAGI-1. Zoom in for best viewing. Additional qualitative comparison on MAGI-1 across all methods.

Explore similar work

Oct 2, 2026

TRAC: Trajectory-aware Reuse and Adaptive Correction for Efficient Autoregressive Video Generation

In this paper, we present trajectory-aware reuse and adaptive correction (TRAC), a training-free framework for efficient autoregressive (AR) video generation. Existing acceleration methods mainly target single-trajectory generation with bidirectional attention. AR video generation, by contrast, sequentially couples chunk-level denoising trajectories. Consequently, approximation errors accumulate and propagate through the generation process. TRAC addresses this challenge with three components, including robust cumulative scheduling (RCS), autoregressive trajectory-aware guidance scheduling (ATGS), and spectral structure correction (SSC). RCS selects cache reuse schedules by cumulative rollout error and cross-chunk/prompt variation. ATGS coordinates CFG refreshes along the global AR trajectory. SSC restores low-frequency structure of the first chunk to correct long-term structural loss. Experiments on SkyReels-V2 and FramePack-F1 show that, compared with existing methods, TRAC achieves both the highest inference efficiency and the best generation quality for AR video generation.

Jiaxing Song, Weiqi Yan, You Huang +4

School of Computer Science and Technology, Hainan University · MAC Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University · Zhejiang Normal University, Jinhua, China

Autoregressive Video Generation Inference Acceleration

Ask PDF

Oct 11, 2026

Frequency-Step Cache: Separating Synchronization and Repair for Efficient Autoregressive Video Diffusion

Autoregressive (AR) video generation enables long-horizon synthesis but requires repeated forward passes through large diffusion transformers. Training-free caching reduces this cost by reusing intermediate computation, yet existing controllers often couple two decisions: when to synchronize the cache exactly and how much computation to repair between synchronizations. We introduce Frequency-Step Cache (FSC), which separates these roles. FSC schedules exact full-computation anchors on a dense-to-sparse trajectory, while a lightweight latent-space risk based on frequency-resolved drift and token age controls inter-anchor repair through whole-step replay, selective token recomputation, or full repair. On SkyReels-V2, whole-step reuse provides most of the acceleration, while selective token computation is most effective as fidelity repair. A controlled ablation shows that using one reactive signal for both synchronization and repair reduces fidelity at matched latency, with the same separation advantage observed on FramePack and Matrix-Game. Across multiple autoregressive video backbones, FSC improves speed-quality trade-offs. On SkyReels-V2, FSC-Speed reaches 2.36x speedup versus MotionCache's 2.275x while improving SSIM by 0.019 and PSNR by 0.5 dB. At MotionCache-matched speed, FSC-Quality improves SSIM by 0.031 and PSNR by 1.0 dB and lowers LPIPS by 0.024. Without retuning, FSC-Quality transfers to SkyReels-V2 14B at 2.12x speedup versus 1.79x for MotionCache.

Kerry Xu, Param Misra, Zachary Stenglein +3

Rice University

Diffusion Model Caching Diffusion Model Inference Acceleration Video Diffusion Models Autoregressive Video Generation Autoregressive Diffusion

Ask PDF

Jun 3, 2026

DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation

Video diffusion transformers have achieved state-of-the-art visual quality, but their high inference cost remains a major bottleneck for real-time applications. Recent distillation frameworks produce autoregressive video diffusion models with reduced latency, yet these models still use a fixed number of denoising steps per frame, wasting computation on predictable frames and under-refining challenging ones. We present DSA, a confidence-guided adaptive computation framework for AR video diffusion. DSA introduces a lightweight confidence head, trained jointly with the generator under a distribution-matching distillation objective, to estimate per-frame denoising reliability. At inference, this confidence signal dynamically adjusts the number of diffusion steps: simple frames terminate early for speed, while complex frames receive additional refinement. Our method requires no extra video data, no heuristics, and little architectural modification. Experiments show that DSA achieves real-time autoregressive video generation, reaching 22.63 FPS with sub-second latency on H100 GPUs, while maintaining competitive or superior VBench quality compared to recent autoregressive and bidirectional video diffusion models. Our results demonstrate that confidence-guided adaptive sampling provides an effective and practical path toward interactive video generation.

Thanh-Tung Le, Yunhan Zhao, Menglei Chai +5

University of California, Irvine · Google · Google DeepMind

Autoregressive Diffusion Real-Time Interactive Video Generation Diffusion Sampling Autoregressive Video Generation Diffusion Model Inference Acceleration

Ask PDF