Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity

Yucheng Zhou; Jianbing Shen

cs.LGApr 8, 2026

Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity

Authors: Yucheng Zhou, Jianbing Shen

Organizations: SKL-IOTSC, CIS, University of Macau

Abstract

Autoregressive models have shown superior performance and efficiency in image generation, but remain constrained by high computational costs and prolonged training times in video generation. In this study, we explore methods to accelerate training for autoregressive video generation models through empirical analyses. Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos. To address these issues, we propose a Local Optimization (Local Opt.) method, which optimizes tokens within localized windows while leveraging contextual information to reduce error propagation. Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos. ReCo utilizes continuity loss to constrain representation changes, improving model robustness and reducing error accumulation. Extensive experiments on class- and text-to-video datasets demonstrate that our approach achieves superior performance to the baseline while halving the training cost without sacrificing quality.

Figures & tables

Figure 1: Left: Autoregressive models exhibit abrupt representation changes, causing temporal inconsistencies. Right: Our ReCo enforces smooth transitions via continuity loss, yielding temporally consistent videos.

Figure 2: Comparison of AR video model training methods: ( Left ) full-video training vs. ( Right ) fewer-frame training.

Method	FFS	SKY	Train Speed ↑
Baseline	73.65	89.09	0.84
Fewer-Frames	229.32	292.41	1.75 ( $\times$ 2.5)

Table 1: Comparison of FVD performance across methods on the FFS and SKY datasets. ‘Train Speed” is the training time for 24 videos on one A100 GPU ( blue : speedup factor).

Figure 3: Comparison of GPU memory usage for Baseline and Fewer-Frames with different model sizes during training ( Left ) and inference ( Right ), measured on a single A100 GPU with 2 videos.

Figure 4: Examples of video frames generated by the Fewer-Frames Model, showing frames 1, 5, 9, and 13 for two generated videos (top and bottom rows). As the frame number increases, error accumulation leads to noticeable inconsistencies in the appearance of the individuals within the video. Black arrow denotes video progression direction.

Figure 5: Comparison of the Fewer-Frames Model with the Baseline using PSNR ( Left ) and Optical Flow ( Right ), measured across varying source-target frames. The left plot shows the PSNR values (higher is better), i.e., the quality of generated frames, while the right plot presents the Optical Flow values, i.e., the consistency of motion across frames.

Figure 6: The Local Optimization (Local-Opt.) training strategy. During training, a window of frames is randomly selected for optimization (blue), while preceding frames serve as a frozen context (gray), preventing gradient flow.

Figure 7: ( Left ) Comparison of GPU memory usage during training for Baseline and Local-Opt. methods with different model sizes. The memory usage is measured on a single A100 GPU with a batch size of 2. Comparison of the Local-Opt. and Fewer-Frames Model with the Baseline using PSNR ( Middle ) and Optical Flow ( Right ), measured across varying source-target frames.

Method	FFS	SKY	Train Speed ↑
Baseline	73.65	89.09	0.84
Fewer-Frames	229.32	292.41	2.10 ( $\times$ 2.5)
Local-Opt.	190.46	256.94	1.47 ( $\times$ 1.7)

Table 2: Performance and training speed comparison of methods on FFS and SKY datasets.

Figure 8: Loss distribution of different optimization strategies on training and generated samples. ( Top ) Comparison of loss distributions between “Local-Opt.” on its generated samples (“Local-Opt. on Gen.”) and its training samples (“Local-Opt. on Train”). ( Bottom ) Comparison of loss distributions for “Local-Opt.” and “Local-Opt. w/ first frame”, where the first frame of the video is provided as ground truth, on generated samples. The red dashed lines indicate frame blocks corresponding to the spatiotemporal compression of VQVAE.

Figure 9: Loss distribution comparison between “Local-Opt. w/ balanced (first frame balanced)” on generated samples, and "Local-Opt." on training samples.

Method	FFS	SKY	Train Speed ↑
Baseline	73.65	89.09	0.84
Local-Opt.	190.46	256.94	1.47 ( $\times$ 1.7)
Local-Opt. ( w/ first frame )	134.73	186.63	1.47 ( $\times$ 1.7)
Local-Opt. ( w/ balanced )	127.11	179.84	1.68 ( $\times$ 2.0)

Table 3: Performance and training speed comparison of Local-Opt. and its variants with the Baseline on the FFS and SKY datasets.

GAN-based Video Generation Model
Method	FFS	SKY	UCF	Taichi
MoCoGAN ( Tulyakov et al., 2018 )	124.7	206.6	2886.9	-
MoCoGAN-HD ( Tulyakov et al., 2018 )	111.8	164.1	1729.6	128.1
DIGAN ( Yu et al., 2022 )	62.5	83.1	1630.2	156.7
Diffusion-based Video Generation Model
PVDM ( Yu et al., 2023 )	355.9	75.5	1141.9	540.2

Table 4: Performance comparison of video generation models across different datasets.

\bigstar

denotes models with larger parameter sizes (770M).

\spadesuit

denotes video model trained with LARP tokenizer Wang et al. (2025) .

Method	Params	CLIP $\uparrow$	FVD $\downarrow$	Training Cost
Loong ( Wang et al., 2024c )	7B	0.2903	274	1.0
ReCo (Ours)	7B	0.3056	212	1.0
ReCo* (Ours)	7B	0.2911	267	0.5

Table 5: Text-to-video generation results on MSR-VTT (zero-shot).

Figure 10: Loss distribution comparison on generated samples. (Top) “Baseline” vs. “Local-Opt. w/ balanced.” (Bottom) “Baseline” vs. “Ours”.

Figure 11: Comparison of the Local-Opt. and Ours with the Baseline using PSNR ( Left ) and Optical Flow ( Right ), measured across varying source-target frames.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Motion Group	Avg. Optical Flow	Baseline	ReCo (Ours)	Improvement
Low Motion	$<3.0$	215.4	204.6	+5.0%
Medium Motion	$3.0{-}6.0$	542.7	390.2	+28.1%
High Motion	$>6.0$	985.2	815.7	+17.2%

Appendix

Table 6: FVD under different motion dynamics on UCF101.

$\lambda$	FFS (FVD $\downarrow$ )	SKY (FVD $\downarrow$ )	Note
0.0	127.11	179.84	Local-Opt only
0.01	91.42	105.30
0.05	73.85	89.22
0.1	72.60	87.50	Default
0.5	76.93	94.15
1.0	88.24	112.67	Over-regularized

Appendix

Table 7: Sensitivity analysis of the continuity weight

\lambda

Overlap	FFS (FVD $\downarrow$ )	SKY (FVD $\downarrow$ )	Speedup
0%	98.4	112.1	$>2.0\times$
50%	72.6	87.5	$\sim 2.0\times$
75%	71.9	86.9	$<2.0\times$

Appendix

Table 8: Effect of window overlap on performance and training speed.

Method	32 Frames	64 Frames
Baseline (Full Context)	83.5	79.2
ReCo (Ours)	78.8	71.1

Appendix

Table 9: Long video generation results on SkyTimelapse.

Figure 12: More Error Cases from Fewer-Frames model.

Autoregressive Video Generation

Autoregressive video generation creates videos frame by frame or in chunks, conditioning each prediction on prior context for streaming and interactive use.

See all 211 papers

FastSTAR: Spatiotemporal Token Pruning for Efficient Autoregressive Video Synthesis
Mar 7, 2026Sungwoong Yune, Suheon Jeong, Joo-Young KimVideo Generation Autoregressive Video Generation
Korea Advanced Institute of Science and Technology
Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolation
Apr 3, 2026Hanshuai Cui, Zhiqing Tang, Zhi Yao +3Autoregressive Video Generation Diffusion Model Inference Acceleration
School of Artificial Intelligence, Beijing Normal University, Beijing, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai, China · Shenzhen University of Advanced Technology, Shenzhen, China
TRAC: Trajectory-aware Reuse and Adaptive Correction for Efficient Autoregressive Video Generation
Oct 2, 2026Jiaxing Song, Weiqi Yan, You Huang +4Autoregressive Video Generation Inference Acceleration
School of Computer Science and Technology, Hainan University · MAC Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University · Zhejiang Normal University, Jinhua, China

Mar 7, 2026J/K move · Enter open · S save

FastSTAR: Spatiotemporal Token Pruning for Efficient Autoregressive Video Synthesis

Sungwoong Yune, Suheon Jeong, Joo-Young Kim

Korea Advanced Institute of Science and Technology

Video Generation Autoregressive Video Generation Token Pruning Visual Token Pruning

Open paper AskPDF arXiv

Visual Autoregressive modeling (VAR) has emerged as a highly efficient alternative to diffusion-based frameworks, achieving comparable synthesis quality. However, as this paradigm extends to Spacetime Autoregressive modeling (STAR) for video generation, scaling resolution and frame counts leads to a "token explosion" that creates a massive computational bottleneck in the final refinement stages. To address this, we propose FastSTAR, a training-free acceleration framework designed for high-quality video generation. Our core method, Spatiotemporal Token Pruning, identifies essential tokens by integrating two specialized terms: (1) Spatial similarity, which evaluates structural convergence across hierarchical scales to skip computations in regions where further refinement becomes redundant, and (2) Temporal similarity, which identifies active motion trajectories by assessing feature-level variations relative to the preceding clip. Combined with a Partial Update mechanism, FastSTAR ensures that only non-converged regions are refined, maintaining fluid motion while bypassing redundant computations. Experimental results on InfinityStar demonstrate that FastSTAR achieves up to a 2.01x speedup with a PSNR of 28.29 and less than 1% performance degradation, proving a superior efficiency-quality trade-off for STAR-based video synthesis.

Abstract

Autoregressive Video Generation

Explore similar work

FastSTAR: Spatiotemporal Token Pruning for Efficient Autoregressive Video Synthesis

Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolation

TRAC: Trajectory-aware Reuse and Adaptive Correction for Efficient Autoregressive Video Generation