Visual AutoRegressive (VAR) models based on next-scale prediction enable efficient hierarchical generation, yet the inference cost grows quadratically at high resolutions. We observe that the computationally intensive later scales predominantly refine high-frequency textures and exhibit substantial spatial redundancy, in contrast to earlier scales that determine the global structural layout. Existing pruning methods primarily focus on high-frequency detection for token selection, often overlooking structural coherence and consequently degrading global semantics. To address this limitation, we propose StepVAR, a training-free token pruning framework that accelerates VAR inference by jointly considering structural and textural importance. Specifically, we employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information. This dual-criterion design enables the model to retain tokens critical for both fine-grained fidelity and overall composition. To maintain valid next-scale prediction under sparse tokens, we further introduce a nearest neighbor feature propagation strategy to reconstruct dense feature maps from pruned representations. Extensive experiments on state-of-the-art text-to-image and text-to-video VAR models demonstrate that StepVAR achieves substantial inference speedups while maintaining generation quality. Quantitative and qualitative evaluations consistently show that our method outperforms existing acceleration approaches, validating its effectiveness and general applicability across diverse VAR architectures.
Figures & tables
Figure 1: Inference Characteristics of Visual Autoregressive Models. (a) The computational bottleneck is heavily skewed towards the final resolution scales, which account for the majority of latency. (b) Sensitivity analysis reveals that early scales are highly sensitive to structural perturbations, whereas later scales exhibit high robustness and error tolerance. (c) Visualizing the residual difference between stages demonstrates that the final scales primarily serve to refine high-frequency textural details.
Figure 2: Overview of the StepVAR Framework. (a) Overall Pipeline: Input tokens are pruned prior to entering the transformer blocks. In the Recover stage, the dense feature map required for the next scale is reconstructed using Nearest Neighbor Feature Propagation. (b) Token Pruning: Our Structure-Texture Guided strategy calculates a joint importance score by weighting a Structural Score ( Sstr ), derived from PCA analysis, and a Textural Score ( Stxt ), derived from spatial high-frequency filter. This ensures the retention of tokens critical for both global layout and local details.
Figure 3: Correlation of first principle component and transformer block residual.
Figure 4: Visual Comparison between original model and our StepVAR generated images.
Methods
Inference Efficiency
GenEval ↑
DPG ↑
# Params
SpeedUp
Latency
Two Obj.
Position
Color Attri.
Overall
Global
Relation
Overall
PixArt-alpha [ 3 ]
0.6B
-
-
0.50
0.08
0.07
0.48
74.97
82.57
71.11
SDXL [ 27 ]
2.6B
-
-
0.74
0.15
0.23
0.55
83.27
86.76
74.65
SD3 [ 7 ]
8B
-
-
0.89
0.34
0.47
0.71
-
-
-
LlamaGen [ 32 ]
0.8B
-
-
0.34
0.07
0.04
0.32
65.16
Show-o [ 39 ]
1.3B
-
-
0.80
0.31
0.50
0.68
-
-
67.48
Table 1: Evaluation on the GenEval and DPG benchmark.
Methods
SpeedUp
Food
Plants
Animals
Overall
FID ↓
CLIP ↑
FID ↓
CLIP ↑
FID ↓
CLIP ↑
FID ↓
CLIP ↑
HART
1.0 ×
30.39
0.28
31.14
0.28
22.35
0.30
11.00
0.283
+FastVAR
1.4 ×
30.92
0.28
36.35
0.27
23.24
0.30
10.52
0.280
+Ours
1.4 ×
30.28
0.28
30.95
0.28
22.00
0.30
10.28
0.283
Infinity
1.0 ×
32.13
0.27
30.49
0.26
25.12
0.29
10.09
0.275
+FastVAR
2.0 ×
31.96
0.27
30.16
0.27
25.24
0.30
10.08
0.278
Table 2: Evaluation on MJHQ-30K Dataset. We report FID and CLIP Score on selected categories and overall performance.
Models
# Params
SpeedUp
Quality
Semantic
Overall
Score
Score
CogVideoX-5B [ 40 ]
5B
-
82.75
77.04
81.61
HunyuanVideo [ 15 ]
13B
-
85.09
75.82
83.24
Emu3 [ 38 ]
8B
-
84.09
68.43
80.96
InfinityStar [ 23 ]
8B
1.0 ×
84.01
83.38
83.88
+Ours
8B
1.4 ×
83.46
82.91
83.35
Table 3: Evaluation on the VBench benchmark.
Table 8
Figure 5: Visual comparison of different recovery strategies.
Visual Autoregressive (VAR) modeling inefficiently applies a fixed computational depth to each position when generating high-resolution images. While existing methods accelerate inference by pruning tokens using frequency maps, their binary hard-pruning approach is fundamentally limited and fails to improve quality even with better frequency estimation. Observing that VAR models possess significant depth redundancy, we propose a paradigm shift from pruning entire tokens to adaptively allocating per-token computational depth. To this end, we introduce DepthVAR, a training-free framework that dynamically allocates computation. It integrates an adaptive depth scheduler, which assigns computational depth via a cyclic rotated schedule for balanced, non-static refinement, with a dynamic inference process that translates these depths into layer-major masks, selectively applies transformer blocks, and blends the resulting codes to ensure each token's influence is proportional to its processing depth. Extensive experiments show that DepthVAR achieves 2.3×-3.1× acceleration with minimal quality loss, offering a competitive compute-performance trade-off compared to existing hard-pruning approaches. Code is available at https://github.com/STOVAGtz/DepthVAR
Chunliang Li, Tianze Cao, Sanyuan Zhao
Beijing Institute of Technology · Beijing Institute of Technology, Zhuhai
Visual Autoregressive (VAR) models deliver high-quality image generation but suffer from significant inference latency at high resolutions. Recent acceleration approaches most rely on heuristic measures with layer features to prune tokens. Such heuristics are sensitive to complex contextual semantics, leading to inaccurate identification of redundant computation and poor adaptability across prompts. We rethink redundancy in VAR from the perspective of its impact on pixel-space generation and introduce Latent Discrepancy. This unified metric quantifies a token's contribution by measuring the change in model states during generation. Our analysis shows that redundancy is more accurately identified when guided by image latent or pixel-space signals. We further observed that in classifier-free guidance (CFG), the convergence trend of the discrepancy between conditional and unconditional branches exhibits high dynamics with different prompts. Based on these findings, we propose LD-Pruning (Latent Discrepancy Pruning), a training-free framework that removes redundancy via latent discrepancy by integrating decoding-free region selection and adaptive unconditional-branch skipping. Extensive experiments show that LD-Pruning substantially reduces inference latency while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
Changwang Mei, Peisong Wang, Zekun Li +7
Nanjing University of Science and Technology · C2DL, Institute of Automation, Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +2
Visual Autoregressive(VAR) models enhance generation quality but face a critical efficiency bottleneck in later stages. In this paper, we present a novel optimization framework for VAR models that fundamentally differs from prior approaches such as FastVAR and SkipVAR. Instead of relying on heuristic skipping strategies, our method leverages attention entropy to characterize the semantic projections across different dimensions of the model architecture. This enables precise identification of parameter dynamics under varying token granularity levels, semantic scopes, and generation scales. Building on this analysis, we further uncover sparsity patterns along three critical dimensions-token, layer, and scale-and propose a set of fine-grained optimization strategies tailored to these patterns. Extensive evaluation demonstrates that our approach achieves aggressive acceleration of the generation process while significantly preserving semantic fidelity and fine details, outperforming traditional methods in both efficiency and quality. Experiments on Infinity-2B and Infinity-8B models demonstrate that ToProVAR achieves up to 3.4x acceleration with minimal quality loss, effectively mitigating the issues found in prior work. Our code will be made publicly available.
Jiayu Chen, Ruoyu Lin, Zihao Zheng +4
School of Computer Science, Peking University · School of Electronics Engineering and Computer Science, Peking University · National Key Laboratory for Multimedia Information Processing, Peking University