Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling

Ibne Farabi Shihab; Abu Sa-Adat Mohamed Moon-Im Al Ahsan; Anuj Sharma

cs.LGAug 24, 2026

Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling

Authors: Ibne Farabi Shihab, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Anuj Sharma

Organizations: Department of Computer Science, Iowa State University · Department of Computer Science & Engineering, BRAC University · Department of Civil, Construction & Environmental Engineering, Iowa State University

Abstract

Diffusion inference repeatedly evaluates the same large network. We ask whether pretrained weights alone can identify residual branches that need not be recomputed throughout the trajectory. Our Spectral Concentration Ratio (SCR) measures leading-versus-tail singular-value energy. Combined with Frobenius magnitude, it yields an offline sensitivity proxy and a deterministic lifetime for each scheduled unit. A frozen unit reuses its cached residual-branch update while the current residual stream and all external conditioning continue to propagate. The method needs no router, calibration prompts, or input-dependent search. At matched layer-step budgets, SCR/Frobenius preserves quality better than random, depth, norm, stable-rank, and Frobenius--stable-rank schedules on LLaDA-8B, DiT-XL/2, U-ViT-L, and SDXL. Broader LLaDA tests cover retrieval, reasoning, code, summarization, and open-ended generation; matched-horizon controls retain the ranking down to ten denoising steps. The complete captured-graph system reaches $2.8\times$ -- $3.0\times$ wall-clock speedup over eager inference. This is a systems-level number: on LLaDA, padded graph execution already gives $2.7\times$ , while eliminating inactive branch work raises it to $3.0\times$ . The perturbation analysis motivates pre-norm attention and MLP components under explicit local assumptions; results on AdaLN, U-shaped, convolutional, and cross-attention blocks are empirical transfer, not certified guarantees.

Figures & tables

Panel of Figure 1 — Figure 1: Spectral-Guided Scheduling. Offline weight spectra produce scores $\widehat{\Lambda}_{\ell}$ and lifetimes $S_{\ell}$ . Online, each frozen residual unit reuses a cached branch update; the current residual stream and external inputs still propagate.

Model	Main block type	Conditioning	Theory coverage	Cached update / current signal
LLaDA-8B	transformer diffusion LM	RMSNorm / timestep conditioning	direct for pre-norm branches	attention/MLP update; current residual stream
DiT-XL/2	diffusion transformer	AdaLN / class conditioning	heuristic for AdaLN	attention/MLP update; current residual stream
U-ViT-L	U-shaped ViT	skip connections / patch features	partial	group update; current main stream and skip input
SDXL	latent UNet	conv, cross-attn, residual blocks	heuristic	valid residual/attention update; current residual and UNet skips

Table 1: Scope and executor semantics. “Direct” means the local argument covers the stated pre-norm branch under its assumptions; it is not a global certificate.

Execution mode	Layer-steps	Latency	Speedup
Full model, eager	3200	$450\pm 6$ ms	1.0 $\times$
Python skipping	1120	$238\pm 7$ ms	1.9 $\times$
Padded static graph	3200	$166\pm 5$ ms	2.7 $\times$
Jagged-stream executor	1120	$150\pm 4$ ms	3.0 $\times$

Table 2: LLaDA execution ablation. The

3.0\times

result combines graph execution and branch skipping relative to eager full inference.

Model	Architecture	Task	Full latency	Scheduled latency	Speedup	Quality change
LLaDA-8B	diffusion LM	GSM8K / Needle	$450\pm 6$ ms	$150\pm 4$ ms	3.0 $\times$	$-0.2$ GSM8K / $-0.2$ Needle
DiT-XL/2	diffusion transformer	ImageNet-256	$1.82\pm 0.03$ s	$0.61\pm 0.01$ s	3.0 $\times$	$+0.08$ FID
U-ViT-L	U-ViT	ImageNet-256	$1.24\pm 0.02$ s	$0.45\pm 0.01$ s	2.8 $\times$	$+0.07$ FID
SDXL	latent UNet	COCO	$4.80\pm 0.05$ s	$1.60\pm 0.03$ s	3.0 $\times$	$+0.25$ COCO FID

Table 3: Cross-model operating points. Speedup is complete-system wall time versus eager full inference; quality change uses the same sampler and evaluation setting.

Model	Metric	Layer-steps	Speedup	Full	Scheduled
LLaDA-8B	GSM8K Acc.	35%	3.0 $\times$	$68.3\pm 0.5$	$68.1\pm 0.5$
DiT-XL/2	FID $\downarrow$	34%	3.0 $\times$	$2.27\pm 0.03$	$2.35\pm 0.04$
U-ViT-L	FID $\downarrow$	36%	2.8 $\times$	$3.08\pm 0.04$	$3.15\pm 0.05$
SDXL	COCO FID $\downarrow$	32%	3.0 $\times$	$7.84\pm 0.07$	$8.09\pm 0.10$

Table 4: Untuned transfer. Hyperparameters are fixed across models. Layer-step ratios and wall-clock speedups are distinct quantities.

Ranking score	LLaDA GSM8K $\uparrow$	DiT FID $\downarrow$	U-ViT FID $\downarrow$	SDXL COCO FID $\downarrow$
Random	$63.4\pm 0.8$	$3.21\pm 0.08$	$4.02\pm 0.09$	$9.45\pm 0.16$
Depth-only	$66.2\pm 0.6$	$2.87\pm 0.06$	$3.61\pm 0.07$	$8.88\pm 0.13$
Frobenius norm	$67.2\pm 0.5$	$2.74\pm 0.06$	$3.44\pm 0.06$	$8.64\pm 0.12$
Spectral norm	$66.8\pm 0.6$	$2.69\pm 0.05$	$3.39\pm 0.06$	$8.57\pm 0.12$
Stable rank	$67.5\pm 0.5$	$2.61\pm 0.05$	$3.31\pm 0.06$	$8.43\pm 0.11$
Frobenius $\times$ stable rank	$67.6\pm 0.5$	$2.55\pm 0.05$	$3.27\pm 0.05$	$8.34\pm 0.11$

Table 5: Static rankings at matched layer-step budgets. Larger raw scores receive longer lifetimes after orienting every baseline toward measured sensitivity.

Method	Steps	Std. latency	Needle	GSM8K	CNN/DM	Needle-4K	MATH	HumanEval	MAUVE
Full	3200	$450\pm 6$ ms	$100.0\pm 0.0$	$68.3\pm 0.5$	$42.1\pm 0.2$	$99.6\pm 0.1$	$28.6\pm 0.6$	$32.9\pm 0.7$	$.921\pm.006$
Random	1120	$150\pm 5$ ms	$81.2\pm 1.0$	$63.4\pm 0.8$	$38.2\pm 0.4$	$77.9\pm 1.1$	$22.1\pm 0.8$	$25.7\pm 0.9$	$.846\pm.010$
Depth	1120	$150\pm 5$ ms	$90.1\pm 0.8$	$66.2\pm 0.6$	$40.1\pm 0.3$	$87.8\pm 0.9$	$25.9\pm 0.7$	$29.8\pm 0.8$	$.884\pm.008$
Spectral	1120	$\mathbf{150\pm 4}$ ms	$\mathbf{99.8\pm 0.1}$	$\mathbf{68.1\pm 0.5}$	$\mathbf{41.9\pm 0.2}$	$\mathbf{99.2\pm 0.2}$	$\mathbf{28.2\pm 0.6}$	$\mathbf{32.4\pm 0.7}$	$\mathbf{.916\pm.006}$
Oracle replay	$930\pm 35$	$125\pm 4$ ms	$100.0\pm 0.0$	$68.3\pm 0.5$	$42.1\pm 0.2$	–	–	–	–

Table 6: LLaDA-8B language evaluation. “Needle” is the original setting; “Needle-4K” uses the 4096-token limit. Standard latency does not represent the max-context run.

$T$	LLaDA Full	Spectral	Depth	Random	DiT Spectral	DiT Depth	DiT Random
100	$68.3\pm.5$	$\mathbf{68.1\pm.5}$	$66.2\pm.6$	$63.4\pm.8$	$\mathbf{+.08\pm.03}$	$+.60\pm.04$	$+.94\pm.05$
50	$67.6\pm.6$	$\mathbf{67.3\pm.6}$	$64.9\pm.7$	$61.9\pm.8$	$\mathbf{+.11\pm.03}$	$+.67\pm.04$	$+.98\pm.05$
30	$66.4\pm.6$	$\mathbf{66.0\pm.6}$	$62.8\pm.7$	$59.4\pm.9$	$\mathbf{+.16\pm.04}$	$+.82\pm.05$	$+1.17\pm.05$
20	$64.9\pm.7$	$\mathbf{64.3\pm.7}$	$60.7\pm.8$	$56.8\pm.9$	$\mathbf{+.23\pm.04}$	$+1.03\pm.05$	$+1.39\pm.06$
10	$59.2\pm.9$	$\mathbf{58.1\pm.9}$	$53.6\pm 1.0$	$49.7\pm 1.1$	$\mathbf{+.39\pm.05}$	$+1.36\pm.06$	$+1.76\pm.06$

Table 7: Matched-ratio short-horizon control. DiT entries are

\Delta

FID from Full at the same

T

; lower is better.

Method	Signal	LLaDA ms / GSM8K $\uparrow$	DiT s / FID $\downarrow$	SDXL s / FID $\downarrow$
Spectral static	weights, offline	$\mathbf{150\pm 4}/68.1\pm.5$	$\mathbf{.61\pm.01}/2.35\pm.04$	$\mathbf{1.60\pm.03}/8.09\pm.10$
Activation drift	per-input drift	$171\pm 6/\mathbf{68.2\pm.5}$	$.69\pm.02/\mathbf{2.32\pm.04}$	$1.78\pm.04/\mathbf{8.02\pm.10}$
Oracle replay	full-output agreement	$125\pm 4/68.3\pm.5$	–	–

Table 8: Static versus input-aware scheduling at matched per-step active counts. Quality differences are within displayed uncertainty.

Method	DiT latency	DiT FID $\downarrow$	U-ViT latency	U-ViT FID $\downarrow$
Full Model	$1.82\pm 0.03$ s	$2.27\pm 0.03$	$1.24\pm 0.02$ s	$3.08\pm 0.04$
DDIM, 50 steps	$0.91\pm 0.02$ s	$3.14\pm 0.08$	–	$3.91\pm 0.09$
DPM-Solver++	$0.38\pm 0.01$ s	$2.51\pm 0.05$	–	$3.42\pm 0.07$
DeepCache	$0.42\pm 0.01$ s	$2.68\pm 0.06$	–	$3.34\pm 0.06$
$\Delta$ -DiT	$1.13\pm 0.02$ s	$2.41\pm 0.05$	–	–
Random static	$0.61\pm 0.01$ s	$3.21\pm 0.08$	–	$4.02\pm 0.09$

Table 9: ImageNet-256 results. Separate latency columns prevent DiT timings from being read as U-ViT timings.

Panel of Figure 2 — Figure 2: Representative LLaDA lifetime map. Colored groups are active; gray groups reuse cached branch updates. Full four-model maps appear in Figure 5 .

Model / batch	Full	Sched.	Cache	Increase
LLaDA / 1	18.9	22.2	3.0	$17\%$
DiT / 16	18.7	20.8	1.8	$11\%$
U-ViT / 16	14.9	16.4	1.3	$10\%$
SDXL / 8	39.8	48.5	8.2	$22\%$

Table 10: Peak H100 memory in GiB. Variation is at most

\pm 0.1

GiB.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Latency	COCO FID $\downarrow$
Full SDXL	$4.80\pm 0.05$ s	$7.84\pm 0.07$
DPM-Solver++	$1.92\pm 0.04$ s	$8.21\pm 0.11$
DeepCache	$2.20\pm 0.04$ s	$8.56\pm 0.12$
Block Caching	$2.00\pm 0.04$ s	$8.42\pm 0.11$
Depth static	$1.60\pm 0.03$ s	$8.88\pm 0.13$
Spectral static	$\mathbf{1.60\pm 0.03}$ s	$\mathbf{8.09\pm 0.10}$

Appendix

Table 11: SDXL COCO text-to-image validation. Bold indicates the best standalone static schedule; composed methods are not bolded.

Model	$\beta_{\mathrm{SCR}}$	Std. err.	$p$ -value	$R^{2}_{\mathrm{depth}}$	$R^{2}_{\mathrm{full}}$
LLaDA-8B	0.42	0.08	$<10^{-3}$	0.31	0.49
DiT-XL/2	0.36	0.07	$<10^{-3}$	0.28	0.42
U-ViT-L	0.31	0.08	$4.0\times 10^{-3}$	0.25	0.36
SDXL	0.39	0.09	$2.0\times 10^{-3}$	0.22	0.38

Appendix

Table 12: Depth-controlled regression of measured freeze error on SCR, layer depth, block type, and Frobenius norm.

Panel of Figure 3 — Figure 3: Measured layer-freeze error versus the static spectral score. Each point is a scheduled layer or block group from the freeze-sensitivity sweep that underlies Table 18 ; Spearman correlations appear in the panel titles.

Panel of Figure 4 — Figure 4: Quality–latency tradeoff across architectures. Circles are measured Spectral-Guided runs at different $\tau$ values; squares are directly reproduced baselines at their default operating points. Coordinates appear in Table 19 .

Panel of Figure 5 — Figure 5: Full displayed lifetime maps. These visual groups omit multiplicities and must not be averaged to reconstruct layer-step budgets.

Model	Dataset	Samples	Resolution	Precision	Batch	Hardware	Primary metric
LLaDA-8B	Needle / GSM8K / CNN/DM / MATH-500 / HumanEval / WikiText-103	full official splits; 500 MATH problems; 164 code tasks; held-out continuation prompts	text	bf16	1	H100 80GB	accuracy / exact match / pass@1 / ROUGE-L / MAUVE
DiT-XL/2	ImageNet-256	50k samples	$256^{2}$	bf16	16	H100 80GB	FID
U-ViT-L	ImageNet-256	50k samples	$256^{2}$	bf16	16	H100 80GB	FID
SDXL	COCO validation prompts	30k samples	$1024^{2}$	fp16	8	H100 80GB	COCO FID

Appendix

Table 13: Evaluation protocol. Within each task, methods share prompts, generation lengths, denoising settings, and metric implementations.

Family	Method	Steps	Sampler / order	Guidance	Budget / status
LLaDA-8B	Full	100	default deterministic	n/a	3200 layer-steps; matched
LLaDA-8B	DDIM / reduced	50	DDIM	n/a	full layers; exploratory
LLaDA-8B	DPM-Solver++	20	second order	n/a	full layers; exploratory
LLaDA-8B	Static	100	default deterministic	n/a	1120 layer-steps; matched
DiT-XL/2	Full / static	100	default	cfg 1.5	full / 34%; matched
DiT-XL/2	DDIM / DPM-Solver++	50 / 20	DDIM / second order	cfg 1.5	full layers; matched

Appendix

Table 14: Baseline and sampler settings. Static rows match the spectral layer-step budget. LLaDA solver rows are retained as exploratory results but excluded from matched claims because their masked-denoising adaptation is insufficiently specified.

Method	Latency	Needle	GSM8K	CNN/DM
DDIM / reduced	$225\pm 4$ ms	$94.2\pm.7$	$65.7\pm.6$	$40.8\pm.3$
DPM-Solver++	$90\pm 3$ ms	$96.1\pm.6$	$67.2\pm.5$	$41.4\pm.2$

Appendix

Table 15: Exploratory LLaDA reduced-step results retained for completeness but excluded from matched claims.

Method	Status	Used in matched tables	Notes
DDIM / DPM-Solver++ (vision)	direct	yes	same checkpoints, precision, resolution, and batch size
DDIM / DPM-Solver++ (LLaDA)	direct, exploratory	no	masked-denoising adaptation not sufficiently documented
DeepCache	direct	yes	compatible image models with method-default cache interval
$\Delta$ -DiT	direct for DiT	yes	unavailable for U-ViT row
Random / depth static	direct	yes	matched layer-step budget to Spectral-Guided
SmoothCache	literature positioning	no	different public settings

Appendix

Table 16: Baseline status. Literature rows are positioning only and are not mixed into matched-compute claims.

Model	Representative active-iteration coordinates by increasing depth
LLaDA-8B, $T=100$	25	31	38	50	63	75	88	100
DiT-XL/2, $T=100$	31	38	44	56	69	81	94	100
U-ViT-L, $T=100$	38	44	56	69	81	88	94	100
SDXL, $T=50$	13	16	19	25	31	38	44	50

Appendix

Table 17: Representative display lifetimes for Figure 2 . Entries omit group multiplicities and are not budget-reconstructible; exact ratios use the unbinned

\sum_{\ell}S_{\ell}/(LT)

Model	Units $n$	Spearman $\rho$	$p$ -value	Error proxy
LLaDA-8B	192	0.76	$<10^{-40}$	normalized denoiser-output deviation
DiT-XL/2	168	0.70	$<10^{-28}$	normalized latent-output deviation
U-ViT-L	132	0.66	$<10^{-18}$	normalized latent-output deviation
SDXL	284	0.71	$<10^{-45}$	normalized UNet-output deviation

Appendix

Table 18: Score–freeze-error correlation data underlying Figure 3 . Freeze error is measured by freezing one scheduled layer or block group at the default operating point and recording the normalized output deviation or task-loss proxy.

Model	$\tau$	Speedup	Metric	Value
LLaDA-8B	full	1.0 $\times$	GSM8K $\uparrow$	68.3
LLaDA-8B	0.5	1.5 $\times$	GSM8K $\uparrow$	68.2
LLaDA-8B	1.0	3.0 $\times$	GSM8K $\uparrow$	68.1
LLaDA-8B	2.0	4.5 $\times$	GSM8K $\uparrow$	65.5
DiT-XL/2	full	1.0 $\times$	FID $\downarrow$	2.27
DiT-XL/2	0.5	1.5 $\times$	FID $\downarrow$	2.28

Appendix

Table 19: Coordinates for the Spectral-Guided Pareto curves in Figure 4 . Each point is a measured run at the corresponding

\tau

; no curve smoothing or interpolation is used.

Model family	Failure mode	Likely cause
Diffusion LM	arithmetic carry / boundary retrieval	late high-sensitivity layers frozen too early
DiT / U-ViT	fine texture loss	high-frequency reconstruction suppressed
SDXL UNet	small text / object count errors	cross-attention and decoder under-updated

Appendix

Table 20: Failure modes at aggressive schedules.

Method	Params	ROUGE-L	Speedup
Full fine-tuning	8.0B	42.1	1.0 $\times$
LoRA, $r=16$	26M	41.4	1.5 $\times$
SCR-freeze bottom 40%	4.8B	41.9	1.6 $\times$
Random-freeze bottom 40%	4.8B	39.8	1.6 $\times$

Appendix

Table 21: SCR-guided freezing during LLaDA-8B fine-tuning on CNN/DailyMail.

Model	$\rho(\widehat{\Lambda},B^{\mathrm{emp}})$	$\rho(\widehat{\Lambda},Q)$	$\mathrm{CV}(Q)$	Drift $R^{2}$	Weight-only	$B^{\mathrm{emp}}$ order
LLaDA-8B	.91	.12	.29	.94	68.1	68.2
DiT-XL/2	.88	.09	.26	.92	2.35	2.33
U-ViT-L	.84	.06	.33	.90	3.15	3.13
SDXL	.79	$-.04$	.41	.86	8.09	8.04

Appendix

Table 22: Audit of non-weight terms. The

B^{\mathrm{emp}}

schedule requires calibration trajectories; task values are GSM8K for LLaDA and FID otherwise.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 509 papers

Oct 2, 2026

SpectralCache: Accelerating Diffusion-Based World Models via Spectral Feature Caching

Diffusion-based world models enable high-quality interactive environment generation but suffer from substantial inference overhead due to repeated Transformer evaluations during denoising. Existing caching methods mainly exploit temporal redundancy at the feature or token level, leaving the underlying mathematical structure of diffusion features largely unexplored. In this work, we reveal that world-model features exhibit highly stable singular subspaces across nearby denoising steps, while their singular values follow predictable evolution patterns. Building on this observation, we propose SpectralCache, a training-free spectral caching framework that reuses stable singular subspaces and estimates only low-dimensional singular values through linear extrapolation. We further exploit the spectral consistency between neighboring full-computation features to skip selected expensive backbone evaluations via singular value scaling. Extensive experiments on representative world models demonstrate that SpectralCache consistently improves inference efficiency while preserving generation quality. On HunyuanWorld-Voyager-13B, SpectralCache achieves 5.22x acceleration while maintaining a WorldScore of 65.90 for static scenes, substantially outperforming existing training-free caching methods in inference efficiency.

Zhendong Mi, Pu Zhao, Ziyu Hu +4

Stevens Institute of Technology · Northeastern University · Technische Universität Darmstadt

Diffusion Model Caching Diffusion Model Inference Acceleration World Model Learning

Ask PDF

Mar 2, 2026

Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration

Diffusion models have become the dominant tool for high-fidelity image and video generation, yet are critically bottlenecked by their inference speed due to the numerous iterative passes of Diffusion Transformers. To reduce the exhaustive compute, recent works resort to the feature caching and reusing scheme that skips network evaluations at selected diffusion steps by using cached features in previous steps. However, their preliminary design solely relies on local approximation, causing errors to grow rapidly with large skips and leading to degraded sample quality at high speedups. In this work, we propose spectral diffusion feature forecaster (Spectrum), a training-free approach that enables global, long-range feature reuse with tightly controlled error. In particular, we view the latent features of the denoiser as functions over time and approximate them with Chebyshev polynomials. Specifically, we fit the coefficient for each basis via ridge regression, which is then leveraged to forecast features at multiple future diffusion steps. We theoretically reveal that our approach admits more favorable long-horizon behavior and yields an error bound that does not compound with the step size. Extensive experiments on various state-of-the-art image and video diffusion models consistently verify the superiority of our approach. Notably, we achieve up to 4.79

\times

speedup on FLUX.1 and 4.67

\times

speedup on Wan2.1-14B, while maintaining much higher sample quality compared with the baselines.

Jiaqi Han, Juntong Shi, Puheng Li +3

Stanford University · ByteDance

Video Diffusion Models Diffusion-Based Image Generation Diffusion Model Caching Diffusion Model Inference Acceleration

Ask PDF

Oct 8, 2026

CacheScope: Exact-Compute Allocation for Training-Free Diffusion Acceleration

Training-free feature reuse accelerates diffusion transformer inference by replacing full denoising computation at selected steps with cached approximations. Existing methods mainly improve the reconstruction of skipped features, but we find that once a stable first-order reconstructor is used, increasing reconstruction complexity brings limited gains, whereas the placement of exact-computation steps has a much larger effect on fidelity. More importantly, absolute shallow-probe variation is jointly affected by input-trajectory displacement and network response strength, and therefore does not directly reflect cache risk. We address this confounding by normalizing probe-output variation with the corresponding input displacement, yielding a directional gain per unit trajectory movement. Based on this signal, we propose CacheScope, which identifies high-risk reuse steps online and selectively falls back to full denoising computation while leaving the underlying cache reconstructor unchanged. CacheScope requires no additional training, gradients, auxiliary networks, or prompt-specific offline schedule optimization. Across FLUX.1-dev, Wan2.1-T2V-1.3B, and Open-Sora 1.2, CacheScope achieves up to 3.57x acceleration and reduces median LPIPS by up to 70.1% over the strongest probe-based baseline.

Qinhan Lyu, Jusheng Zhang, Zhixing Zhang +4

Sun Yat-sen University

Adaptive Computation for Diffusion Models Diffusion Model Caching Diffusion Model Inference Acceleration Diffusion Models Diffusion Timestep Scheduling

Ask PDF

Abstract

Diffusion Model Inference Acceleration

Explore similar work

SpectralCache: Accelerating Diffusion-Based World Models via Spectral Feature Caching

Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration

CacheScope: Exact-Compute Allocation for Training-Free Diffusion Acceleration