Does Semantic Noise Initialization Transfer from Images to Videos? A Paired Diagnostic Study

Yixiao Jing; Chaoyu Zhang; Zixuan Zhong; Peizhou Huang

cs.CVMar 3, 2026

Does Semantic Noise Initialization Transfer from Images to Videos? A Paired Diagnostic Study

Authors: Yixiao Jing, Chaoyu Zhang, Zixuan Zhong, Peizhou Huang

Organizations: University of Michigan · University College London

Abstract

Semantic noise initialization has been reported to improve robustness and controllability in image diffusion models. Whether these gains transfer to text-to-video (T2V) generation remains unclear, since temporal coupling can introduce extra degrees of freedom and instability. We benchmark semantic noise initialization against standard Gaussian noise using a frozen VideoCrafter-style T2V diffusion backbone and VBench on 100 prompts. Using prompt-level paired tests with bootstrap confidence intervals and a sign-flip permutation test, we observe a small positive trend on temporal-related dimensions; however, the 95 percent confidence interval includes zero (p ~ 0.17) and the overall score remains on par with the baseline. To understand this outcome, we analyze the induced perturbations in noise space and find patterns consistent with weak or unstable signal. We recommend prompt-level paired evaluation and noise-space diagnostics as standard practice when studying initialization schemes for T2V diffusion.

Figures & tables

Dimension	Baseline	NPNet	$\Delta$
aesthetic_quality	0.638083	0.634992	-0.003091
imaging_quality	0.715084	0.707932	-0.007151
background_consistency	0.976592	0.976812	+0.000220
subject_consistency	0.977814	0.978034	+0.000220
temporal_style	0.076961	0.078716	+0.001754

Table 1: Mean VBench scores on 100 prompts.

Metric	Open-Sora2	VideoCrafter
$\lVert z_{g}-z\rVert/\lVert z\rVert$	0.022	0.110
$\cos(z,z_{g})$	0.9997	0.9939
Directional Stability (DirStab)	0.631	0.200
CV ∥d∥	0.064	0.110
Explained Variance Ratio (EVR1)	0.464	0.343

Table 2: Global geometry and directional consistency of golden-induced perturbations for Open-Sora2 and VideoCrafter.

	Open-Sora2			VideoCrafter
Metric	Mean	P10	P90	Mean	P10	P90
$\Delta$ Spatial HF ( $z_{g}-z$ )	-0.00049	-0.00114	0.00006	-0.01489	-0.02086	-0.00745
$\mathrm{sp\_hf}(d)$	0.17996	0.06667	0.27212	0.24804	0.14243	0.42861
$\mathrm{t\_hf}(d)$	0.85502	0.82826	0.87007	0.59898	0.45872	0.73720
$\mathrm{tDiffRel}(d)$	0.60583	0.41060	0.75598	0.54638	0.45229	0.64628

Table 3: Frequency summary: global invariance between

z

and

z_{g}

, and spatiotemporal characteristics of the displacement

d=z_{g}-z

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Qualitative comparison on VideoCrafter. (Full size view) We visualize samples from Baseline (columns 1, 3, 5) versus our NPNet initialization (columns 2, 4, 6). Our method improves visual fidelity and detail consistency (e.g., the fur of the squirrel and the scales of the lizard) without changing the diffusion backbone.

Metric	Baseline	NPNet	Mean $\Delta$	95% CI (bootstrap)	$p$ (perm.)
temporal_style	0.076961	0.078716	+0.001754	[ $-0.000658$ , $0.004166$ ]	0.1687

Appendix

Table 4: Prompt-level paired significance analysis over 100 prompts (seed-averaged). Mean

\Delta

is NPNet minus Baseline.

Metric	Definition / notes
RelDisp	$\\|d\\|_{2}/\\|z\\|_{2}$ on flattened tensors.
CosSim	$\cos(z,z_{g})$ on flattened tensors.
DirStab	Mean pairwise cosine similarity of unit displacements across $S=5$ seeds (Eq. 4 ); computed per prompt, then averaged.
CV ∥d∥	$\mathrm{Std}(\\|d_{s}\\|_{2})/\mathrm{Mean}(\\|d_{s}\\|_{2})$ over seeds; computed per prompt.
EVR1	Top explained-variance ratio from PCA over $\{d_{s}\}_{s=1}^{5}$ (Eq. 5 ); computed per prompt.
sp_hf( $x$ )	Spatial HF power ratio from $\mathrm{rFFT2}$ over $(H,W)$ averaged across $(c,t)$ (Eq. 8 ).

Appendix

Table 5: Metric card for the qualitative noise-space analysis.

Neural Network Initialization

Neural network initialization studies how starting weights shape signal propagation, optimization, generalization, and reproducibility.

See all 48 papers

NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation
Aug 31, 2026Yujiang Pu, Yu KongDiffusion Model Guidance Video Diffusion Models
Michigan State University, East Lansing, MI 48824, USA
On the Resilience of Text-to-Video Diffusion Models to Hardware Faults
Aug 30, 2026Zachary Coalson, A M Aahad, Stella Doehring +2Video Diffusion Models T2V Generation
Oregon State University, Corvallis, OR, USA.
Early Failure Detection and Intervention in Video Diffusion Models
Mar 15, 2026Kwon Byung-Ki, Sohwi Lim, Nam Hyeon-Woo +2Video Diffusion Models T2V Generation
Grad. School of Artificial Intelligence, POSTECH · School of Electrical Engineering, KAIST · Dept. of Electrical Engineering, POSTECH +1

Aug 31, 2026J/K move · Enter open · S save

NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation

Yujiang Pu, Yu Kong

Michigan State University, East Lansing, MI 48824, USA

Diffusion Model Guidance Video Diffusion Models T2V Generation Compositional Video Generation

Open paper AskPDF arXiv

Diffusion models have recently advanced text-to-video (T2V) generation, yet they still struggle with fine-grained compositional alignment, such as attribute binding, spatial relations, and object interactions. While reward-based fine-tuning improves alignment, it is susceptible to reward hacking and adapts poorly to new prompt distributions. In this work, we propose NoisEasier, a test-time scaling framework that improves T2V generation through differentiable reward-guided noise optimization without modifying the underlying model. By combining efficient short-step generators with a multi-objective reward formulation, NoisEasier enables stable and practical test-time optimization under realistic inference budgets. Our key insight is that jointly optimizing the entire stochastic trajectory accelerates reward convergence and improves compositional alignment over optimizing only the initial latent, with negligible additional computational and time cost. Experiments on VBench and T2V-CompBench demonstrate consistent improvements across multiple backbones, achieving over 10% average gains on challenging dimensions such as attribute binding, object interaction, and numeracy. Overall, NoisEasier serves as both a flexible alternative and a complementary enhancement to reward-based fine-tuning, establishing test-time scaling as an effective paradigm for controllable text-to-video generation.

Abstract

Neural Network Initialization

Explore similar work

NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation

On the Resilience of Text-to-Video Diffusion Models to Hardware Faults

Early Failure Detection and Intervention in Video Diffusion Models