Seeking Physics in Diffusion Noise

Chujun Tang; Lei Zhong; Fangqiang Ding

cs.CVMar 15, 2026

Seeking Physics in Diffusion Noise

Authors: Chujun Tang, Lei Zhong, Fangqiang Ding

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Brown University · University of Edinburgh

Abstract

Do video diffusion models encode signals predictive of physical plausibility? We probe intermediate denoising representations of pretrained Diffusion Transformers (DiTs) and find that physically plausible and implausible videos are partially separable in mid-layer feature space, even at high noise levels. Within-source and perceptual-quality controls suggest that this signal is not fully explained by generator identity or generic visual quality. We distill the signal into a lightweight, backbone-specific physics verifier trained on frozen features and use it in two complementary inference-time mechanisms under a fixed multi-trajectory budget: progressive trajectory selection, which scores trajectories at intermediate checkpoints and prunes weak candidates early, and reward-gradient guidance, which steers surviving trajectories by backpropagating through only the first few DiT blocks. Experiments on PhyGenBench and Physics-IQ across CogVideoX-2B/5B and Wan 2.1-14B show that progressive selection matches verifier-based Best-of-4 on CogVideoX-2B while reducing wall-clock inference time by 37%, whereas reward-gradient guidance substantially improves physical consistency on CogVideoX-5B, all without fine-tuning the video generator.

Figures & tables

Figure 1: Overview of our physics-aware inference framework. For both T2V and I2V generation, a lightweight backbone-specific verifier scores intermediate DiT features. Its predictions are used for reward-gradient guidance and progressive trajectory selection, steering promising trajectories while pruning low-scoring candidates early.

Figure 2: Frozen-feature extraction and physics-verifier training. Given a labeled video–prompt pair, we encode the video into VAE latents, add diffusion noise at timestep $t$ , and pass the noisy latents through a frozen diffusion transformer. Hidden states from a selected layer $\ell$ are extracted, with text tokens removed and video tokens spatially pooled to obtain per-frame features $\mathbf{f}^{(\ell)}_{t}\in\mathbb{R}^{F\times D}$ . A lightweight verifier aggregates these features with causal temporal attention and predicts physical consistency and semantic accuracy, supervised by labels $y^{\mathrm{pc}},y^{\mathrm{sem}}\in\{0,1\}$ using binary cross-entropy loss. The same frozen features are also used for the probing analysis.

Panel of Figure 3 — Figure 3: Source structure in DiT features. (a) UMAP of CogVideoX-2B DiT features extracted after $\ell{=}10$ at $t{=}200$ , colored by the original video generator, reveals strong source clustering. (b) Cross-source probing AUC matrix: within-source evaluation (diagonal, 5-fold CV; mean 0.611) is substantially higher than cross-source transfer (off-diagonal mean 0.538).

Source	# Videos	AUC
LaVie ( Wang et al. 2025c )	528	0.712
VideoCrafter2 ( Chen et al. 2024 )	468	0.631
ZeroScope ( Cerspense 2023 )	612	0.613
Gen-2 ( RunwayML 2023 )	477	0.612
OpenSora ( Zheng et al. 2024 )	610	0.597
SVD ( Blattmann et al. 2023 )	492	0.579

Table 1: Within-source linear probing AUC using CogVideoX-2B ( Yang et al. 2025b ) DiT features. We report results on the seven non-CogVideoX sources in VideoPhy ( Bansal et al. 2024 ) .

Timestep	VAE	$\ell{=}5$	$\ell{=}10$	$\ell{=}15$	$\ell{=}20$	$\ell{=}25$
$t{=}200$	0.539	0.569	0.606	0.604	0.574	0.579
$t{=}400$	0.534	0.545	0.610	0.608	0.624	0.594
$t{=}600$	0.537	0.565	0.638	0.595	0.583	0.585

Table 2: Linear probing AUC (5-fold CV) on CogVideoX-2B DiT features. Every DiT layer improves on VAE probing, by up to

+0.101

(

\ell{=}10

t{=}600

). Best DiT layer per row in bold .

Backbone	Method	Final	S1	S2	S3	Mech.	Opti.	Ther.	Mate.	Pairw.
CogVideoX-2B ( Yang et al. 2025b )	Base Model	0.370 ∗	—	—	—	0.38	0.43	0.34	0.39	—
	Best-of- $N$	0.515	1.96	0.87	1.73	0.50	0.59	0.46	0.48	—
	$+$ Selection (Ours)	0.515	1.98	0.91	1.69	0.49	0.58	0.47	0.49	66.1%
CogVideoX-5B ( Yang et al. 2025b )	Base Model	0.363	1.54	0.58	1.21	0.283	0.493	0.322	0.308	—
	$+$ Selection (Ours)	0.365	1.52	0.53	1.30	0.292	0.456	0.256	0.408	60.2%
	$+$ Reward Gradient	0.496	1.79	0.95	1.71	0.417	0.607	0.456	0.467	66.7%

Table 3: Main results on PhyGenBench across three backbones. ∗ Official single-seed CogVideoX-2B result from ( Meng et al. 2025 ) . Pairwise results compare each method against the corresponding base model. For standard entries, we report the win rate among non-tied comparisons. † For these comparisons, the judge returns ties for nearly all prompts (Wan reward gradient: 157/160; DPO: 153/160 on CogVideoX-5B and 157/160 on Wan). The displayed tie-adjusted rates are therefore provided only for completeness and should not be interpreted as conventional win rates.

Panel of Figure 4 — Figure 4: Qualitative comparison on PhyGenBench prompts (4 uniformly sampled frames per video). Mechanics : baseline pours oil downward under terrestrial gravity; ours forms a floating liquid mass consistent with microgravity. Optics : the texture magnified through the baseline’s lens appears incoherent with the surrounding leaf venation; ours produces a magnified view whose structure is consistent with the underlying leaf surface. Thermal and Physical Properties : baseline shows a liquid stream (incorrect); ours shows rising vapor consistent with sublimation.

Method	Physics-IQ	Spatial IoU	MSE $\downarrow$
Baseline	0.242	0.292	0.0064
$+$ Reward Gradient	0.262	0.317	0.0063

Table 4: Physics-IQ results on CogVideoX-5B-I2V (42 of the 66 scenarios, all three camera perspectives; protocol in the supplementary). The gain concentrates in spatial IoU (

0.292\rightarrow 0.317

), which scores the regions where the scene actually moves.

Verifier	CogX-2B	CogX-5B	Wan 2.1-14B
LikePhys (training-free)	0.554	0.488	0.627
Ours (trained)	0.684	0.660	0.657

Table 5: Verifier AUC on the held-out split of each verifier’s training annotations.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Operation	Output Shape
Input	Spatially pooled per-frame DiT features	$[B,F,D]$
Input Projection	$\mathrm{Linear}(D\rightarrow d)$	$[B,F,d]$
Positional Embedding	Add learnable $\mathbf{p}_{1:F}\in\mathbb{R}^{F\times d}$	$[B,F,d]$
Layer Normalization	Pre-normalization	$[B,F,d]$
Causal Self-Attention	Multi-head ( $h=4$ ) with a causal mask	$[B,F,d]$
Residual Connection	$\tilde{\mathbf{f}}+\mathrm{CausalAttn}(\mathrm{LN}(\tilde{\mathbf{f}}))$	$[B,F,d]$

Appendix

Table S1: Architecture of the proposed physics verifier. The input projection layer is adapted to the hidden dimension

D

of the corresponding backbone, while all remaining components are shared across backbones.

	CogVideoX-2B	CogVideoX-5B	Wan 2.1-14B
Hidden dimension $D$	1920	3072	5120
Number of transformer layers	30	42	40
Text tokens in sequence	Yes (226 tokens)	Yes (226 tokens)	No (cross-attention)
Feature extraction hook	transformer_blocks	transformer_blocks	blocks
Positional encoding	Learned	Rotary (RoPE)	—
Sampling scheduler	DDIM	DDIM	UniPCMultistep

Appendix

Table S2: Backbone-specific configurations of the three DiT backbones evaluated in this work. The physics verifier architecture (

d=256

h=4

) is shared across all settings; only the input projection layer and the feature extraction hook differ across backbones.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$1\times 10^{-3}$
Weight decay	0.01
Batch size	32
Learning-rate schedule	OneCycleLR (10% warmup + cosine annealing)
Early stopping	Patience of 20 epochs on validation AUC

Appendix

Table S3: Training hyperparameters of the physics verifier, shared across all backbones.

Parameter	CogVideoX-2B	CogVideoX-5B	Wan 2.1-14B
Resolution	$480\times 720$	$480\times 720$	$480\times 720$
Number of frames	49 (13 latent)	49 (13 latent)	49 (13 latent)
Sampling scheduler	DDIM	DDIM	UniPCMultistep
Guidance scale	6.0	6.0	6.0
Number of trajectories ( $N$ )	4	4	4
Selection checkpoints ( $\mathcal{C}$ )	$\{600,400\}$	$\{600,400\}$	$\{600,400\}$

Appendix

Table S4: Summary of the video generation configuration for the three backbones evaluated in this work.

Backbone (GPU)	Component	Baseline	Rand. Sel.	Best-of-4	Ours
CogVideoX-2B (A5000 24 GB)	Sampling	183s	454s	732s	448s
	Scoring	—	—	3s	3s
	Decode + offload	60s	60s	60s	60s
	Total	204s	477s	778s	490s
CogVideoX-5B (L40S 48 GB)	Sampling	137s	—	—	357s
	Decode + offload	12s	—	—	12s

Appendix

Table S5: Wall-clock time breakdown per video across backbones and modes.

Method	Final	S1	S2	S3	Time
VideoREPA	0.492	1.87	0.97	1.63	$1\times$
DPO (LoRA)	0.475	1.74	0.78	1.68	$1\times$
$+$ Selection $+$ Reward Gradient (Ours)	0.496	1.79	0.95	1.71	$2.0\times$

Appendix

Table S6: VideoREPA comparison on PhyGenBench (CogVideoX-5B, all methods measured by us under one shared protocol, seed 42, and scored with the same PhyGenEval pipeline). Time is relative single-prompt inference cost.

Panel of Figure 16 — Figure S1: Verifier score analysis. Top: kept vs. dropped trajectory scores at checkpoints. Bottom: per-prompt score spread (max $-$ min) at the first checkpoint. 5B shows meaningful separation; Wan shows near-complete overlap.

Panel of Figure 17 — Figure S2: Per-category score distributions for CogVideoX-5B (top) and Wan 2.1-14B (bottom). On 5B, temporal phenomena (Color, Deposition, Boiling) score highest while spatial/material-specific phenomena (Direct Radiation, Flame Reaction) score lowest. On Wan, categories cluster in $[0.42,0.52]$ .

Panel of Figure 18 — Figure S3: Qualitative comparison on PhyGenBench prompts (Wan-14B, 4 uniformly sampled frames per video). Each row shows baseline (left) vs. guided (right), the generation prompt, and a brief physical analysis.

Panel of Figure 19 — Figure S4: Failure cases (Wan-14B). Both rows show cases where all candidate trajectories are physically incorrect, leaving the verifier unable to select a better outcome.

Physical Consistency in Video Generation

Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.

See all 191 papers

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show
Jun 3, 2026Parsa Esmati, Somjit Nath, Katja Hofmann +3Video Diffusion Models Video Representation Learning
University of Bristol · McGill University · Mila–Quebec AI Institute +2
Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models
Oct 2, 2026Jonas Kneifl, Jakub Skalski, Bartłomiej Twardowski +1Video Diffusion Models Physical Reasoning in Video
IDEAS Research Institute, Warsaw, Poland · Computer Vision Center, Universitat Autònoma de Barcelona, Spain · Warsaw University of Technology, Poland
PhysVid: Physics Aware Local Conditioning for Generative Video Models
Mar 27, 2026Saurabh Pathak, Elahe Arani, Mykola Pechenizkiy +1Physics-Guided Video Generation Physical Consistency in Video Generation
Eindhoven University of Technology

Jun 3, 2026J/K move · Enter open · S save

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show

Parsa Esmati, Somjit Nath, Katja Hofmann, Derek Nowrouzezahrai+2

University of Bristol · McGill University · Mila–Quebec AI Institute+2

Video Diffusion Models Video Representation Learning Physical Reasoning in Video

Open paper AskPDF arXiv

Modern video diffusion models generate increasingly realistic and temporally coherent videos, motivating their use as candidate world simulators. Yet it remains unclear whether these models internally encode physical structure, or merely reproduce motion patterns seen during training. We study this question by probing video diffusion models along latent trajectories corresponding to real videos with known physical plausibility. To obtain such trajectories, we approximately invert the deterministic sampling process by integrating the learned velocity field backward from a clean video latent to noise, giving access to the model's intermediate states and attention maps. Using these recovered trajectories, we show that physical plausibility is linearly decodable from diffusion transformer states across IntPhys and InfLevel, reaching around 81.27% average accuracy and outperforming dedicated representation-learning baselines such as V-JEPA and VideoMAE. Surprisingly, this signal is absent from the VAE latent input and emerges inside the denoising transformer itself, despite the model not being trained with a self-supervised predictive objective. These findings suggest that physically meaningful representations can arise as a byproduct of generative denoising.

Abstract

Physical Consistency in Video Generation

Explore similar work

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show

Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models

PhysVid: Physics Aware Local Conditioning for Generative Video Models