Recent studies report that video diffusion models "know more than they show": physical properties decode better from their activations than from their inputs or outputs. But a diffusion model computes its clean-sample estimate from those same activations, so such a comparison measures how easily a readout extracts the information. Whether the model hides anything is a separate question. We set out to find hidden physical knowledge in kinematic text-to-motion diffusion models, where the physics of every intermediate estimate can be computed exactly, and recorded every layer along the sampling trajectories of MDM (two checkpoints) and MotionDiffuse. Against the usual baselines the layers do look ahead of the model's own estimate on balance and flight. With every readout tuned the same way, the estimate closes the gap: for balance and flight, adding a layer gains at most 0.01, a size that a positive control shows we would detect. The fall rate of jumps is the one clear exception. Early in denoising, MDM's layers add about 0.08 in R2 (up to 0.11 in a refit) that the estimate does not yet show, and the gain fades as sampling proceeds; on MotionDiffuse it is small and not robust. The estimate also flags floating samples without training, but aborting them saves under a tenth of the compute. In a video model, plausibility that linear probes miss in Wan 2.1's clean latent becomes readable after one fixed nonlinearity.
Figures & tables
Figure 1: The model’s own clean-motion estimate already shows the physics of the final motion. Views from the side and slightly above of two recorded samples from the 50-step MDM, drawn from its clean-motion estimate at 20% and 40% of denoising (top and middle rows) and from the final motion (bottom row). (a) A jump: the center of mass (line, red while airborne) already follows a ballistic arc at 20%; the fitted fall rate is printed above each row. (b) A slow walk that floats: red poses lie in floating stretches (airborne for at least 0.2 s with a fitted fall rate below 2 m/s 2 ), and the numbers give the lowest foot’s height. The estimate at 20% already floats where the final motion does. Poses in (b) are spread horizontally for readability. Bodies are SMPL meshes ( Loper et al., 2015 ) fitted to the generated joints for display ( Bogo et al., 2016 ) , with a mean joint error of 1.0 to 1.2 cm; shadows are vertical projections onto the floor. Every number in the figure is computed from the joints.
Figure 2: What we compare, at one denoising step. The model receives the noisy motion xt and the text u ; its layers produce activations aℓ , and its output gives the clean-motion estimate x^0 , from which joints and every label follow. Two readouts are trained to predict a label y of the final motion x0 (here the fall rate of a jump): one from the estimate alone (teal), one from the estimate and layer ℓ together (deep blue). The gain from the layer is the difference in their scores ( R2 or AUC), which for R2 measures the conditional usable information (Proposition 1 ). Drawn on a recorded jump of the 50-step MDM at 20% of denoising: the estimate (center) already shows the jump of the final motion (right; center of mass red while airborne). Bottom left: where the two models differ (Section 3.1 ). The grid for the noisy input is schematic.
Figure 3: How the balance and gravity labels are computed, on recorded final motions of the 50-step model. (a) A walk at its most and least balanced frames: the support region on the floor (gray), the center of mass projected to the floor (dark dot) and the extrapolated center of mass (XCoM; green inside the support, red outside). (b) Over the same window, the dynamic margin is negative through most of each step. Dots mark the two frames in (a). (c) A jump: the parabola fitted to the center-of-mass height over the airborne stretch falls at close to 9.81 m/s 2 . (d) A floating walk: the body is unsupported, with the lowest foot 6 to 11 cm up (median 9 cm), but its center of mass barely falls, so g^ is below the 2 m/s 2 cut-off; inset: one floating frame. Bodies are SMPL meshes fitted to the joints for display, as in Figure 1 .
Raised contact ( > 5 cm), % of contact frames, test captions
2.9–3.3
4.9–5.4
Table 1: Physics of the generated motions (range over seeds; 1,000 motions per seed). Jump gravity is the median fitted fall rate of unsupported airborne stretches that do fall (real flight: 9.81 m/s 2 ); floating counts unsupported stretches that barely fall. Here “jump prompts” are the 14 curated jump prompts of the main runs (Section 3.1 ); elsewhere the term means the jump-heavy runs.
Figure 4: Probe scores across denoising for the 50-step model (lines: mean over three seeds; bands: range). The best layer (deep blue) reads the final motion’s dynamic balance and flight early, and so does an MLP reading everything the model’s own estimate shows (teal diamonds, measured at 20% and 40%; bars: standard deviation). In most figures, teal marks readouts of the estimate, deep blue the model’s layers, gray the context, and red the physics of a motion (airborne arcs, floating).
Model
Target
Best layer
Explicit
Estimate, linear
Noisy input
50-step
Dynamic balance ( R2 )
0.838 ± 0.001
0.783 ± 0.009
0.723 ± 0.007
0.383 ± 0.006
Static balance ( R2 )
0.759 ± 0.014
0.769 ± 0.019
0.673 ± 0.014
0.241 ± 0.007
Lowest foot ( R2 )
0.868 ± 0.022
0.921 ± 0.005
0.817 ± 0.029
0.396 ± 0.022
In flight (AUC)
0.973 ± 0.002
0.876 ± 0.005
0.935 ± 0.008
0.801 ± 0.010
Flight gravity ( R2 )
0.820 ± 0.035
0.707 ± 0.079
0.547 ± 0.053
0.209 ± 0.046
1000-step
Dynamic balance ( R2 )
0.697 ± 0.029
0.655 ± 0.008
0.550 ± 0.031
0.175 ± 0.003
Table 2: The standard comparison at 20% of denoising (mean ± standard deviation over seeds). Best layer: the best of the eight layers per seed, linear probe. Explicit: the label computed from the model’s clean-motion estimate. Estimate, linear: a linear probe on the estimate’s 263 features. Noisy input: a linear probe on xt .
Tuned on held-out prompts
Gain from adding the layer [95% CI]
Target
Model
Explicit
Layer, ridge
Estimate
Layer
Both
Pooled score
Within prompt and frame
Dynamic balance
50-step
0.784
0.850
0.889
0.863
0.889
+0.000 [ −0.001 , +0.001 ]
+0.000 [ −0.001 , +0.001 ]
Dynamic balance
1000-step
0.655
0.720
0.779
0.765
0.777
−0.002 [ −0.005 , −0.000 ]
−0.001 [ −0.003 , +0.000 ]
Dynamic balance
MotionDiffuse
0.124
0.620
0.823
0.700
0.823
0 (all folds)
0 (all folds)
Static balance
50-step
0.771
0.780
0.830
0.800
0.828
−0.001 [ −0.004 , +0.002 ]
−0.002 [ −0.006 , +0.002 ]
Static balance
1000-step
0.548
0.592
0.671
0.647
0.671
0 (all folds)
0 (all folds)
Table 3: Tuned readouts at 20% of denoising, seeds pooled, every readout tuned on held-out prompts (Section 3.3 ). Pooled R2 , or AUC for flight and floats or falls; the best tuned readout in bold. Both: estimate and layer, tuned with fallback. Gain: Both minus Estimate, with 95% prompt-bootstrap intervals, pooled and within prompt and frame. “Jump prompts”: the jump-heavy runs. –: too few prompts for an interval. 0 (all folds): the fallback kept the estimate in every fold, so the gain is zero by construction (Table 8 ).
Figure 5: The fair comparison (tuned readouts, seeds pooled, 95% prompt-bootstrap intervals). (a) Tuned estimate minus tuned layer: positive favors the estimate. (b) The gain from adding the layer to the estimate, with fallback. Marker shape gives the score; color gives the verdict: teal where the estimate is ahead, deep blue where the layer is ahead or adds, gray where the interval covers zero, and open markers at zero where the fallback kept the estimate in every fold. In (b), clear gains appear only for the fall rate on the jump prompts (“jumps”) and for floating in the small main data sets; balance and flight show none. Not drawn: scores without an interval (too few prompts) and the main data’s fall rate (1,147 and 1,620 falling frames, intervals up to about ±0.25 ; Table 3 ).
Figure 6: The gain from adding one layer, or all eight, to the tuned readout of the estimate (with fallback, 20% of denoising, 95% intervals; colors as in Figure 5 ). Circles: main data; diamonds: jump prompts; open markers and dashed lines: the 1000-step model. Top row, MDM: only the fall rate rises well above zero, from layer 3 on (layer 2 on the 1000-step model); the floating gains stay at about 0.01 or less. Bottom row, MotionDiffuse: the same pattern with a smaller fall-rate gain, from layers 5 to 7, and a floating gain only on the main data’s 50 prompts. The main data’s fall rate is not shown (too few falling frames).
Figure 7: Held-out predictions of the final fall rate on the jump prompts (50-step model, 20% of denoising, tuned readouts; density of falling stretches in cells of 0.25 m/s 2 , darker for more). (a) From the estimate alone and (b) with layer 5 added (with fallback); dashed: perfect prediction; R2 per frame, as in Table 3 . (c) Among jumps that travel at least 0.4 m, the three where adding the layer reduces the error most, selected for that reason: the final motion at take-off, apex and landing, its center-of-mass path (red while airborne), and the estimate’s path at 20% (dashed).
Figure 8: The fall rate of jumps across denoising (jump prompts, layer 5, tuned readouts, seeds pooled). (a) A typical jump, drawn from the estimate at 0% and 20% of denoising and from the final motion (center of mass red while airborne), and its center-of-mass height at every recorded step with that step’s fitted fall rate. At 0% the estimate is a crouched average pose; from 20% the jump is in place, and its height and fall rate change by only a few centimeters and about 1 m/s 2 afterwards. The jump is chosen by a fixed rule: among falling stretches of the 50-step runs that travel at least 0.4 m, the one whose 20% fall-rate error is the median (1.0 m/s 2 ). (b, c) R2 for the final fall rate from the estimate and from estimate and layer together; the shaded gap is the gain. Gray: the estimate’s own fitted fall rate, and the layer alone. (d) The gain, with 95% intervals: largest at 20%, below 0.02 by 60%.
Figure 9: Tuned readouts trained on fewer prompts (seeds pooled, same test folds; two draws per size below the full set; lines: mean, bands: range). (a) Dynamic balance: the explicit label wins at 25 prompts, and from 50 on the estimate leads the layer. (b) The fall rate of jumps, 50-step model: the layer leads at every size. (c) The same target, 1000-step model: the gain from combining estimate and layer grows from 0.02 to 0.08 only with the full training set.
Table 13
Figure 10: Delivered floating against compute, with the final check kept and the retry budget varied from 1 to 20 tries (labels on the finish-then-filter curve). Aborting at 20% of denoising when the decoded estimate shows floating (teal) moves the curve only slightly, and a layer probe (deep blue) moves it about as much. The shaded region between finish-then-filter and a perfect detector bounds what any early signal could save.
Readout of the clean VAE latent
Per-video (%)
Pairwise (%)
Linear in the latent
Pooled over time and space
50.1 ± 0.5
52.1 ± 1.5
Time kept
51.5 ± 0.7
50.4 ± 1.3
Time + 4×4 space
50.9 ± 1.1
54.4 ± 1.2
Time + 4×4 space + signed changes
52.8 ± 1.1
52.2 ± 2.2
Linear after a fixed nonlinearity: absolute frame-to-frame change
Table 6: IntPhys development set, plausibility read from the clean Wan 2.1 VAE latent (360 videos, 90 scenarios; scenario-grouped 5-fold cross-validation, 10 repeats, 3 for the nonlinear temporal probe; mean ± standard deviation). Pairwise: share of possible/impossible pairs within a scenario in which the possible video scores higher. Chance is 50% for both.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Estimate
Layer
Target
Model
raw, ridge
raw, MLP
all it shows, ridge
all it shows, MLP
ridge
MLP
Dynamic balance
50-step
0.767
0.869
0.866
0.889
0.850
0.862
Dynamic balance
1000-step
0.594
0.762
0.746
0.779
0.720
0.765
Dynamic balance
MotionDiffuse
0.583
0.825
0.606
0.823
0.620
0.700
Static balance
50-step
0.697
0.798
0.825
0.830
0.780
0.801
Static balance
1000-step
0.544
0.646
0.641
0.671
0.592
0.647
Appendix
Table 7: Pooled scores at 20% of denoising, seeds pooled: the estimate read from its raw features or from everything it shows, and the layer, each with ridge and with the tuned MLP family. † The MLP family diverged in one fold, in every configuration (MotionDiffuse, fall rate, inputs that include the estimate’s labels); the tuned readouts chose ridge in that fold, so Table 3 is unaffected.
Target
Model
Gain with fallback
Gain without fallback
Estimate kept
Gain, rescaled
Dynamic balance
50-step
+0.000 [ −0.001 , +0.001 ]
−0.005 [ −0.010 , −0.000 ]
4
+0.001
Dynamic balance
1000-step
−0.002 [ −0.005 , −0.000 ]
−0.002 [ −0.006 , +0.002 ]
3
− 0.006
Dynamic balance
MotionDiffuse
0 (all folds)
−0.015 [ −0.023 , −0.007 ]
5
+0.000
Static balance
50-step
−0.001 [ −0.004 , +0.002 ]
−0.004 [ −0.008 , −0.000 ]
2
− 0.004
Static balance
1000-step
0 (all folds)
−0.006 [ −0.018 , +0.002 ]
5
+0.000
Static balance
MotionDiffuse
0 (all folds)
−0.031 [ −0.051 , −0.012 ]
5
+0.000
Appendix
Table 8: The gain from adding the layer in detail (20% of denoising, seeds pooled, pooled score, 95% prompt-bootstrap intervals). With fallback: as in Table 3 . Without fallback: the tuned readout of estimate and layer against the tuned estimate. Estimate kept: outer folds, of five, in which the fallback chose a readout of the estimate alone, so that fold’s gain is zero by construction; when all five do, no interval is defined. Rescaled: for R2 , the conditional usable information in nats by Proposition 1 , 21log((1−Restimate2)/(1−Rboth2)) ; for AUC, the share of misranked pairs that adding the layer removes.
20%
40%
60%
80%
Model
αˉ
SNR
αˉ
SNR
αˉ
SNR
αˉ
SNR
MDM, 50-step
0.094
0.104
0.341
0.517
0.617
1.614
0.879
7.273
MDM, 1000-step
0.094
0.104
0.341
0.517
0.646
1.825
0.898
8.781
MotionDiffuse
0.002
0.002
0.026
0.027
0.194
0.240
0.656
1.910
Appendix
Table 9: Noise level at the recorded steps: αˉt and the signal-to-noise ratio αˉt/(1−αˉt) of the noisy input, from each model’s schedule (MDM cosine, MotionDiffuse linear; scripts/noise_levels.py ).
Added to the estimate
Pooled score
Within prompt and frame
Dynamic balance ( R2 )
the layer (real)
+0.000 [ −0.001 , +0.001 ]
+0.000 [ −0.001 , +0.001 ]
injected, 0.005
+0.013 [ +0.009 , +0.017 ]
+0.018 [ +0.013 , +0.022 ]
injected, 0.01
+0.025 [ +0.020 , +0.030 ]
+0.035 [ +0.031 , +0.040 ]
injected, 0.02
+0.040 [ +0.034 , +0.047 ]
+0.055 [ +0.049 , +0.061 ]
In flight (AUC)
the layer (real)
−0.004 [ −0.008 , +0.000 ]
−0.008 [ −0.014 , −0.002 ]
injected, 0.005
−0.010 [ −0.021 , −0.002 ]
+0.018 [ +0.009 , +0.028 ]
Appendix
Table 10: Positive control (50-step model, 20% of denoising, seeds pooled; gain with fallback over the tuned estimate, 95% prompt-bootstrap intervals). Injected: the tuned estimate readout’s out-of-fold residual plus noise, sized so that its best linear use would add the stated R2 on the squared-error scale, added to the layer along one random direction with the variance of a typical direction of the standardized activations ( fair_readouts.py --layer-spec inject ). Tuned readouts can extract more than this linear amount. The real-layer rows come from the same rerun as the injected rows and differ from Table 3 by refit noise of at most 0.001.
Figure 11: Fair comparison at 20% of denoising. Each row is a readout; filled circles show the 50-step model and open circles the 1000-step model (mean, with the standard deviation over seeds). The dashed line marks the MLP on everything the estimate shows (shaded row) for the 50-step model. For both balance measures it matches or beats every readout that uses the layer on both MDM checkpoints, and adding the layer to it (black) does not help; for flight this holds on the 50-step model, while on the 1000-step model the layer’s linear probe is slightly ahead (0.926 against 0.918). Flight gravity is the exception in these data, on both checkpoints; Section 4.3 tests it with tuned readouts and about ten times as many falling frames.
Model
Target
Explicit
Estimate, linear
Estimate, MLP
Layer, linear
Layer, MLP
Estimate + layer, MLP
50-step
Dynamic balance ( R2 )
0.783 ± 0.009
0.723 ± 0.007
0.878 ± 0.005
0.838 ± 0.001
0.829 ± 0.018
0.867 ± 0.007
Static balance ( R2 )
0.769 ± 0.019
0.673 ± 0.014
0.816 ± 0.028
0.759 ± 0.014
0.712 ± 0.059
0.791 ± 0.028
In flight (AUC)
0.876 ± 0.005
0.935 ± 0.008
0.972 ± 0.001
0.972 ± 0.003
0.964 ± 0.004
0.967 ± 0.008
Flight gravity ( R2 )
0.707 ± 0.079
0.547 ± 0.053
0.703 ± 0.068
0.814 ± 0.040
0.846 ± 0.010
0.828 ± 0.038
1000-step
Dynamic balance ( R2 )
0.655 ± 0.008
0.550 ± 0.031
0.747 ± 0.014
0.697 ± 0.029
0.725 ± 0.026
0.746 ± 0.022
Static balance ( R2 )
0.546 ± 0.019
0.508 ± 0.066
0.620 ± 0.043
0.557 ± 0.063
0.570 ± 0.058
0.601 ± 0.057
Appendix
Table 11: Pooled scores at 20% of denoising (mean ± standard deviation over seeds; best per row in bold). Estimate, MLP reads everything the estimate shows; Estimate + layer adds the layer’s activations.
Model
Target
Explicit
Estimate, linear
Estimate, MLP
Layer, linear
Layer, MLP
Estimate + layer, MLP
50-step
Dynamic balance ( r )
0.797 ± 0.011
0.756 ± 0.007
0.892 ± 0.003
0.851 ± 0.000
0.867 ± 0.002
0.889 ± 0.002
Static balance ( r )
0.746 ± 0.020
0.644 ± 0.025
0.801 ± 0.023
0.729 ± 0.015
0.749 ± 0.020
0.795 ± 0.019
In flight ( r )
0.710 ± 0.007
0.562 ± 0.014
0.755 ± 0.010
0.662 ± 0.010
0.734 ± 0.007
0.760 ± 0.008
Flight gravity ( r )
0.439 ± 0.053
0.219 ± 0.047
0.340 ± 0.039
0.445 ± 0.071
0.536 ± 0.109
0.508 ± 0.081
1000-step
Dynamic balance ( r )
0.736 ± 0.004
0.664 ± 0.008
0.816 ± 0.001
0.772 ± 0.011
0.800 ± 0.006
0.817 ± 0.003
Static balance ( r )
0.607 ± 0.010
0.572 ± 0.010
0.679 ± 0.003
0.610 ± 0.010
0.642 ± 0.010
0.673 ± 0.005
Appendix
Table 12: Within-prompt scores at 20% of denoising: Pearson r after subtracting each prompt’s mean from target and prediction, so that only sample-specific information counts (mean ± standard deviation over seeds; best per row in bold).
Figure 12: Dynamic balance at 20% of denoising (50-step model) with the probes trained on fewer prompts; same test folds; each point averages three draws of the training prompts in each of the three seeds (lines: mean; bands: standard deviation). The 512-feature layer probe overtakes the one-feature explicit label between 52 and 100 training prompts; the pilot (gray band) trained on 52.
Floating, test captions (% of frames)
Jump gravity, curated jump prompts (m/s 2 )
Early rule at 20%
Setting
50-step
1000-step
50-step
1000-step
Precision
Recall
Default
3.8–4.7
8.6–9.7
9.54–9.75
9.13–9.22
0.73–0.89
0.44–0.68
Contact heights ×0.75
6.4–7.6
13.1–13.8
9.68–9.92
9.03–9.40
0.66–0.78
0.46–0.58
Contact heights ×1.25
2.8–3.6
6.1–6.6
9.39–9.72
9.18–9.38
0.77–0.94
0.49–0.69
Support rule 10 cm
3.8–4.7
8.7–9.7
9.54–9.75
9.13–9.22
0.72–0.91
0.45–0.65
Support rule 20 cm
3.8–4.7
8.6–9.6
9.54–9.75
9.13–9.22
0.74–0.90
0.44–0.67
Appendix
Table 13: Label thresholds changed one at a time (range over the three seeds of each model; the early rule “abort if the decoded estimate shows floating” is scored over the seeds of both MDM checkpoints).
Video generation models produce strikingly realistic sequences and are increasingly proposed as world models, yet recent benchmarks reveal pronounced deficits in their physical reasoning. This raises the question of whether these models internalize physical principles or merely reproduce familiar motion patterns. We address this by probing internal representations of video Diffusion Transformers (DiTs) for simulator-derived ground-truth physical quantities spanning kinematic motion and rigid-body dynamics under gravity and contact. We find that these quantities are linearly decodable with high accuracy early in the denoising process, substantially outperforming a baseline decoded directly from the model's own noised latents, indicating that the relevant physical information is actively constructed during denoising rather than already present in the input. Additionally, we show that activations at on-object tokens carry the relevant physical information and that quantities defined over multiple frames are readable from single latent frames. Hence, information is sharply localized within the token sequence and is computed globally but stored locally. The probes further show partial extrapolation, transferring to scene variations and object configurations outside their training regime, so what they read is not simply a correlate of the scenes they were fit on. When fitted directly in the full-resolution activation space, the probing directions can serve as steering vectors to change the model's output.
Jonas Kneifl, Jakub Skalski, Bartłomiej Twardowski +1
IDEAS Research Institute, Warsaw, Poland · Computer Vision Center, Universitat Autònoma de Barcelona, Spain · Warsaw University of Technology, Poland
Modern video diffusion models generate increasingly realistic and temporally coherent videos, motivating their use as candidate world simulators. Yet it remains unclear whether these models internally encode physical structure, or merely reproduce motion patterns seen during training. We study this question by probing video diffusion models along latent trajectories corresponding to real videos with known physical plausibility. To obtain such trajectories, we approximately invert the deterministic sampling process by integrating the learned velocity field backward from a clean video latent to noise, giving access to the model's intermediate states and attention maps. Using these recovered trajectories, we show that physical plausibility is linearly decodable from diffusion transformer states across IntPhys and InfLevel, reaching around 81.27% average accuracy and outperforming dedicated representation-learning baselines such as V-JEPA and VideoMAE. Surprisingly, this signal is absent from the VAE latent input and emerges inside the denoising transformer itself, despite the model not being trained with a self-supervised predictive objective. These findings suggest that physically meaningful representations can arise as a byproduct of generative denoising.
Parsa Esmati, Somjit Nath, Katja Hofmann +3
University of Bristol · McGill University · Mila–Quebec AI Institute +2
Do video diffusion models encode signals predictive of physical plausibility? We probe intermediate denoising representations of pretrained Diffusion Transformers (DiTs) and find that physically plausible and implausible videos are partially separable in mid-layer feature space, even at high noise levels. Within-source and perceptual-quality controls suggest that this signal is not fully explained by generator identity or generic visual quality. We distill the signal into a lightweight, backbone-specific physics verifier trained on frozen features and use it in two complementary inference-time mechanisms under a fixed multi-trajectory budget: progressive trajectory selection, which scores trajectories at intermediate checkpoints and prunes weak candidates early, and reward-gradient guidance, which steers surviving trajectories by backpropagating through only the first few DiT blocks. Experiments on PhyGenBench and Physics-IQ across CogVideoX-2B/5B and Wan 2.1-14B show that progressive selection matches verifier-based Best-of-4 on CogVideoX-2B while reducing wall-clock inference time by 37%, whereas reward-gradient guidance substantially improves physical consistency on CogVideoX-5B, all without fine-tuning the video generator.
Chujun Tang, Lei Zhong, Fangqiang Ding
The Hong Kong University of Science and Technology (Guangzhou) · Brown University · University of Edinburgh