Diffusion models for temporal prediction typically corrupt each future frame with independent noise, leaving temporal dependence entirely to the denoiser. We propose Dynamical State-Space Diffusion (DSSD), which learns the temporal structure of the corruption process itself. DSSD builds a temporal kernel from the damped oscillatory modes of a stable diagonal state-space model, whose learned decay rates and frequencies let the corruption capture multiple timescales and oscillatory correlations, while the denoiser architecture remains unchanged. Running this model on a finite clip, however, requires the state before the first frame, which is unobserved; starting from a zero state leaves the early frames with less noise than the diffusion schedule prescribes. We avoid this by using the Cholesky factor of the model's stationary covariance as the temporal operator. We prove that this operator is causal, exactly invertible, keeps each frame's noise variance equal to that of standard diffusion, and reduces to the identity at the clean end of the process, and that it is exactly the Kalman innovations form of the state-space model. The denoiser is trained to predict the temporally correlated noise, and sampling removes and reapplies the temporal mixing at each step. On scientific-field and robot-video datasets, DSSD outperforms diffusion baselines that share its denoiser, and the evolution of its learned modes during training shows that the temporal operator adapts to each dataset.
Figures & tables
Figure 1: DSSD forward process. Learned state-space dynamics causally mix observed and future latents and separately correlate noise within the future window. The mixed future latents and correlated noise are combined at diffusion step t to form zt .
Figure 2: Noise correlations and state-space modes in DSSD. (a) Illustrative noise correlation matrices for DPM, DyDiff, and DSSD. (b) Corresponding noise correlations by lag: DyDiff’s EMA decays exponentially, whereas DSSD can express oscillatory correlations. (c) An illustrative six-mode DSSD configuration shows discrete poles pn=eΔγλn and their conjugates inside the unit circle. Filled and hollow markers denote conjugate partners.
Dataset
Method
CRPS ↓
CSI w5 ↑
Temporal variogram ↓
Energy score ↓
RMSE ↓
Raw
w8avg
w8max
SEVIR
DPM
4.174
2.737
6.599
0.5518
2.339
10.862
15.420
DyDiff
4.642
3.169
7.955
0.3911
4.492
13.024
25.540
DSSD (Ours)
3.975
2.421
5.995
0.5707
2.129
10.263
14.585
Turbulence
DPM
0.01148
0.00751
0.01065
0.9641
0.00262
0.01920
0.02600
DyDiff
0.01100
0.00752
0.01123
0.9674
0.00260
0.01791
0.02427
Table 1: Scientific-field prediction results.
Dataset
Method
Forecast metrics
Video metrics
CRPS ↓
RMSE ↓
Energy ↓
Vario. ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
BAIR
DPM
17.788
42.907
30.176
11.078
15.036
0.5938
0.2246
1329.2
DyDiff
17.691
42.815
30.092
11.412
15.149
0.5999
0.2220
1417.4
DSSD (Ours)
16.825
42.367
29.661
8.548
15.717
0.6205
0.2007
766.0
RoboNet
DPM
8.782
23.523
15.130
5.939
22.680
0.7889
0.1370
659.2
DyDiff
8.383
23.735
14.705
6.504
22.630
0.7865
0.1377
770.6
Table 2: Robot-video prediction results.
Figure 3: BAIR forecast quality across future frames. (a) Example predictions and absolute-error maps. (b) Test-set frame-wise pooled CRPS and temporal variogram scores; lower is better.
Figure 4: DSSD mode-parameter diagnostics. (a) SEVIR centered residue logits and their norm. (b) Normalized residue entropy on four datasets.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Defined in
Data and diffusion
N , M
Numbers of observed and future frames
§ 3.1
L
Length of the window on which a temporal operator acts; L∈{N,M,N+M}
§ 4.1
d
Dimension of a flattened latent frame
§ 3.1
o∈RN×d
Observed latents (clean conditioning history)
§ 3.1
x∈RM×d
Future latents to be predicted
§ 3.1
Appendix
Table 3: Notation.
Dataset
Input
Resolution
N→M
Input transform
SEVIR
VIL, 1 channel
1282
7→6
x↦x/127.5−1
Turbulence
Velocity, 2 channels
642
4→11
x↦x/4
BAIR
RGB video
642
1→15
x↦x/127.5−1
RoboNet
RGB video
642
2→10
x↦x/127.5−1
Appendix
Table 4: Forecasting tasks. N and M are the numbers of observed and future frames.
Component
Setting
Video U-Net
64 base channels; channel multipliers (1,2,4,4) ; two residual blocks per level.
Attention and time mixing
Attention at downsampling factors (4,2,1) ; four heads; transformer depth 1; dropout 0; temporal convolution kernel (3,1,1) and learned spatial/temporal mixing.
Diffusion
1,000 timesteps; cosine schedule ( Nichol and Dhariwal, 2021 ) with s=0.008 ; ϵ prediction.
Optimization
Adam ( Kingma and Ba, 2015 ) , (β1,β2)=(0.9,0.999) , ϵAdam=10−8 , weight decay 0; effective batch size 16; FP32. No denoiser-weight EMA.
Sampling
50 deterministic DDIM steps on the same released_uniform timestep grid.
Appendix
Table 5: Shared model, optimization, and sampling settings.
Table 6: Method-specific temporal operator and learning-rate schedule.
Method
SEVIR
Turbulence
BAIR
RoboNet
s/upd.
Total
Mem.
s/upd.
Total
Mem.
s/upd.
Total
Mem.
s/upd.
Total
Mem.
DPM
2.22
31.8
48.8
1.06
14.8
18.4
0.98
21.5
11.5
0.82
17.4
9.2
DyDiff
2.19
31.3
48.8
1.07
15.0
18.4
0.98
21.9
11.5
0.83
17.8
9.2
DSSD (Ours)
2.22
31.9
48.8
1.02
14.8
18.4
1.00
22.7
11.5
0.86
17.7
9.2
Appendix
Table 7: Training cost over the first 50,000 updates on one A100 80GB GPU. s/upd.: median time per update (s); Total: wall-clock time (h); Mem.: reserved GPU memory (GiB).
Figure 5: Example scientific-field forecasts and absolute errors on (a) SEVIR and (b) Turbulence. Turbulence shows the v velocity component. Errors above the color-bar maximum saturate.
Figure 6: Example robot-video forecasts and RGB-mean absolute errors on (a) BAIR and (b) RoboNet. Errors above the color-bar maximum saturate.
Backbone
Method
CRPS ↓
CSI w5 ↑
Temporal variogram ↓
Energy score ↓
RMSE ↓
Raw
w8 avg
w8 max
U-Net
DPM
4.174
2.737
6.599
0.5518
2.339
10.862
15.420
DyDiff
4.642
3.169
7.955
0.3911
4.492
13.024
25.540
DSSD (Ours)
3.975
2.421
5.995
0.5707
2.129
10.263
14.585
DiT
DPM
5.425
3.862
7.453
0.4314
3.559
10.437
23.689
DyDiff
5.260
3.789
7.327
0.4383
3.326
10.777
24.905
Appendix
Table 8: Backbone ablation on SEVIR. Results with a U-Net and a DiT denoiser. Bold : best; underline : second best within each backbone.
Figure 7: Mode-resolved residue diagnostics on the four datasets, measured every 10k training steps (10k–50k) from saved DSSD checkpoints. (a) Softmax residue shares of the six temporal modes, each corresponding to a conjugate pair. (b) Absolute centered residue logits of each mode (bars) and their total L2 norm (diamonds).
Dataset
D
N
M
Method
CRPS ↓
RMSE ↓
MAE ↓
Energy ↓
Variogram ↓
Exchange
8
30
30
DPM
0.37573
0.68591
0.56123
0.59404
0.78559
DyDiff
0.37640
0.76033
0.64038
0.54593
0.60261
DSSD (Ours)
0.29896
0.61315
0.51080
0.46610
0.44125
Solar
137
24
24
DPM
1.43163
2.31897
1.95040
1.89547
2.25310
DyDiff
1.48259
2.34964
1.97462
1.92896
2.37242
DSSD (Ours)
1.41653
2.31678
1.95240
1.88563
2.26805
Appendix
Table 9: Multivariate time-series test results at 50,000 updates. Bold and underlined values are best and second best within each dataset. All scores use training-set normalization; lower is better. D is the number of jointly modeled series; N and M are observed and predicted time steps.
In many domains, practitioners seek models that produce accurate forecasts while faithfully capturing latent system dynamics. Existing approaches typically sacrifice one of these goals: deep state space models often assume Gaussian latent transitions, limiting fit and forecasting, while diffusion models are highly expressive but lack principled inference for the underlying dynamics. To combine the strengths of both, we introduce the Diffusion-Driven State Space Model (DDSSM), which replaces the conventional Gaussian transition distribution with a diffusion model. Our DDSSM resolves the open problem of how to jointly train an autoencoder and a diffusion model on sequential data, thereby extending the literature on latent diffusion models for time series. Moreover, we find that the DDSSM empirically outperforms a state-of-the-art deep SSM at fitting and forecasting a simulated time series with multimodal transitions.
Diffusion models have become a widely used framework for probabilistic time series forecasting, modeling the distribution of future values given an observed history. In time series forecasting, however, the future continues the observed history, creating an asymmetry the standard diffusion process leaves unaddressed, with slowly-varying content largely determined by the observed continuity while higher-frequency dynamics carry most of the residual uncertainty. Existing diffusion-based forecasters decouple this asymmetry through an external rule before generation, leaving the corruption trajectory blind to which parts of the target the history can already anchor. We propose DiffDiff, a diffusion framework that embeds this predictability asymmetry into the diffusion trajectory itself, so that a single end-to-end diffusion process becomes aware of which parts of the target the history can already anchor. DiffDiff makes the forward operator step-dependent so that the noisy intermediate state progressively shifts from the target itself toward its second-order differenced structure, while a conditioning pathway supplies the denoiser with both value-domain and differential history information balanced by a stage-adaptive gate at each diffusion step. The terminal distribution approaches a standard Gaussian, preserving compatibility with existing samplers. On seven benchmarks across four prediction horizons, DiffDiff outperforms six diffusion baselines, while our analyses show that its trajectory reallocates modeling effort according to history predictability and improves distributional fidelity.
Chen Su, Yuanhe Tian, Yan Song
University of Science and Technology of China · Zhongguancun Academy
Deep state-space models (DSSMs) enable temporal predictions by learning the underlying dynamics of observed sequence data. They are often trained by maximising the evidence lower bound. However, as we show, this does not ensure the model actually learns the underlying dynamics. We therefore propose a constrained optimisation framework as a general approach for training DSSMs. Building upon this, we introduce the extended Kalman VAE (EKVAE), which combines amortised variational inference with classic Bayesian filtering/smoothing to model dynamics more accurately than RNN-based DSSMs. Our results show that the constrained optimisation framework significantly improves system identification and prediction accuracy on the example of established state-of-the-art DSSMs. The EKVAE outperforms previous models w.r.t. prediction accuracy, achieves remarkable results in identifying dynamical systems, and can furthermore successfully learn state-space representations where static and dynamic features are disentangled.
Alexej Klushyn, Richard Kurle, Maximilian Soelch +2
Technical University of Munich · AWS AI Labs · Machine Learning Research Lab, Volkswagen Group, Munich +1