Forecasting the behavior of spatio-temporal systems often requires more than accurate predictions: models should uncover the dynamical modes governing future evolution. Existing temporal graph and latent dynamical models achieve strong forecasting performance, but their latent representations often entangle multiple dynamical patterns, limiting interpretability. Conversely, regime-inference methods explicitly model such modes but typically do not support generative forecasting over graph-structured systems. We introduce HERON, a self-supervised regime-aware generative framework that unifies graph-structured trajectory generation with regime identification. HERON factorizes the latent state into discrete system-level regimes, capturing global dynamical modes, and continuous entity-level dynamics, yielding a modular regime interface compatible with existing spatio-temporal backbones. Crucially, the inferred regime actively conditions system dynamics, enabling regime tracking and controlled simulations under alternative dynamical modes. Without regime annotations, HERON jointly learns forecasts and regime dynamics in a self-supervised manner, recovering meaningful modes across controlled synthetic and real-world datasets while maintaining competitive forecasting performance. These results open a path toward interpretable, controllable generative modeling of spatio-temporal dynamics.
Figures & tables
Figure 1: Example of regime-conditioned graph rollouts. A) The system can evolve according to distinct dynamics referred to as regimes. B) Given an observed graph sequence acting as context , we forecast future node features while inferring the latent regime trajectory that conditions the rollout . The regime exposes which dynamical mode drives the predictions and allows alternative regime-aware futures beyond forecasting alone.
Figure 2: Model architecture. A) Framework: At each time step, the model first updates the deterministic recurrent state ht . Conditioned on this state, the encoder infers the regime rt and the entity latents zti . Together with ht , they are then used by the decoder to predict the next-step features Xt+1 . B) Encoder: The encoder proceeds hierarchically. A GNN first extracts node- and graph-level embeddings ( embti , embtG ), which are used to infer the regime rt via a transition module. This regime then conditions a mixture-of-experts (MoE) to produce the entity-level latent states zti .
Figure 3: Hierarchical state representation. Entity-level embeddings are aggregated and combined with the global state htG to infer a latent regime rt that governs system-wide dynamics. This regime then conditions a MoE module to produce entity-level latent states zti , linking global behavior and individual evolution.
Synthetic
Real-world
Model
Linear
Gompertz
Network
NBA
AMASS
Persistence (Static)
5.595
2.521
0.105
4.48/7.86
0.093/0.135
Persistence (Velocity)
2.896
2.906
0.058
2.55/5.75
0.129/0.271
S4 Gu et al. [2022]
4.072
4.307
0.162
2.09/5.11
0.422/0.616
rSLDS Glaser et al. [2020]
2.744
3.171
0.052
2.93/5.40
0.074/0.144
RED-SDS Ansari et al. [2021]
3.287
3.158
0.051
4.84/8.57
0.077/0.120
Table 1: Regime-conditioned forecasting performance. Test ADE is reported for generated datasets, and ADE/FDE for motion datasets. Best results are in bold and second-best underlined. N/A means that the method was not applicable.
Synthetic
Real-world
Method
Linear
Gompertz
Network
NBA
AMASS
Latent-HMM
0.945
0.897
0.657
0.593
0.318
rSLDS Glaser et al. [2020]
0.953
0.860
0.357
0.358
0.294
RED-SDS Ansari et al. [2021]
0.947
0.926
0.557
0.451
0.299
HERON
0.961
0.927
0.913
0.891
0.412
Table 2: Self-supervised regime recovery. Test macro-F1 quantifies how well the learned regimes align with ground-truth dynamical labels, which are used only for evaluation. Best results are in bold and second-best results are underlined.
Figure 4: Regime-conditioned rollout probes. From a shared context, we clamp regimes during rollout. Dashed lines show regime-specific trajectories; the solid line is the observed future. Regime names are assigned post hoc from dominant motion patterns.
Forecasting
Regime recovery (Macro-F1)
Model
Network ADE
NBA ADE/FDE
AMASS ADE/FDE
Network
NBA
AMASS
HERON
0.038
0.81 / 1.49
0.047 / 0.066
0.913
0.891
0.412
A1: no-regime, capacity-matched
0.048
1.52/2.97
0.055/0.078
0.248
0.546
0.290
A2: passive regime
0.049
1.52/2.99
0.056/0.084
0.786
0.782
0.369
A3a: fixed regime
0.052
1.35/2.09
0.053 /0.075
0.163
0.357
0.217
A3b: random regime
0.049
1.38/2.09
0.055/0.075
0.371
0.501
0.307
Table 3: Component ablations. Test forecasting performance (ADE on Network and ADE/FDE on NBA and AMASS) and test regime-recovery performance (Macro-F1). Lower forecasting errors and higher Macro-F1 are better. Best results are in bold and second-best results are underlined.
Synthetic
Real-world
Linear
Gompertz
Network
NBA
AMASS
Number of entities per graph
50-100
50-100
50-100
11
22
Number of sequences
∼103
∼103
∼103
∼105
∼105
Number of regimes
3
3
3
2
4
Metrics
ADE
ADE
ADE
ADE/FDE
ADE/FDE
Dynamic topology
–
✓
–
–
–
Table 4: Summary of dataset properties. The table highlights complementary dynamics and scales. ✓ indicates presence; – indicates absence.
Regime type
ak
bk
Behavior
Decay
0.8
0
Exponential decay
Stable
0.0
1
Constant drift independent of features value
Increase
1.2
0
Exponential increase
Table 5: Linear dataset regimes. The three dynamics corresponds to different linear evolution patterns.
Regime type
ak
ck
Behavior
Tumor regression
0.05
0.08
Shrinks toward small volume
Stable disease
0.5
0.00
Approximately constant
Progressive disease
1
0.08
Grows beyond initial features range
Table 6: Gompertz regimes. The three dynamics corresponds to biological relevant tumor growth patterns, namely regression, stable and progression.
Family
Flow class
Regime signal
Propagation pattern
R2
Saturating regulatory influence
Degree-averting
Peripheral pathways dominate
P
Self-limited growth with neighbor reinforcement
Degree-driven
Hubs dominate propagation
E
Recovery and susceptibility-limited contagion
Homogeneous
Propagation is broadly distributed
Table 7: Network interaction regimes. The three network dynamics differ in the mechanism by which information propagates over the same graph topology.
Regime type
Example of Action Names
Global Motion Pattern
Step/Walk
‘walk away‘, ‘slow walk‘, ‘step forward‘, …
Directional feet movement
Stand
‘look‘, ‘poses‘, ‘wait‘, …
Most body static
Turn
‘twist body to the left‘, ‘rotate‘, ‘twirl‘, …
Body rotating
Sit/Lower
‘sit‘, ‘lean‘, ‘crouch‘, …
Body joints lowering to the ground
Table 8: AMASS/BABEL regimes. The four dynamics corresponds to dominant motion patterns in the AMASS [ Mahmood et al., 2019a ] dataset aligned with BABEL labels [ Punnakkal et al., 2021a ] .
Synthetic
Real-world
Model
Linear
Gompertz [ Gompertz, 1825 ]
Network [ Harush and Barzel, 2017 ]
NBA [ Linou et al., 2016 ]
AMASS [ Mahmood et al., 2019a ]
Persistence (Static)
5.518 ( ± 0.055)
2.498 ( ± 0.025)
0.103 ( ± 0.001)
4.51 ( ± 0.07) / 7.78 ( ± 0.13)
0.092 ( ± 0.002) / 0.133 ( ± 0.003)
Persistence (Velocity)
2.911 ( ± 0.029)
2.881 ( ± 0.029)
0.057 ( ± 0.001)
2.51 ( ± 0.05) / 5.81 ( ± 0.10)
0.131 ( ± 0.003) / 0.276 ( ± 0.006)
S4 Gu et al. [2022]
4.156 ( ± 0.125)
4.378 ( ± 0.131)
0.165 ( ± 0.006)
2.13 ( ± 0.08) / 5.04 ( ± 0.18)
0.428 ( ± 0.018) / 0.624 ( ± 0.028)
rSLDS Glaser et al. [2020]
2.720 ( ± 0.058)
3.133 ( ± 0.047)
0.054 ( ± 0.001)
2.93 ( ± 0.02) / 5.40 ( ± 0.05)
0.073 ( ± 0.004) / 0.133 ( ± 0.006)
RED-SDS Ansari et al. [2021]
3.388 ( ± 0.031)
3.152 ( ± 0.128)
0.046 ( ± 0.002)
4.72 ( ± 0.07) / 8.36 ( ± 0.13)
0.071 ( ± 0.002) / 0.119 ( ± 0.002)
Table 9: Regime-conditioned forecasting performance. Train ADE is reported for generated datasets, and ADE/FDE for motion datasets. Values are mean ( ± standard deviation) over 4 folds. Best results are in bold and second-best underlined. N/A means that the method was not applicable.
Synthetic
Real-world
Method
Linear
Gompertz
Network
NBA
AMASS
Latent-HMM
0.979 ( ± 0.051)
0.965 ( ± 0.044)
0.716 ( ± 0.023)
0.604 ( ± 0.016)
0.329 ( ± 0.017)
rSLDS Glaser et al. [2020]
0.958 ( ± 0.059)
0.946 ( ± 0.016)
0.331 ( ± 0.024)
0.362 ( ± 0.019)
0.330 ( ± 0.025)
RED-SDS Ansari et al. [2021]
0.946 ( ± 0.026)
0.939 ( ± 0.019)
0.570 ( ± 0.007)
0.493 ( ± 0.023)
0.318 ( ± 0.017)
HERON
0.948 ( ± 0.031)
0.931 ( ± 0.012)
0.920 ( ± 0.009)
0.878 ( ± 0.011)
0.419 ( ± 0.018)
Table 10: Self-supervised regime recovery. Train macro-F1 quantifies how well the learned regimes align with ground-truth dynamical labels, which are used only for evaluation. Best results are in bold and second-best results are underlined. Values are mean ( ± standard deviation) over 4 folds.
Network
True regime
Clamped regime
Regulatory
Epidemic
Mutualistic
Regulatory
1.00
1.42
1.19
Epidemic
1.83
1.00
1.29
Mutualistic
1.21
1.15
1.00
Table 11: Forecasting results with regime clamping. Rows indicate the regime imposed during forecasting, while columns indicate the true regime. ADE is normalized column-wise by the matched intervention, such that diagonal entries equal 1.
AMASS condition
Bone-length violation (%)
Motion jerk ( × real)
Ground-truth future
0.00
1.00
Clamp = correct regime
2.51
2.89
Clamp = wrong regime
2.66
2.87
Table 12: Constraint violation with regime clamping on AMASS.
NBA condition
Out-of-court (%)
Player acceleration ( × real)
Ground-truth future
2.21
1.00
Clamp = correct regime
3.37
1.19
Clamp = wrong regime
3.65
1.20
Table 13: Constraint violation with regime clamping on NBA.
Table 15: Overlapping regimes. Hard versus posterior-weighted soft routing.
Dataset
K
Inter-run ARI
Matched accuracy
Worst-mode Jaccard
Null ARI (95th %ile)
Network
3
0.92±0.02
0.98±0.02
0.96±0.03
0.005
NBA
2
0.83±0.04
0.96±0.01
0.90±0.02
0.004
AMASS
4
0.48±0.07
0.61±0.08
0.57±0.05
0.005
Table 16: Cross-seed regime reproducibility. Statistics over all ten pairs from five independent seeds.
Synthetic
Real-world
Metric
Linear
Gompertz
Network
NBA
AMASS
Macro-F1
0.766
0.703
0.790
0.872
0.32
AUC
0.703
0.944
0.868
0.899
0.408
Table 17: Self-supervised regime recovery. Test macro-F1 and AUC quantify how well learned regimes align with ground-truth dynamical labels, which are used only for evaluation.
Network
NBA
K
ADE
macro-F1
AUC
Experts used
K
ADE
FDE
macro-F1
AUC
Experts used
2
0.043
0.55
0.75
2
2
0.81
1.49
0.89
0.91
2
3
0.038
0.91
0.97
3
3
0.89
1.73
0.83
0.86
2
4
0.038
0.81
0.88
3
4
0.84
1.67
0.87
0.87
2
5
0.041
0.73
0.76
3
Table 18: Sensitivity to misspecification of the number of regimes. Bold K marks the reference regime count; metric bolding follows the best value in each dataset.
Selection rule
Network ( K=3 )
NBA ( K=2 )
AMASS ( K=4 )
Expert usage
3
2
6
Effective- K
3
2
5
Held-out likelihood
3
2
4
Velocity baseline
2
4
5
Table 19: Label-free K selection and diagnostics. Reference regime counts are shown in parentheses.
Hyperparameter
Synthetic Datasets
NBA Dataset
AMASS Dataset
batch size
256
64
128
entity embedding dim
32
64
64
encoder hidden dim
48
32
32
encoder num layers
2
2
4
entity hidden dim
32
48
16
system hidden dim
32
32
32
Table 20: Hyperparameters for all datasets. List of the relevant hyperparameters and their values per dataset.
Model
Dataset/configuration
Trainable parameters
GraphSSM
Synthetic
300K
GraphSSM
NBA
450K
GraphSSM
AMASS
450K
HERON
Synthetic
400K
HERON
NBA
600K
HERON
AMASS
600K
Table 21: Trainable parameter counts. Counts correspond to the instantiated configurations used in the experiments.
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Hannover, Germany · Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed), Hannover, Germany