Forecasting the behavior of spatio-temporal systems often requires more than accurate predictions: models should uncover the dynamical modes governing future evolution. Existing temporal graph and latent dynamical models achieve strong forecasting performance, but their latent representations often entangle multiple dynamical patterns, limiting interpretability. Conversely, regime-inference methods explicitly model such modes but typically do not support generative forecasting over graph-structured systems. We introduce HERON, a self-supervised regime-aware generative framework that unifies graph-structured trajectory generation with regime identification. HERON factorizes the latent state into discrete system-level regimes, capturing global dynamical modes, and continuous entity-level dynamics, yielding a modular regime interface compatible with existing spatio-temporal backbones. Crucially, the inferred regime actively conditions system dynamics, enabling regime tracking and controlled simulations under alternative dynamical modes. Without regime annotations, HERON jointly learns forecasts and regime dynamics in a self-supervised manner, recovering meaningful modes across controlled synthetic and real-world datasets while maintaining competitive forecasting performance. These results open a path toward interpretable, controllable generative modeling of spatio-temporal dynamics.
Figures & tables
Figure 1: Example of regime-conditioned graph rollouts. A) The system can evolve according to distinct dynamics referred to as regimes. B) Given an observed graph sequence acting as context , we forecast future node features while inferring the latent regime trajectory that conditions the rollout . The regime exposes which dynamical mode drives the predictions and allows alternative regime-aware futures beyond forecasting alone.
Figure 2: Model architecture. A) Framework: At each time step, the model first updates the deterministic recurrent state ht . Conditioned on this state, the encoder infers the regime rt and the entity latents zti . Together with ht , they are then used by the decoder to predict the next-step features Xt+1 . B) Encoder: The encoder proceeds hierarchically. A GNN first extracts node- and graph-level embeddings ( embti , embtG ), which are used to infer the regime rt via a transition module. This regime then conditions a mixture-of-experts (MoE) to produce the entity-level latent states zti .
Figure 3: Hierarchical state representation. Entity-level embeddings are aggregated and combined with the global state htG to infer a latent regime rt that governs system-wide dynamics. This regime then conditions a MoE module to produce entity-level latent states zti , linking global behavior and individual evolution.
Synthetic
Real-world
Model
Linear
Gompertz
Network
NBA
AMASS
Persistence (Static)
5.595
2.521
0.105
4.48/7.86
0.093/0.135
Persistence (Velocity)
2.896
2.906
0.058
2.55/5.75
0.129/0.271
S4 Gu et al. [2022]
4.072
4.307
0.162
2.09/5.11
0.422/0.616
rSLDS Glaser et al. [2020]
2.744
3.171
0.052
2.93/5.40
0.074/0.144
RED-SDS Ansari et al. [2021]
3.287
3.158
0.051
4.84/8.57
0.077/0.120
Table 1: Regime-conditioned forecasting performance. Test ADE is reported for generated datasets, and ADE/FDE for motion datasets. Best results are in bold and second-best underlined. N/A means that the method was not applicable.
Synthetic
Real-world
Method
Linear
Gompertz
Network
NBA
AMASS
Latent-HMM
0.945
0.897
0.657
0.593
0.318
rSLDS Glaser et al. [2020]
0.953
0.860
0.357
0.358
0.294
RED-SDS Ansari et al. [2021]
0.947
0.926
0.557
0.451
0.299
HERON
0.961
0.927
0.913
0.891
0.412
Table 2: Self-supervised regime recovery. Test macro-F1 quantifies how well the learned regimes align with ground-truth dynamical labels, which are used only for evaluation. Best results are in bold and second-best results are underlined.
Figure 4: Regime-conditioned rollout probes. From a shared context, we clamp regimes during rollout. Dashed lines show regime-specific trajectories; the solid line is the observed future. Regime names are assigned post hoc from dominant motion patterns.
Forecasting
Regime recovery (Macro-F1)
Model
Network ADE
NBA ADE/FDE
AMASS ADE/FDE
Network
NBA
AMASS
HERON
0.038
0.81 / 1.49
0.047 / 0.066
0.913
0.891
0.412
A1: no-regime, capacity-matched
0.048
1.52/2.97
0.055/0.078
0.248
0.546
0.290
A2: passive regime
0.049
1.52/2.99
0.056/0.084
0.786
0.782
0.369
A3a: fixed regime
0.052
1.35/2.09
0.053 /0.075
0.163
0.357
0.217
A3b: random regime
0.049
1.38/2.09
0.055/0.075
0.371
0.501
0.307
Table 3: Component ablations. Test forecasting performance (ADE on Network and ADE/FDE on NBA and AMASS) and test regime-recovery performance (Macro-F1). Lower forecasting errors and higher Macro-F1 are better. Best results are in bold and second-best results are underlined.
Synthetic
Real-world
Linear
Gompertz
Network
NBA
AMASS
Number of entities per graph
50-100
50-100
50-100
11
22
Number of sequences
∼103
∼103
∼103
∼105
∼105
Number of regimes
3
3
3
2
4
Metrics
ADE
ADE
ADE
ADE/FDE
ADE/FDE
Dynamic topology
–
✓
–
–
–
Table 4: Summary of dataset properties. The table highlights complementary dynamics and scales. ✓ indicates presence; – indicates absence.
Regime type
ak
bk
Behavior
Decay
0.8
0
Exponential decay
Stable
0.0
1
Constant drift independent of features value
Increase
1.2
0
Exponential increase
Table 5: Linear dataset regimes. The three dynamics corresponds to different linear evolution patterns.
Regime type
ak
ck
Behavior
Tumor regression
0.05
0.08
Shrinks toward small volume
Stable disease
0.5
0.00
Approximately constant
Progressive disease
1
0.08
Grows beyond initial features range
Table 6: Gompertz regimes. The three dynamics corresponds to biological relevant tumor growth patterns, namely regression, stable and progression.
Family
Flow class
Regime signal
Propagation pattern
R2
Saturating regulatory influence
Degree-averting
Peripheral pathways dominate
P
Self-limited growth with neighbor reinforcement
Degree-driven
Hubs dominate propagation
E
Recovery and susceptibility-limited contagion
Homogeneous
Propagation is broadly distributed
Table 7: Network interaction regimes. The three network dynamics differ in the mechanism by which information propagates over the same graph topology.
Regime type
Example of Action Names
Global Motion Pattern
Step/Walk
‘walk away‘, ‘slow walk‘, ‘step forward‘, …
Directional feet movement
Stand
‘look‘, ‘poses‘, ‘wait‘, …
Most body static
Turn
‘twist body to the left‘, ‘rotate‘, ‘twirl‘, …
Body rotating
Sit/Lower
‘sit‘, ‘lean‘, ‘crouch‘, …
Body joints lowering to the ground
Table 8: AMASS/BABEL regimes. The four dynamics corresponds to dominant motion patterns in the AMASS [ Mahmood et al., 2019a ] dataset aligned with BABEL labels [ Punnakkal et al., 2021a ] .
Synthetic
Real-world
Model
Linear
Gompertz [ Gompertz, 1825 ]
Network [ Harush and Barzel, 2017 ]
NBA [ Linou et al., 2016 ]
AMASS [ Mahmood et al., 2019a ]
Persistence (Static)
5.518 ( ± 0.055)
2.498 ( ± 0.025)
0.103 ( ± 0.001)
4.51 ( ± 0.07) / 7.78 ( ± 0.13)
0.092 ( ± 0.002) / 0.133 ( ± 0.003)
Persistence (Velocity)
2.911 ( ± 0.029)
2.881 ( ± 0.029)
0.057 ( ± 0.001)
2.51 ( ± 0.05) / 5.81 ( ± 0.10)
0.131 ( ± 0.003) / 0.276 ( ± 0.006)
S4 Gu et al. [2022]
4.156 ( ± 0.125)
4.378 ( ± 0.131)
0.165 ( ± 0.006)
2.13 ( ± 0.08) / 5.04 ( ± 0.18)
0.428 ( ± 0.018) / 0.624 ( ± 0.028)
rSLDS Glaser et al. [2020]
2.720 ( ± 0.058)
3.133 ( ± 0.047)
0.054 ( ± 0.001)
2.93 ( ± 0.02) / 5.40 ( ± 0.05)
0.073 ( ± 0.004) / 0.133 ( ± 0.006)
RED-SDS Ansari et al. [2021]
3.388 ( ± 0.031)
3.152 ( ± 0.128)
0.046 ( ± 0.002)
4.72 ( ± 0.07) / 8.36 ( ± 0.13)
0.071 ( ± 0.002) / 0.119 ( ± 0.002)
Table 9: Regime-conditioned forecasting performance. Train ADE is reported for generated datasets, and ADE/FDE for motion datasets. Values are mean ( ± standard deviation) over 4 folds. Best results are in bold and second-best underlined. N/A means that the method was not applicable.
Synthetic
Real-world
Method
Linear
Gompertz
Network
NBA
AMASS
Latent-HMM
0.979 ( ± 0.051)
0.965 ( ± 0.044)
0.716 ( ± 0.023)
0.604 ( ± 0.016)
0.329 ( ± 0.017)
rSLDS Glaser et al. [2020]
0.958 ( ± 0.059)
0.946 ( ± 0.016)
0.331 ( ± 0.024)
0.362 ( ± 0.019)
0.330 ( ± 0.025)
RED-SDS Ansari et al. [2021]
0.946 ( ± 0.026)
0.939 ( ± 0.019)
0.570 ( ± 0.007)
0.493 ( ± 0.023)
0.318 ( ± 0.017)
HERON
0.948 ( ± 0.031)
0.931 ( ± 0.012)
0.920 ( ± 0.009)
0.878 ( ± 0.011)
0.419 ( ± 0.018)
Table 10: Self-supervised regime recovery. Train macro-F1 quantifies how well the learned regimes align with ground-truth dynamical labels, which are used only for evaluation. Best results are in bold and second-best results are underlined. Values are mean ( ± standard deviation) over 4 folds.
Network
True regime
Clamped regime
Regulatory
Epidemic
Mutualistic
Regulatory
1.00
1.42
1.19
Epidemic
1.83
1.00
1.29
Mutualistic
1.21
1.15
1.00
Table 11: Forecasting results with regime clamping. Rows indicate the regime imposed during forecasting, while columns indicate the true regime. ADE is normalized column-wise by the matched intervention, such that diagonal entries equal 1.
AMASS condition
Bone-length violation (%)
Motion jerk ( × real)
Ground-truth future
0.00
1.00
Clamp = correct regime
2.51
2.89
Clamp = wrong regime
2.66
2.87
Table 12: Constraint violation with regime clamping on AMASS.
NBA condition
Out-of-court (%)
Player acceleration ( × real)
Ground-truth future
2.21
1.00
Clamp = correct regime
3.37
1.19
Clamp = wrong regime
3.65
1.20
Table 13: Constraint violation with regime clamping on NBA.
Table 15: Overlapping regimes. Hard versus posterior-weighted soft routing.
Dataset
K
Inter-run ARI
Matched accuracy
Worst-mode Jaccard
Null ARI (95th %ile)
Network
3
0.92±0.02
0.98±0.02
0.96±0.03
0.005
NBA
2
0.83±0.04
0.96±0.01
0.90±0.02
0.004
AMASS
4
0.48±0.07
0.61±0.08
0.57±0.05
0.005
Table 16: Cross-seed regime reproducibility. Statistics over all ten pairs from five independent seeds.
Synthetic
Real-world
Metric
Linear
Gompertz
Network
NBA
AMASS
Macro-F1
0.766
0.703
0.790
0.872
0.32
AUC
0.703
0.944
0.868
0.899
0.408
Table 17: Self-supervised regime recovery. Test macro-F1 and AUC quantify how well learned regimes align with ground-truth dynamical labels, which are used only for evaluation.
Network
NBA
K
ADE
macro-F1
AUC
Experts used
K
ADE
FDE
macro-F1
AUC
Experts used
2
0.043
0.55
0.75
2
2
0.81
1.49
0.89
0.91
2
3
0.038
0.91
0.97
3
3
0.89
1.73
0.83
0.86
2
4
0.038
0.81
0.88
3
4
0.84
1.67
0.87
0.87
2
5
0.041
0.73
0.76
3
Table 18: Sensitivity to misspecification of the number of regimes. Bold K marks the reference regime count; metric bolding follows the best value in each dataset.
Selection rule
Network ( K=3 )
NBA ( K=2 )
AMASS ( K=4 )
Expert usage
3
2
6
Effective- K
3
2
5
Held-out likelihood
3
2
4
Velocity baseline
2
4
5
Table 19: Label-free K selection and diagnostics. Reference regime counts are shown in parentheses.
Hyperparameter
Synthetic Datasets
NBA Dataset
AMASS Dataset
batch size
256
64
128
entity embedding dim
32
64
64
encoder hidden dim
48
32
32
encoder num layers
2
2
4
entity hidden dim
32
48
16
system hidden dim
32
32
32
Table 20: Hyperparameters for all datasets. List of the relevant hyperparameters and their values per dataset.
Model
Dataset/configuration
Trainable parameters
GraphSSM
Synthetic
300K
GraphSSM
NBA
450K
GraphSSM
AMASS
450K
HERON
Synthetic
400K
HERON
NBA
600K
HERON
AMASS
600K
Table 21: Trainable parameter counts. Counts correspond to the instantiated configurations used in the experiments.
Action-conditioned time-series forecasting requires accounting for how future actions and exogenous forcings influence multiple targets through partially observed effects with different delays and persistence. Direct conditioning leaves the evolution and target-specific influence of these effects implicit in the predictor, while static relational graphs specify connections without tracking evolving effects. This motivates representing future-driver influence through structured latent states that evolve over the forecast horizon and route information to individual targets. We introduce BeliefGraph-JEPA, a structured latent world model that factorizes driver influence into typed latent-effect states. These states are rolled forward under future drivers and routed through a graph to target-specific nodes, forming the predictive base of a joint-embedding predictive architecture. A capacity-controlled residual supplements this base with direct driver information. On four multi-target clinical, agricultural, environmental, and industrial systems, the framework outperforms a range of pretrained and supervised known-future-covariate baselines. Matched controls isolate latent dynamics, future rollout, graph routing, and residual capacity; future rollout and graph-first residual routing improve forecasting across all four systems.
Yue Li, Kangqi Ni, Zhen Tan +1
Carnegie Mellon University · University of North Carolina at Chapel Hill · Stevens Institute of Technology
Spatio-temporal processes often exhibit highly heterogeneous and non-intuitive responses to localized disruptions, limiting the effectiveness of conventional message passing approaches in modeling local heterogeneity. We reformulate spatio-temporal forecasting as the problem of learning information flow over locally structured spaces, rather than propagating globally aligned node representations. To this end, we introduce a spatio-temporal sheaf diffusion graph neural network (ST-Sheaf GNN) that embeds graph topology into sheaf-based vector spaces connected by learned linear restriction maps. Unlike prior approaches relying on static or globally shared transformations, our model learns dynamic restriction maps that evolve over time and adapt to local spatio-temporal patterns, enabling more expressive interactions. The proposed framework both theoretically guarantees and empirically demonstrates evidence that the proposed diffusion mechanism mitigates oversmoothing, preserving discriminative node representations even with increasing diffusion layer depth. Experiments on diverse real-world spatio-temporal forecasting benchmarks across multiple domains demonstrate state-of-the-art performance, highlighting the effectiveness of sheaf topological representations as a principled foundation for spatio-temporal graph learning. The code is available at: https://anonymous.4open.science/r/ST-SheafGNN-6523/.
Abeer Mostafa, Raneen Younis, Zahra Ahmadi
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Hannover, Germany · Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed), Hannover, Germany
Hierarchical forecasting exploits aggregation structures that expose multiple views of the same underlying spatiotemporal forecasting problem, while ensuring coherence among forecasts. Individual series provide fine-grained information, while progressively coarser aggregates reveal temporal and spatial patterns at different scales. In the associated deep learning architectures, the hierarchy determines which aggregate series are available and how they are processed together with individual series (e.g., through reconciliation or message-passing operators). Different hierarchies thus provide distinct learning signals and inductive biases for forecasting the same collection of time series. Nonetheless, existing architectures are typically trained on a single, often fixed, structure. In this work, we propose a graph deep learning framework that leverages variations across hierarchies as multi-task regularization. We show that jointly learning across these structures can reduce over-specialization to any single hierarchy and improve forecasting performance. Indeed, training across hierarchies provides an inductive bias toward representations that are useful across different structures, and as such, generalize better. Making forecasts coherent with the aggregation constraints for each sampled hierarchy is computationally expensive: to complement the framework, we propose a novel operator-based on a top-down approach-that produces coherent forecasts with linear time and space complexity w.r.t. the hierarchy size. Experiments show that our approach improves forecasting accuracy over training on a single structure and is competitive with current state-of-the-art methods.
Valentina Moretti, Cesare Alippi, Andrea Cini
IDSIA USI-SUPSI, Universit`a della Svizzera italiana · Politecnico di Milano · EPFL