Relative Geometry of Neural Forecasters: Linking Accuracy and Alignment in Learned Latent Geometry

Deniz Kucukahmetler; Maximilian Jean Hemmann; Julian Mosig von Aehrenfeld; Maximilian Amthor; Christian Deubel; Nico Scherf; Diaaeldin Taha

cs.LGFeb 17, 2026

Relative Geometry of Neural Forecasters: Linking Accuracy and Alignment in Learned Latent Geometry

Authors: Deniz Kucukahmetler, Maximilian Jean Hemmann, Julian Mosig von Aehrenfeld, Maximilian Amthor, Christian Deubel, Nico Scherf, Diaaeldin Taha

Organizations: Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany · School of Embedded Composite Artificial Intelligence (SECAI), Dresden/Leipzig, Germany · Leipzig University, Leipzig, Germany · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Dresden/Leipzig, Germany ·

Abstract

Neural networks can accurately forecast complex dynamical systems, yet how they internally represent underlying latent geometry remains poorly understood. We study neural forecasters through the lens of representational alignment, introducing anchor-based, geometry-agnostic relative embeddings that remove rotational and scaling ambiguities in latent spaces. Applying this framework across seven canonical dynamical systems - ranging from periodic to chaotic - we reveal reproducible family-level structure: multilayer perceptrons align with other MLPs, recurrent networks with RNNs, while transformers and echo-state networks achieve strong forecasts despite weaker alignment. Alignment generally correlates with forecasting accuracy, yet high accuracy can coexist with low alignment. Relative geometry thus provides a simple, reproducible foundation for comparing how model families internalize and represent dynamical structure.

Figures & tables

Figure 1: Relative embeddings reveal consistent geometric structure across model families while removing rotational and scaling ambiguities. (a) Encoder–propagator–decoder forecasters take an input window of $L$ past states $\mathbf{x}_{t-L+1:t}$ , embed it into a latent vector $\mathbf{z}$ , and decode a prediction of the next $H$ states $\widehat{\mathbf{x}}_{t+1:t+H}$ . To compare different forecasters, we compute absolute latent embeddings from data, transform them into anchor-based relative embeddings using Moschella et al. (2023) , and quantify alignment between forecasters using representational similarity scores. (b) Alignment–performance endpoints after training for RNNs (blue) and MLPs (green). RNNs achieve higher representational similarity and prediction accuracy (MSE), while MLPs show a clearer correlation between alignment and performance across seeds. (c-f) Example systems: Lorenz–63 (c), double pendulum (d), random skew (e), limit cycle (f). Columns display system trajectories, absolute embeddings (PCA; two or three principal components depending on dimensionality), relative embeddings (PCA), cross-forecaster similarity heatmaps averaged over five seeds—ordered as True System, MLP , Koopman MLP , NODE MLP , RNN , Autoregressive RNN , Koopman RNN , NODE RNN , Transformer , NODE Transformer , Koopman Transformer , and ESN ; —and alignment–performance scatter plots across hyperparameter settings. Additional systems are shown in Appendix Figure 4 and 5 .

Trajectories and dynamical systems (Section 3.2 )
Symbol	Description
$T$	Time index set (discrete or continuous).
$X$	State space of the dynamical system.
$\Phi:T\times X\rightarrow X$	Evolution map of the dynamical system.
$x\in X$	Initial condition or system state.
$\Phi_{x}(t)$	Trajectory starting from $x$ .

Table 1: Summary of notation used for trajectories, latent states, anchors, and representational similarity.

Model	Encoder	Propagator	Decoder
MLP	MLP (feed-forward)	Identity ( $\mathcal{P}(\mathbf{z})=\mathbf{z}$ )	MLP (feed-forward)
RNN	RNN (GRU)	Identity	RNN (GRU)
A-RNN	RNN (GRU, autoregressive)	Identity	RNN (GRU, autoregressive)
Transformer (TF)	Transformer (causal attention)	Identity	Transformer (causal attention)
N–MLP, RNN, TF	Same as base model	NODE: $\dot{\mathbf{z}}=f_{\Theta}(\mathbf{z},t)$	Same as base model
K–MLP, RNN, TF	Same as base model	Linear: $\mathbf{z}_{k+1}=\bm{K}\mathbf{z}_{k}$	Same as base model

Table 2: Encoder–Propagator–Decoder decomposition across model families.

	Performance ( $\downarrow$ )			Similarity ( $\uparrow$ )
Model	MSE	MAE	RMSE	Cosine	Top-1	Spearman $\rho$
MLP	0.3828 ± 0.0080	0.2804 ± 0.0050	0.6187 ± 0.0064	0.7148 ± 0.0193	0.3786 ± 0.0025	0.7121 ± 0.0187
Koopman MLP	0.3597 ± 0.0288	0.2824 ± 0.0179	0.5994 ± 0.0245	0.6372 ± 0.0252	0.3722 ± 0.0048	0.6344 ± 0.0271
NODE MLP	0.3684 ± 0.0081	0.2939 ± 0.0176	0.6070 ± 0.0067	0.7489 ± 0.0088	0.3928 ± 0.0008	0.7476 ± 0.0115
RNN	0.0096 ± 0.0018	0.0570 ± 0.0073	0.0979 ± 0.0092	0.9125 ± 0.0128	0.5100 ± 0.0162	0.9031 ± 0.0117
Autoregressive RNN	0.0422 ± 0.0082	0.1172 ± 0.0106	0.2045 ± 0.0203	0.9084 ± 0.0124	0.5084 ± 0.0124	0.8990 ± 0.0150

Table 3: Performance and Alignment for lorenz

Figure 2: Performance–alignment trade-offs across training, noise, and input conditions. Columns show (a-m) training time evolution, (b-n) effects of input noise, (c-o) effects of sequence length $L$ , and (d-p) test performance across model families ( MLP , K-MLP: Koopman MLP, N-MLP: NODE MLP, RNN , A-RNN: Autoregressive RNN, K-RNN: Koopman RNN, N-RNN: NODE RNN, TF: Transformer, N-TF: NODE Transformer, K-TF: Koopman Transformer, ESN ). Each point represents the mean squared error (MSE) and the representational similarity score (RSS) of a given a forecaster trained with a different random seed (color-coded by forecaster family; same-colored lines/points denote different initializations of the same forecaster). MLPs and RNNs exhibit consistent performance–alignment relationships, while transformers show larger variability; ESNs are excluded due to their no–backpropagation-through-time (no-BPTT) training. Increasing input noise consistently degrades both alignment and accuracy, whereas varying $L$ produces system-dependent effects, highlighting differences in robustness across model families. Test results (d-p) indicate that no single family dominates across all dynamical systems. Results for additional systems in Appendix 6 .

Figure 3: Temporal evolution of representational alignment across dynamical systems. Each row shows the true system (left), reconstructed trajectories from different model families (MLPs, RNNs, transformers, ESN; same colour coding as in Figure 2 ), and their temporal similarity profiles (right; line thickness encodes time, $T=300$ ). For visualisation purposes we use relative coordinates $z^{\prime}_{i}$ with respect to three anchor points (axes $z^{\prime}_{1}$ , $z^{\prime}_{2}$ , $z^{\prime}_{3}$ ). For the Lorenz–63, double pendulum, and random skew systems, MLPs and RNNs maintain representations closely aligned with the true dynamics, whereas transformers and ESNs diverge. For the limit cycle and other periodic systems (see Appendix 7 ), all families capture similarly structured representations. The Logistic map is omitted due to its one-dimensional, contractive behavior.

enc/dec	MLP		N-MLP		K-MLP		TF		N-TF		K-TF
	Abs.	Rel.	Abs.	Rel.	Abs.	Rel.	Abs.	Rel.	Abs.	Rel.	Abs.	Rel.
MLP	1.655	0.383	2.334	0.479	2.459	2.818	0.293	0.825	1.181	1.067	0.923	0.988
N-MLP	2.195	0.404	3.916	0.491	3.511	3.078	0.233	0.813	0.545	1.040	1.235	0.925
K-MLP	1.621	0.753	2.224	0.759	2.523	0.891	0.290	0.587	1.233	0.974	0.835	0.679
TF	1.538	2.019	2.389	1.754	2.118	9.517	0.265	0.043	1.383	0.587	0.599	0.076
N-TF	1.514	1.780	2.003	1.580	2.039	7.112	0.184	0.061	1.017	0.757	0.689	0.256

Table 4: Cross-architecture average stitching loss (MSE) over encoder–decoder pairs for absolute (Abs.) and relative (Rel.) stitching. Each decoder column is independently normalized; darkest cell shows highest MSE and lightest shows lowest MSE respectively. Lower value per pair in bold.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Forecasting and representational alignment. (a, b) Example systems: proper orthogonal decomposition (POD)-wake (a), Hopf (b). Columns show time series trajectories, absolute embeddings (visualized with principal component analysis (PCA); we plot the first 2 components for 2-dimensional systems and 3 components for the rest of the systems), relative embeddings (PCA), cross-forecaster similarity heatmaps (averaged over five seeds) with the order of True System, MLP , Koopman MLP , NODE MLP , RNN , Autoregressive RNN , Koopman RNN , NODE RNN , Transformer , NODE Transformer , Koopman Transformer , and ESN ; and alignment–performance of forecasters with different hyperparameter settings.

	Performance ( $\downarrow$ )			Similarity ( $\uparrow$ )
Model	MSE	MAE	RMSE	Cosine	Top-1	Spearman $\rho$
MLP	0.4122 ± 0.0096	0.4573 ± 0.0049	0.6420 ± 0.0075	0.6745 ± 0.0105	0.4262 ± 0.0065	0.6754 ± 0.0115
Koopman MLP	0.4314 ± 0.0267	0.4871 ± 0.0202	0.6565 ± 0.0202	0.6097 ± 0.0241	0.4112 ± 0.0076	0.6033 ± 0.0287
NODE MLP	0.3573 ± 0.0102	0.4364 ± 0.0033	0.5977 ± 0.0085	0.6212 ± 0.0076	0.4186 ± 0.0025	0.6211 ± 0.0100
RNN	0.2879 ± 0.0117	0.3618 ± 0.0095	0.5365 ± 0.0109	0.8254 ± 0.0059	0.5044 ± 0.0151	0.8197 ± 0.0048
Autoregressive RNN	0.3994 ± 0.0494	0.4450 ± 0.0407	0.6310 ± 0.0397	0.8791 ± 0.0241	0.5526 ± 0.0434	0.8708 ± 0.0256

Appendix

Table 5: Results for double_pendulum

	Performance ( $\downarrow$ )			Similarity ( $\uparrow$ )
Model	MSE	MAE	RMSE	Cosine	Top-1	Spearman $\rho$
MLP	0.9778 ± 0.0532	0.6652 ± 0.0174	0.9886 ± 0.0271	0.7533 ± 0.0227	0.8138 ± 0.0034	0.7724 ± 0.0159
Koopman MLP	0.8137 ± 0.1571	0.5918 ± 0.0409	0.8989 ± 0.0837	0.8523 ± 0.0229	0.8304 ± 0.0038	0.8440 ± 0.0279
NODE MLP	0.9145 ± 0.0960	0.6545 ± 0.0195	0.9552 ± 0.0509	0.7384 ± 0.0174	0.8090 ± 0.0047	0.7536 ± 0.0159
RNN	0.8815 ± 0.0283	0.6481 ± 0.0081	0.9388 ± 0.0151	0.8155 ± 0.0204	0.7840 ± 0.0091	0.8058 ± 0.0209
Autoregressive RNN	0.7683 ± 0.0171	0.5823 ± 0.0089	0.8765 ± 0.0097	0.8099 ± 0.0153	0.8178 ± 0.0134	0.8059 ± 0.0153

Appendix

Table 6: Results for random_skew

	Performance ( $\downarrow$ )			Similarity ( $\uparrow$ )
Model	MSE	MAE	RMSE	Cosine	Top-1	Spearman $\rho$
MLP	0.0002 ± 0.0000	0.0059 ± 0.0004	0.0128 ± 0.0010	0.8848 ± 0.0122	0.4020 ± 0.0029	0.8788 ± 0.0104
Koopman MLP	0.0002 ± 0.0001	0.0049 ± 0.0006	0.0136 ± 0.0035	0.8700 ± 0.0200	0.4200 ± 0.0035	0.8573 ± 0.0232
NODE MLP	0.0003 ± 0.0001	0.0067 ± 0.0012	0.0178 ± 0.0028	0.8796 ± 0.0127	0.4004 ± 0.0067	0.8698 ± 0.0121
RNN	0.0002 ± 0.0001	0.0108 ± 0.0017	0.0153 ± 0.0027	0.9917 ± 0.0039	0.6222 ± 0.0442	0.9906 ± 0.0038
Autoregressive RNN	0.0017 ± 0.0004	0.0298 ± 0.0044	0.0407 ± 0.0044	0.9936 ± 0.0040	0.5424 ± 0.0452	0.9921 ± 0.0033

Appendix

Table 7: Results for spiral

	Performance ( $\downarrow$ )			Similarity ( $\uparrow$ )
Model	MSE	MAE	RMSE	Cosine	Top-1	Spearman $\rho$
MLP	0.0270 ± 0.0046	0.0862 ± 0.0071	0.1637 ± 0.0141	0.9511 ± 0.0107	0.8886 ± 0.0079	0.9024 ± 0.0175
Koopman MLP	0.0331 ± 0.0111	0.0931 ± 0.0125	0.1800 ± 0.0303	0.9277 ± 0.0179	0.8810 ± 0.0099	0.8679 ± 0.0308
NODE MLP	0.0317 ± 0.0075	0.0942 ± 0.0115	0.1770 ± 0.0227	0.9367 ± 0.0063	0.8758 ± 0.0118	0.8783 ± 0.0082
RNN	0.0242 ± 0.0063	0.0841 ± 0.0110	0.1547 ± 0.0195	0.9402 ± 0.0586	0.8876 ± 0.0558	0.9170 ± 0.0467
Autoregressive RNN	0.0422 ± 0.0097	0.1305 ± 0.0096	0.2043 ± 0.0239	0.9781 ± 0.0086	0.9328 ± 0.0186	0.9450 ± 0.0197

Appendix

Table 8: Results for pod

	Performance ( $\downarrow$ )			Similarity ( $\uparrow$ )
Model	MSE	MAE	RMSE	Cosine	Top-1	Spearman $\rho$
MLP	0.0002 ± 0.0001	0.0080 ± 0.0006	0.0146 ± 0.0028	0.9672 ± 0.0088	0.5190 ± 0.0078	0.9596 ± 0.0119
Koopman MLP	0.0002 ± 0.0001	0.0061 ± 0.0006	0.0135 ± 0.0039	0.9463 ± 0.0119	0.5440 ± 0.0175	0.9300 ± 0.0227
NODE MLP	0.0003 ± 0.0001	0.0096 ± 0.0007	0.0169 ± 0.0027	0.9550 ± 0.0082	0.5160 ± 0.0162	0.9463 ± 0.0107
RNN	0.0001 ± 0.0000	0.0042 ± 0.0022	0.0070 ± 0.0031	0.9907 ± 0.0040	0.5474 ± 0.0297	0.9905 ± 0.0044
Autoregressive RNN	0.0015 ± 0.0001	0.0332 ± 0.0009	0.0386 ± 0.0012	0.9888 ± 0.0068	0.5638 ± 0.0358	0.9904 ± 0.0052

Appendix

Table 9: Results for hopf

	Performance ( $\downarrow$ )			Similarity ( $\uparrow$ )
Model	MSE	MAE	RMSE	Cosine	Top-1	Spearman $\rho$
MLP	0.0002 ± 0.0000	0.0118 ± 0.0006	0.0157 ± 0.0008	0.9489 ± 0.0074	0.0232 ± 0.0040	0.7468 ± 0.0035
Koopman MLP	0.3713 ± 0.4895	0.4102 ± 0.4452	0.4505 ± 0.4586	0.9906 ± 0.0047	0.0232 ± 0.0051	0.7468 ± 0.0044
NODE MLP	0.8986 ± 0.0191	0.8939 ± 0.0109	0.9479 ± 0.0101	0.9165 ± 0.0085	0.0228 ± 0.0046	0.7446 ± 0.0044
RNN	0.0025 ± 0.0038	0.0325 ± 0.0245	0.0421 ± 0.0302	0.7056 ± 0.1526	0.0302 ± 0.0059	0.6182 ± 0.1131
Autoregressive RNN	0.0065 ± 0.0041	0.0648 ± 0.0197	0.0771 ± 0.0255	0.6875 ± 0.2004	0.0240 ± 0.0074	0.5479 ± 0.2504

Appendix

Table 10: Results for logistic_map

Figure 5: Cross-model similarity in Logistic Maps.

Figure 6: Perturbation experiments for POD-wake, Hopf and Logistic Maps. (X) The noise experiment could not be performed because the data were obtained from Brunton et al. (2016) , and only the principal components—not the original data—were available.

Figure 7: Temporal alignment visualization for POD-wake and Hopf.

Figure 8: Temporal alignment rotated visualization of the True System of Lorenz.

Figure 9: Anchor ablation and baseline. (Blue) Alignment vs. number of anchors $K$ ; lines show mean over 30 repeats. Stabilization occurs for $K\geq 16$ ; we choose $K=80$ (vertical marker) for the main experiments. (Orange) Random baseline with disjoint anchor sets across spaces, yielding near-zero alignment.

Figure 10: Hyperparameter Tuning for All Systems and Models. ESN is exluded since it needed additional manual hyperparameter tuning due to its sensitivity to hyperparameters and unstable nature.

Figure 11: Cross-model similarity using absolute embeddings.

Figure 12: Average cosine similarity over all 5 seeds for each forecaster pair, excluding exact same forecaster and seed. Models were trained on relative latent spaces rather than absolute spaces.

Figure 13: Mean test-set $R^{2}$ of a linear ridge probe decoding the current system state $x(t)$ from absolute latent representations $z(t)$ for the Lorenz–63 system, averaged over three random seeds (error bars denote standard deviation).

Figure 14: Cross model alignment using centered kernel alignment (CKA), representational similarity analysis (RSA), and Procrustes-based alignment. Values indicate the average of three seeds.

Figure 15: A. Cross model alignment in forecasters trained on forecasting iEEG dataset. B. Performances of forecasters.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 255 papers

Sep 28, 2026

Evaluating Dynamical Fidelity through Predictive Structure in Physical Representations

Machine-learning models for physical systems are currently evaluated primarily through errors between predicted and reference states and, increasingly, through tests of physical consistency. These metrics assess whether predictions are accurate and satisfy selected physical requirements, but provide limited insight into whether learned trajectories reproduce the underlying dynamics. Domain experts examine such relationships through physical representations that expose relevant processes, interactions, and responses, but these analyses are often separated from typical machine-learning evaluation. We introduce a practical framework for evaluating dynamical fidelity through predictive structure in physical representation spaces. Experts define the representations, while reference trajectories determine which relationships are predictive and retained as evaluation tests. We demonstrate the approach in atmospheric forecasting using ERA5 representations of planetary-wave activity and Northern Annular Mode evolution, and evaluate Pangu-Weather, GraphCast, and FengWu. The models exhibit distinct departures from reference predictive structure that are not reflected by conventional forecast errors. The framework thereby turns domain-expert representations into systematic tests of learned physical dynamics without prescribing the relationships in advance.

Oskar Bohn Lassen, Joao Paulo de Souza Boger, Simon Driscoll +4

Department of Technology, Management, and Economics Technical University of Denmark · Department of Applied Mathematics and Theoretical Physics University of Cambridge · Department of Mathematics and Statistics University of Exeter

Spatiotemporal Forecasting Weather Forecasting

Ask PDF

Aug 4, 2026

SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors

Joint-embedding predictive architectures learn abstract states by predicting target embeddings from context embeddings, but their transition models are typically opaque neural maps. We introduce SJEPA, a reconstruction-free JEPA framework that learns predictive representations whose induced dynamics admit compact symbolic descriptions. Its hybrid transition combines a symbolic law with a regularised neural correction for dynamics outside the selected grammar. The central principle is to learn the simplest adequate dynamics: representation constraints preserve informative, non-collapsed predictive coordinates, while operator compression favours low-complexity symbolic-neural transitions that remain predictively adequate. We formalise this principle through induced-dynamics complexity, analyse predictive-coordinate non-identifiability, and show that unconstrained operator compression creates a direct shortcut to representation collapse. The framework supports both alternating representation-equation learning and symbolic dynamics fitted to fixed representations. In controlled pendulum experiments, joint learning discovers substantially simpler symbolic dynamics with lower long-horizon rollout error and divergence than post-hoc fitting, while an unconstrained one-step diagnostic realises the predicted collapse shortcut. Under grammar misspecification, correction regularisation preserves the representable symbolic mechanism and directs the neural component towards residual dynamics. The results expose a controllable trade-off among predictive fidelity, representation quality, symbolic parsimony, and symbolic-neural allocation.

Yongchao Huang

Latent Dynamics Modeling Joint-Embedding Predictive Architecture Predictive Representation Learning Neurosymbolic Learning

Ask PDF

Mar 31, 2026

Similarity of Neural Network Representations in Superposition

Comparing internal representations is a central goal in neuroscience and machine learning, but standard linear alignment metrics (Representational Similarity Analysis, Centered Kernel Alignment, and linear regression) are frequently applied to neural activity coordinates rather than on the underlying features. We show this matters when neural systems operate in superposition, encoding more features than they have neurons via linear compression. Closed-form derivations prove that these metrics depend on the Gram matrices of each system's projection, not on the latent features themselves: alignment thus combines what a system represents with how it is encoded. For those interested in what features two systems share, this is a problem: Two networks can have identical feature content yet appear more dissimilar than networks exhibiting partial feature overlap. This apparent misalignment need not reflect lost information as compressed sensing guarantees sparse features remain recoverable from the compressed activity. We confirm this by training supervised TopK sparse autoencoders that realize solvable compressed sensing by construction, finding alignment on recovered latents restored even when raw-activation alignment remains deflated. We extend the result to unsupervised SAEs trained without ground-truth latents, and to pretrained vision and language model SAEs, where SAE-latent alignment exceeds raw-activation alignment, consistent with superposition in real systems.

Sunny Liu, Habon Issa, André Longon +4

Cold Spring Harbor Laboratory · UC San Diego · Anthropic +2

Feature Superposition Representational Similarity Analysis Neural Representation Geometry Compressed Sensing Neural Encoding

Ask PDF

Abstract

Neural Representation Geometry

Explore similar work

Evaluating Dynamical Fidelity through Predictive Structure in Physical Representations

SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors

Similarity of Neural Network Representations in Superposition