Intrinsic explainability remains a challenging problem, particularly in contexts where multilayer perceptrons (MLPs) require dynamic re-training within an optimization environment. This paper investigates how MLPs and their training dynamics can be represented and studied in non-Euclidean spaces; our representation features the Poincaré model of hyperbolic geometry. We aim to capture the geometric evolution of their weighted topology and self-organization over time. Instead of restricting the analysis to single checkpoints---as per established measure-based explainability methods---we construct temporal parameter graphs, i.e., snapshots over time T steps of the optimization/training process for MLPs. This reflects the view that neural networks encode information not only in their weights but also in the trajectory traced during training. Drawing on the idea that many complex networks admit embeddings in hidden metric spaces where distances correspond to connection likelihood, we present a geometric and temporal graph-based metalearning framework for obtaining dynamic hyperbolic representations of the underlying neural parameter graphs. Our model embeds temporal parameter graphs in the Poincaré model ball, and learns from them while maintaining equivariance to within-snapshot neuron permutations and invariance to permutations of past snapshots. In doing so, the approach preserves functional equivalence over time and recovers the latent evolving geometry of the network. Experiments on regression and classification tasks with trained MLPs show strong meta-network performance, accompanied by hyperbolic temporal representations. This reveals how the network structure emerges over time under specific training environments, thus providing insights into the network's self-organization.
Figures & tables
Figure 1: Architecture of the Geometric Temporal Hyperbolic Graph Meta-Network (GTH-GMN).
Model
MNIST INR
Fashion-MNIST INR
MLP
17.55 ± 0.01
19.91 ± 0.47
MLP + Perm. aug
29.26 ± 0.18
22.76 ± 0.13
MLP + Alignment
58.98 ± 0.52
47.79 ± 1.03
INR2Vec (Arch.)
23.69 ± 0.10
22.33 ± 0.41
Transformer
26.57 ± 0.18
26.97 ± 0.33
DWSNets
85.71 ± 0.57
67.06 ± 0.29
Table 1: INR classification in static vs. temporal settings. Average test accuracies ( ± standard error) on MNIST and Fashion-MNIST INRs. Baselines are reproduced from DWSNets ( Navon et al., 2024 ) and NFNs ( Zhou et al., 2023a ) . Our spatio-temporal variant uses pooled representations across K traces.
Table 3
Figure 2: Hyperbolic embeddings of parameter graphs from a 13-layer MLP trained as an INR on CIFAR-10, shown at checkpoints t=11 (left) and t=97 (right). Embeddings are learned in d=10 dimensions with 4 HGAT layers and 4 EvolveGCN–H kernels (tangent dim. 128) and projected to the Poincaré disk via PCA. Nodes are colored by layer index, with edges in gray, illustrating the progressive reorganization and separation of layers over training.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter Graphs & Features
Gt
Parameter-graph snapshot at epoch t
V,Et
Node set; edge set at epoch t
wuv⋆
Signed edge label on {u,v}
euv
Edge context (bias/kernel, interlayer flag)
L,d,c
#layers; embedding dim; curvature parameter
ℓ(u)
Layer label of node u
Appendix
Table 5
Figure 3: Predicted versus actual generalization performance on CIFAR-10 checkpoint traces. Each point corresponds to a model trial at a given epoch, where predictions are obtained from the learned GTH-GMN embeddings. The strong monotonic trend (Kendall’s τ=0.846±0.004 ) indicates that hyperbolic embeddings capture meaningful structure in the parameter dynamics, even if absolute correlation lags slightly behind NFN baselines.
Figure 7
Figure 4: Relationship between hyperbolic distance d and edge weights w on CIFAR-10 INR traces, shown as log-count hexbins. (Top Left, Bottom Left) Distance versus absolute weight ∣w∣ (at t∈T ), highlighting the concentration of large weights at small distances and the rapid decay at larger distances. (Top Right, Bottom Right) Distance versus log-magnitude log∣w∣ (at t′∈T ), which reveals an approximately linear inverse trend, consistent with a power–law dependence of weight magnitudes on hyperbolic distance. Together, these plots illustrate that the hyperbolic geometry learned by GTH-GMN aligns distances with functional weight strengths, capturing both edge existence and graded magnitude in a geometrically coherent manner.
Component
Hyperparameter
Range / Value
Manifold
Curvature ∣c∣
fixed to 1.0
Embedding dimension d
80−125
HGAT (per layer)
# layers L
4
# heads H
4−8 (tuned)
Head temperature
learned, init 0.5−1.0
Residual gate
learned scalar ∈(0,1)
Appendix
Table 5: Architecture and data settings for our Experiments.
Component
Hyperparameter
Range / Value
Euclidean params
Optimizer / LR
Adam, lr=1×10−3−3×10−3
(kernels, heads)
Weight decay
10−6−10−4
LR schedule
StepLR ( step=10−20 , γ=0.6−0.8 )
Epochs
30−50
Manifold params
Optimizer
RAMSGrad (Riemannian)
(node positions z )
LR / trust radius
lr=1×10−3−3×10−3 , τ=0.05−0.15
Appendix
Table 6: Optimization and loss settings for INR-trace image classification. Ranges correspond to grid or random search over hyperparameters.
∇zi(t)RiemL=λzi(t)−2∇zi(t)EucL
Appendix
Algorithm 1 Intrinsic Riemannian Optimization of Node Positions on the Poincaré Ball
Figure 5: Hyperbolic embeddings of parameter graphs from a 25-layer MLP trained as an INR on CIFAR-10, shown at checkpoints. Embeddings are learned in d=10,25 dimensions with 4 HGAT layers and 4 EvolveGCN–H kernels (tangent dim. 128) and projected to the Poincaré disk via PCA. Nodes are colored by layer index, with edges in gray, illustrating the progressive reorganization and separation of layers over training.
Variant
Question
LFD
Lwrec
Lsign
Lα
Lrank
Lsmooth
Full objective
–
✓
✓
✓
✓
✓
✓
No ranking loss
(1)
✓
✓
✓
✓
✗
✓
No ranking & no smoothness
(2)
✓
✓
✓
✓
✗
✗
No signed regression head
(3)
✓
✗
✗
✗
✓
✓
Appendix
Table 7: Ablation scenarios and their relation to the three guiding questions. Each row indicates which loss terms are active in the corresponding variant.
Condition
Metric
MNIST Value
Fashion-MNIST Value
Full Objective
τ
0.946
0.935
Slope ( b )
−0.617
−0.322
No Ranking Loss
τ
0.811
0.797
Slope ( b )
−0.473
−0.189
No Rank/Smooth
τ
0.765
0.764
Slope ( b )
−0.365
−0.061
Appendix
Table 8: Ablation Study Table
Figure 6: Kendall τ over the full objective for the INR classification task on MNIST and Fashion-MNIST datasets. GTH-GMN with full objective achieves a Kendall’s τ of 0.946 on MNIST and 0.935 on Fashion–MNIST datasets.
Figure 7: Analysis of the relationship between hyperbolic distance ( d ) and edge weights for the ”full objective” condition. The top row displays results for MNIST, while the bottom row shows Fashion-MNIST. For each dataset, the left image shows the hexbin plot of distance vs. log∣w∣ with a fitted linear regression line (dashed orange). The right image shows the hexbin plot of distance vs. the raw weight magnitude ∣w∣ . The fitted slope for MNIST in this condition ( −0.617d ) indicates a strong decay of edge weights, significantly steeper than for Fashion-MNIST ( −0.322d ), reflecting distinct geometric influences on model training for each dataset.
Figure 8: Kendall τ over the ”No ranking loss” condition for the INR classification task on MNIST and Fashion-MNIST datasets. GTH-GMN without Lrank achieves a Kendall’s τ of 0.811 on MNIST and 0.797 on Fashion–MNIST datasets, indicating a noticeable degradation in the monotonicity of the predictive signal compared to the full objective.
Figure 9: Analysis of the relationship between hyperbolic distance ( d ) and edge weights for the ”No ranking loss” condition. The top row displays results for MNIST, while the bottom row shows Fashion-MNIST. For each dataset, the left image shows the hexbin plot of distance vs. log∣w∣ with a fitted linear regression line (dashed orange). The right image shows the hexbin plot of distance vs. the raw weight magnitude ∣w∣ . The fitted slope for MNIST in this condition ( −0.473d ) indicates a moderate decay of edge weights, which is substantially shallower than the full objective model, while the decay for Fashion-MNIST ( −0.189d ) is notably weaker. This indicates that removing the ranking loss leads to a loss of the crisp, monotone geometric ordering.
Figure 10: Kendall τ over the ”No ranking and no temporal smoothness” condition for the INR classification task on MNIST and Fashion-MNIST datasets. GTH-GMN without both Lrank and Lsmooth achieves a Kendall’s τ of 0.765 on MNIST and 0.764 on Fashion–MNIST datasets, indicating a further decline in the ordering coherence of the predictive signal.
Figure 11: Kendall τ over the ”No signed regression head” condition for the INR classification task on MNIST and Fashion-MNIST datasets. GTH-GMN without the signed head achieves a Kendall’s τ of 0.738 on MNIST and 0.726 on Fashion–MNIST datasets, representing the strongest degradation in the geometric ordering of the predictive signal.
Figure 12: Analysis of the relationship between hyperbolic distance ( d ) and edge weights for the ”No ranking and no temporal smoothness loss” condition. The top row displays results for MNIST, while the bottom row shows Fashion-MNIST. For each dataset, the left image shows the hexbin plot of distance vs. log∣w∣ with a fitted linear regression line (dashed orange). The right image shows the hexbin plot of distance vs. the raw weight magnitude ∣w∣ . The fitted slope for MNIST in this condition ( −0.365d ) indicates a substantially shallower decay of edge weights compared to the full model, while the decay for Fashion-MNIST is minimal ( −0.061d ), reflecting a loss of coherent geometric structure when both regularization terms are removed.
Figure 13: Analysis of the relationship between hyperbolic distance ( d ) and edge weights for the ”No signed regression head” condition. The top row displays results for MNIST, while the bottom row shows Fashion-MNIST. For each dataset, the left image shows the hexbin plot of distance vs. log∣w∣ with a fitted linear regression line (dashed orange). The right image shows the hexbin plot of distance vs. the raw weight magnitude ∣w∣ . The fitted slope for MNIST in this condition ( −0.211d ) is near-zero, and the slope for Fashion-MNIST ( +0.125d ) is positive , indicating that the expected decay of weights with distance has collapsed or reversed. This confirms that the signed head is the primary mechanism binding the hyperbolic geometry to the functional magnitude of the weights.
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.