cs.LGNov 18, 2026

Learning Trajectories Leave Decodable Traces in Neural Network Weight Space and Internal Representations

Authors: Patrick Krauss, Achim Schilling, Andreas Maier, Thomas Kinfe, Claus Metzner

Organizations: Cognitive Computational Neuroscience Group, Pattern Recognition Lab, Friedrich-Alexander-University Erlangen-Nürnberg (FAU), Erlangen, Germany · Neural Interface Lab, University Hospital Mannheim, University Heidelberg, Mannheim, Germany · Neuroscience Lab, University Hospital Erlangen, Erlangen, Germany · BG Clinik Ludwigshafen, Ludwigshafen am Rhein, Germany

Abstract

Neural networks trained on the same task develop related weight-space structures, yet remain distinguishable after learning. Do these differences merely reflect alternative solutions, or do they preserve information about individual learning histories? We tested whether final weights reveal initialization, task order, and earlier trajectory states. Training reorganized the ensemble: initial pairwise distances did not consistently predict final distances. Nevertheless, all 30 trained networks were correctly matched to their initial weights among ten candidates per dataset. Networks sharing an initialization also developed recognizable final hidden representations across different batch schedules, although direct comparisons with initial activity did not recover identity. Learning order remained readable as well. A linear probe decoded the order of two earlier tasks with 100% accuracy on 80 held-out networks after ten epochs of a common final task. Earlier intermediate weight states were partly predictable, but individual initial weights could not be reliably reconstructed. Final weights thus retain selective historical information rather than a complete trajectory record. Perturbed networks recovered classification performance without returning to their unperturbed training paths. This separation persisted across datasets, intervention times, and network layers. Moreover, 2,157 of 2,160 aligned endpoint pairs showed no positive loss barrier at sampled interpolation points, indicating low-loss connections between distinct solutions rather than convergence to one weight vector. Together, these findings show that a shared task constrains networks to related successful solutions without erasing their individual histories. Final weights reflect not only what a network has learned, but also where learning began and how it unfolded.

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. When does a network's training history predict its future learning better than its current state? Evidence from a response probe and a forecasting screen

    Oct 7, 2026Martin Hofmann, Patrick MäderNeural Network Training Dynamics

  2. Early Learning Shapes Later Directions Of Representation Change In Continual Learning

    Sep 28, 2026Yuantao Deng, Jinnuo Liu, Kaizhen Tan +1Neural Representation GeometryVisual Representation Learning