Traditional autoencoders rely on static network architectures to learn latent representations. Recently we suggested a dynamic architecture inspired by Hebbian plasticity, enabling topological self-organization among convolutional feature maps (R. Koç et al., Proceedings of the SIU, 2026). Early implementations of this paradigm infer relationships among feature maps using global average pooling and model their interactions through scalar-valued connections. This approach falls short to fully exploit the underlying spatial information, embedded in input images. Furthermore, abrupt transitions between different loss functions and training dynamics can introduce representation shocks, leading to unstable feature learning. In this paper, we introduce a Spatially-Aware Self-Organizing Graph Autoencoder (S-LooM) to address these structural bottlenecks. By using spatial cross-correlation, and spatial kernels, S-LooM establishes dynamic connections among feature maps. We replace static scalar message passing of autoencoders with learnable spatial kernels and eliminate high-frequency deconvolutional artifacts observed in decoders using nearest-neighbor upsampling. Via a soft-blending training curriculum, the latent topology aligns smoothly during training. Our experiments show that S-LooM yields a more stable topology, compared to the state of the art dynamic architectures. Meaning, the graph structure that emerge has more resistance to feature entanglement and noise. It forms functional communities responsible for visual attributes, such as illumination, shape and texture even under high connection densities.
Figures & tables
Fig. 1: Graphical abstract of the S-LooM architecture. The encoder extracts feature maps, which branch into localized spatial correlations ( Cij ) to be used for the Hebbian Matrix ( WH ), and spatial maps themselves ( fi ) are fed into the Learnable Graph ( WL ). The red arrow between the graph structures represents the topological alignment loss.
Fig. 2: Qualitative evaluation of S-LooM reconstructions. The model yields high quality reconstructions while maintaining a strictly governed topological latent space.
Sparsity Ratio ( p )
Prior LooM Loss
S-LooM Loss
0.05
0.0272
0.0114
0.10
0.0283
0.0103
0.20
0.0305
0.0104
0.30
0.0322
0.0107
0.40
0.0340
0.0097
0.50
0.0361
0.0106
TABLE I: COMPARISON OF TARGET SPARSITY ON VALIDATION LOSS
Fig. 3: Matrix visualization comparing the converged learnable spatial graph and the localized Hebbian teacher graph with target sparsity p=0.05 , demonstrating strong topological alignment.
Fig. 4: Community Ablation Analysis. The difference maps isolate the distinct semantic responsibilities of each Louvain-detected community. Note the minimal impact of the isolated nodes, demonstrating the graph’s ability to quarantine uninformative filters.
Fig. 5: Single-Edge Lesioning. Severing a single graph connection (10 ↔ 31) results in a highly localized disruption (isolated in the difference map), indicating the network’s spatial precision.
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.
Sparse autoencoders (SAEs) have become central to mechanistic interpretability, decomposing transformer activations into monosemantic features. Yet existing analyses characterise features almost exclusively through top-activating token lists or decoder weight vectors, leaving the higher-order co-occurrence structure shared across features largely unexamined. We introduce a graph-structured representation in which each SAE feature is modelled as a token co-occurrence graph: nodes are the tokens most frequent near strong activations, and edges connect pairs that co-occur within local context windows. A custom WL-style, frequency-binned graph kernel then provides a similarity measure over this structural space. Applied as a proof of concept to features from a large SAE trained on GPT-2 Small and probed with a synthetic mixed-domain corpus, our clustering recovers heuristic motif families (punctuation-heavy patterns, language and script clusters, and code-like templates) that are not recovered by clustering on decoder cosine similarity. A token-histogram baseline achieves higher overall purity, so the contribution of the graph view is complementary rather than dominant: it surfaces structural relationships that token-frequency and decoder-weight views alone do not capture. Cluster assignments are stable across graph-construction hyperparameters and random seeds.
Ruben Fernandez-Boullon, Pablo Magariños-Docampo, Javier Perez-Robles
Sparse autoencoders (SAEs) expose interpretable features in large language model activations, yet existing structured SAEs impose single-parent trees or forests, while post-hoc graphs permit multiple parents but neither guide feature learning nor ensure reliable relation recovery. We introduce the Adaptive Graph Sparse Autoencoder (AG-SAE), a structure-guided training paradigm that treats each feature's complete parent set as an atomic structural hypothesis and lets evidence select zero, one, or multiple parents. By competing complete parent sets against null, subset, and alternative explanations, AG-SAE identifies jointly necessary multi-parent relations while rejecting redundant or spurious alternatives and verifying that each child contributes beyond its parents. The induced topology over SAE features then defines a differentiable structural loss that guides SAE training, while topology-guided refinement mitigates feature absorption and uses persistent reconstruction gaps exposed by the learned structure to initialize new features. The entire graph is then induced again from the revised dictionary by reassessing every feature's complete parent set, closing the dictionary-graph self-consistency cycle. Experiments demonstrate exact mixed-topology recovery in a controlled toy model, greater relational reliability and semantic validity than structured and post-hoc baselines on real LLM activations, and stronger feature-level causal interventions than conventional SAE features. AG-SAE thereby turns recovered mixed-topology feature structure into an unsupervised training signal that improves the dictionary, enables reliable feature organization beyond the topological limitations of trees, and exhibits stronger causal control beyond reconstruction.
Xiaozuo Shen, Yifei Cai, Tian Tan +3
University of Arizona · Iowa State University · Old Dominion University
Graph autoencoders (GAEs) are widely used for learning representations of dynamic graphs. However, their optimisation objectives typically do not take structural heterogeneity across nodes into account. We propose three distance-based GAE variants that incorporate structural penalties into the reconstruction loss. All variants share a two-layer Graph Convolutional Network encoder and a Euclidean-distance decoder trained with distance-based reconstruction objectives. We extend sparsity-corrected loss with two node-level regularization terms: (i) a hub penalty based on degree centrality, and (ii) a penalty based on Natural Community Local Intrinsic Dimensionality (NC-LID). The paper is motivated by prior evidence linking high NC-LID to reduced embedding quality. The proposed methods are designed to emphasize reconstruction errors for structurally ambiguous nodes. Experiments on multiple dynamic graph data sets show that incorporating NC-LID-based regularization consistently improves reconstruction performance over the baseline without structural regularization and the method using hub-aware regularization. These findings highlight NC-LID as a useful structural signal for enhancing distance-based graph autoencoders in dynamic settings.
Aleksandar Tomčić, Miloš Savić, Miloš Radovanović
Department of Mathematics and Informatics, Faculty of Sciences, University of Novi Sad, Serbia Trg Dositeja Obradovi´ca 4, 21000 Novi Sad, Serbia