Neural Representation Geometry

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 125% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 182

All topics

The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws
May 11, 2026Eslam Zaher, Maciej Trzaskowski, Quan Nguyen +1Neural Representation Geometry Sparse Autoencoders
ARC Training Centre for Information Resilience (CIRES) · School of Mathematics and Physics, University of Queensland · Institute for Molecular Bioscience, University of Queensland +2
Flag Varieties: A Geometric Framework for Deep Network Alignment
May 11, 2026Jingchuan Xiao, Xinyi Sui, Cihan RuanNeural Representation Geometry Neural Collapse
Department of Mathematics and Computer Studies, Mary Immaculate College, Ireland · Department of Computer Science and Engineering, Santa Clara University, USA
SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations
May 10, 2026Mario Edoardo Pandolfo, Enrico Grimaldi, Lorenzo Marinucci +4Neural Representation Geometry Visual Representation Learning
Dept. Computer, Control and Management Engineering, Sapienza University of Rome, Rome, Italy. · National Inter-University Consortium for Telecommunications (CNIT), Parma, Italy. · Dept. of Statistical Sciences, Sapienza University of Rome, Rome, Italy. +1
HyNeuralMap: Hyperbolic Mapping of Visual Semantics to Neural Hierarchies
May 10, 2026Zihan Ma, Tian Xia, Kexin Wang +3Neural Representation Geometry Cross-Modal Alignment
School of Electronic Information (School of Artificial Intelligence), the Xi’an Key Laboratory of Radiomics and Intelligent Perception, Northwest University, Xi’an 710127, China
The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?
May 10, 2026Zhaoyang Zhang, Run Shao, Dongyue Wu +4Neural Representation Geometry Cross-Modal Alignment
1Central South University · 4Ant Group · 2Huazhong University of Science and Technology +1
Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbations
May 9, 2026Gabriele Lombardo, Luigi Maiorana, Liliana Lo Presti +1Neural Representation Geometry VLM Interpretability
University of Palermo Palermo, Italy
A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases
May 9, 2026Gianfranco Lombardo, Giuseppe Trimigno, Stefano CagnoniNeural Representation Geometry LLM Interpretability
Department of Engineering and Architecture University of Parma Parma, IT 43125
When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search
May 9, 2026John T. Robertson, Jianing Zhu, Haris Vikalo +1Neural Representation Geometry Activation Steering
The University of Texas at Austin
Bilinear autoencoders find interpretable manifolds
May 9, 2026Thomas Dooms, Ward Gauderis, Geraint Wiggins +1Neural Representation Geometry Autoencoders
University of Antwerp · Vrije Universiteit Brussel · University of Antwerp, sqIRL
The Propagation Field: A Geometric Substrate Theory of Deep Learning
May 8, 2026Xingrui GuNeural Network Generalization Neural Representation Geometry
1EECS, University of California, Berkeley, Berkeley, CA.
The Geometric Structure of Models Learning Sparse Data
May 8, 2026Thomas Walker, T. Mitchell Roddenberry, Ahmed Imtiaz Humayun +2Neural Representation Geometry Statistical Learning Theory
Rice University · Rice University† · Now at Google Research +1
Is Class Signal Clustered or Routed in Task-Induced Implicit Neural Representation Weight Spaces?
May 8, 2026Xinyi Guo, Mingyi He, Haobin Ding +7Neural Representation Geometry Implicit Neural Representations
South China Normal University · Beijing University of Chemical Technology · Tsinghua University +1
Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative
May 8, 2026Sibayan Mitra, Dhruv KumarNeural Representation Geometry Representation Geometry in Language Models
BITS Pilani · Birla Institute of Technology and Science, Pilani, India.
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
May 8, 2026Feilong LiuTransformer Interpretability Neural Representation Geometry
A Generalized Singular Value Theory for Neural Networks
May 7, 2026Brian Charles Brown, Robert Bridges, David Grimsman +2Neural Representation Geometry Geometric Representation Learning
AI Sweden Gothenburg, Sweden
Towards Metric-Faithful Neural Graph Matching
May 7, 2026Jyotirmaya Shivottam, Subhankar MishraGraph Neural Networks Neural Representation Geometry
1National Institute of Science Education and Research (NISER), Bhubaneswar, India · 2Homi Bhabha National Institute, Mumbai, India
Invariant Features in Language Models: Geometric Characterization and Model Attribution
May 7, 2026Agnibh Dasgupta, Abdullah Tanvir, Xin ZhongNeural Representation Geometry Invariant Representation Learning
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA
Topological Signatures of Grokking
May 7, 2026Yifan Tang, Qiquan Wang, Inés García-Redondo +1Neural Network Generalization Neural Representation Geometry
Imperial College London · Queen Mary University of London · University of Fribourg
Normalized Architectures are Natively 4-Bit
May 7, 2026Maxim Fishman, Brian Chmiel, Ron Banner +2LLM Quantization Neural Representation Geometry
∧NVIDIA · ◦Department of Electrical and Computer Engineering - Technion, Haifa, Israel
Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
May 6, 2026Daniel Wurgaft, Can Rager, Matthew Kowal +13Neural Representation Geometry Manifold Learning
Stanford University · University College London · Northeastern University +2
Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks
May 6, 2026Xuan Qi, Yi Wei, Fanqi Yu +3Neural Representation Geometry
AI for Good Istituto Italiano di Tecnologia, Genoa, Italy Department of Computer Science University of Verona, Verona, Italy
A geometric relation of the error introduced by sampling a language model's output distribution to its internal state
May 6, 2026Albert F. ModenbachNeural Representation Geometry LLM Interpretability
Department of Mathematics, King’s College London, United Kingdom.
Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
May 5, 2026Hao Yan, Haolin Yang, Yiqiao ZhongTransformer Interpretability Neural Representation Geometry
University of Wisconsin–Madison · University of Chicago
Beyond Activation Alignment: The Geometry of Neural Sensitivity
May 4, 2026Amirhossein Yavari, Farnaz Zamani EsfahlaniNeural Representation Geometry Neural Decoding
Stephenson School of Biomedical Engineering, University of Oklahoma · Data Science and Analytics Institute, University of Oklahoma
Geometric and Spectral Alignment for Deep Neural Network II
May 4, 2026Ziran Liu, Wei Wang, Jinhao Wang +5Neural Representation Geometry
Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai 200433, China · Futurewei Technologies, Inc., San Jose, CA 95131 · Dept. of Computer Science and Engineering, Santa Clara University, Santa Clara, CA 95050 +1
Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations
May 2, 2026Pratyush Acharya, Nuraj Rimal, Habish DhakalTransformer Interpretability Representation Learning
Spectral Graph Sparsification Preserves Representation Geometry in Graph Neural Networks
May 1, 2026Sanjukta KrishnagopalGraph Neural Networks Neural Representation Geometry
Department of Computer Science, University of California Santa Barbara
Diffusion Operator Geometry of Feedforward Representations
May 1, 2026Kanishka ReddyNeural Representation Geometry Feedforward Neural Networks
Department of Applied Mathematics University of Washington Seattle, WA
Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probing
May 1, 2026Leonid BedratyukRepresentation Learning Neural Representation Geometry
Aitchison Embeddings for Learning Compositional Graph Representations
May 1, 2026Nikolaos Nakis, Chrysoula Kosma, Panagiotis Promponas +2Neural Representation Geometry Graph Representation Learning
Human Nature Lab, Yale University, New Haven, USA · Universit´e Paris Saclay, Universit´e Paris Cit´e, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli, Gif-sur-Yvette, France · Yale Institute for Network Science, Yale University, New Haven, USA +2

May 11, 2026, cs.LGJ/K move · Enter open · S save

The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws

Eslam Zaher, Maciej Trzaskowski, Quan Nguyen, Fred Roosta

ARC Training Centre for Information Resilience (CIRES) · School of Mathematics and Physics, University of Queensland · Institute for Molecular Bioscience, University of Queensland+2

Neural Representation Geometry Sparse Autoencoders

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) operationalise the linear representation hypothesis: they reconstruct model activations as sparse linear combinations of interpretable dictionary atoms, on the implicit assumption that activation space is well approximated by a globally linear structure. Their reconstruction error varies sharply across layers in ways that existing scaling laws, fitted at single layers, do not explain. We argue that this variation is the empirical trace of a geometric mismatch: where the activation manifold is curved and its intrinsic dimension varies across layers, no sparse linear dictionary can match it uniformly, and the SAE's width-sparsity scaling becomes a layer-dependent function of manifold structure rather than a single universal law. We conduct the first cross-layer SAE scaling study, fitting and regressing on 844 residual-stream Gemma Scope SAE checkpoints across 68 layers of Gemma 2 2B and 9B. Stage 1 fits a per-layer scaling-law surface; Stage 2 regresses the fitted parameters and the derived per-layer width exponents on four layerwise geometric summaries. We find that manifold geometry predicts the per-layer width exponent in both models, and that the same regression coefficients learnt on one model predict the other model's per-layer exponents under cross-model transfer, indicating a transferable geometric law. At the showcase layers where richer width grids permit identification of the asymptotic floor, we find that the fitted floor tracks the layerwise geometric ordering: higher curvature and intrinsic dimension correspond to higher floor, consistent with the irreducible second-order residual that any sparse linear approximation of a curved manifold must leave behind. SAEs thus encounter not a finite-resource ceiling but a geometry-dependent wall, set by the manifold they are trying to reconstruct.

Momentum

Related topics

Latest papers 182