Neural Representation Geometry

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 125% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 181

All topics

The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws
May 11, 2026Eslam Zaher, Maciej Trzaskowski, Quan Nguyen +1Neural Representation Geometry Sparse Autoencoders
ARC Training Centre for Information Resilience (CIRES) · School of Mathematics and Physics, University of Queensland · Institute for Molecular Bioscience, University of Queensland +2
Flag Varieties: A Geometric Framework for Deep Network Alignment
May 11, 2026Jingchuan Xiao, Xinyi Sui, Cihan RuanNeural Representation Geometry Neural Collapse
Department of Mathematics and Computer Studies, Mary Immaculate College, Ireland · Department of Computer Science and Engineering, Santa Clara University, USA
SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations
May 10, 2026Mario Edoardo Pandolfo, Enrico Grimaldi, Lorenzo Marinucci +4Neural Representation Geometry Visual Representation Learning
Dept. Computer, Control and Management Engineering, Sapienza University of Rome, Rome, Italy. · National Inter-University Consortium for Telecommunications (CNIT), Parma, Italy. · Dept. of Statistical Sciences, Sapienza University of Rome, Rome, Italy. +1
HyNeuralMap: Hyperbolic Mapping of Visual Semantics to Neural Hierarchies
May 10, 2026Zihan Ma, Tian Xia, Kexin Wang +3Neural Representation Geometry Cross-Modal Alignment
School of Electronic Information (School of Artificial Intelligence), the Xi’an Key Laboratory of Radiomics and Intelligent Perception, Northwest University, Xi’an 710127, China
The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?
May 10, 2026Zhaoyang Zhang, Run Shao, Dongyue Wu +4Neural Representation Geometry Cross-Modal Alignment
1Central South University · 4Ant Group · 2Huazhong University of Science and Technology +1
Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbations
May 9, 2026Gabriele Lombardo, Luigi Maiorana, Liliana Lo Presti +1Neural Representation Geometry VLM Interpretability
University of Palermo Palermo, Italy
A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases
May 9, 2026Gianfranco Lombardo, Giuseppe Trimigno, Stefano CagnoniNeural Representation Geometry LLM Interpretability
Department of Engineering and Architecture University of Parma Parma, IT 43125
When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search
May 9, 2026John T. Robertson, Jianing Zhu, Haris Vikalo +1Neural Representation Geometry Activation Steering
The University of Texas at Austin
Bilinear autoencoders find interpretable manifolds
May 9, 2026Thomas Dooms, Ward Gauderis, Geraint Wiggins +1Neural Representation Geometry Autoencoders
University of Antwerp · Vrije Universiteit Brussel · University of Antwerp, sqIRL
The Propagation Field: A Geometric Substrate Theory of Deep Learning
May 8, 2026Xingrui GuNeural Network Generalization Neural Representation Geometry
1EECS, University of California, Berkeley, Berkeley, CA.
The Geometric Structure of Models Learning Sparse Data
May 8, 2026Thomas Walker, T. Mitchell Roddenberry, Ahmed Imtiaz Humayun +2Neural Representation Geometry Statistical Learning Theory
Rice University · Rice University† · Now at Google Research +1
Is Class Signal Clustered or Routed in Task-Induced Implicit Neural Representation Weight Spaces?
May 8, 2026Xinyi Guo, Mingyi He, Haobin Ding +7Neural Representation Geometry Implicit Neural Representations
South China Normal University · Beijing University of Chemical Technology · Tsinghua University +1
Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative
May 8, 2026Sibayan Mitra, Dhruv KumarNeural Representation Geometry Representation Geometry in Language Models
BITS Pilani · Birla Institute of Technology and Science, Pilani, India.
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
May 8, 2026Feilong LiuTransformer Interpretability Neural Representation Geometry
A Generalized Singular Value Theory for Neural Networks
May 7, 2026Brian Charles Brown, Robert Bridges, David Grimsman +2Neural Representation Geometry Geometric Representation Learning
AI Sweden Gothenburg, Sweden
Towards Metric-Faithful Neural Graph Matching
May 7, 2026Jyotirmaya Shivottam, Subhankar MishraGraph Neural Networks Neural Representation Geometry
1National Institute of Science Education and Research (NISER), Bhubaneswar, India · 2Homi Bhabha National Institute, Mumbai, India
Invariant Features in Language Models: Geometric Characterization and Model Attribution
May 7, 2026Agnibh Dasgupta, Abdullah Tanvir, Xin ZhongNeural Representation Geometry Invariant Representation Learning
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA
Topological Signatures of Grokking
May 7, 2026Yifan Tang, Qiquan Wang, Inés García-Redondo +1Neural Network Generalization Neural Representation Geometry
Imperial College London · Queen Mary University of London · University of Fribourg
Normalized Architectures are Natively 4-Bit
May 7, 2026Maxim Fishman, Brian Chmiel, Ron Banner +2LLM Quantization Neural Representation Geometry
∧NVIDIA · ◦Department of Electrical and Computer Engineering - Technion, Haifa, Israel
Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
May 6, 2026Daniel Wurgaft, Can Rager, Matthew Kowal +13Neural Representation Geometry Manifold Learning
Stanford University · University College London · Northeastern University +2
Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks
May 6, 2026Xuan Qi, Yi Wei, Fanqi Yu +3Neural Representation Geometry
AI for Good Istituto Italiano di Tecnologia, Genoa, Italy Department of Computer Science University of Verona, Verona, Italy
A geometric relation of the error introduced by sampling a language model's output distribution to its internal state
May 6, 2026Albert F. ModenbachNeural Representation Geometry LLM Interpretability
Department of Mathematics, King’s College London, United Kingdom.
Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
May 5, 2026Hao Yan, Haolin Yang, Yiqiao ZhongTransformer Interpretability Neural Representation Geometry
University of Wisconsin–Madison · University of Chicago
Beyond Activation Alignment: The Geometry of Neural Sensitivity
May 4, 2026Amirhossein Yavari, Farnaz Zamani EsfahlaniNeural Representation Geometry Neural Decoding
Stephenson School of Biomedical Engineering, University of Oklahoma · Data Science and Analytics Institute, University of Oklahoma
Geometric and Spectral Alignment for Deep Neural Network II
May 4, 2026Ziran Liu, Wei Wang, Jinhao Wang +5Neural Representation Geometry
Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai 200433, China · Futurewei Technologies, Inc., San Jose, CA 95131 · Dept. of Computer Science and Engineering, Santa Clara University, Santa Clara, CA 95050 +1
Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations
May 2, 2026Pratyush Acharya, Nuraj Rimal, Habish DhakalTransformer Interpretability Representation Learning
Spectral Graph Sparsification Preserves Representation Geometry in Graph Neural Networks
May 1, 2026Sanjukta KrishnagopalGraph Neural Networks Neural Representation Geometry
Department of Computer Science, University of California Santa Barbara
Diffusion Operator Geometry of Feedforward Representations
May 1, 2026Kanishka ReddyNeural Representation Geometry Feedforward Neural Networks
Department of Applied Mathematics University of Washington Seattle, WA
Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probing
May 1, 2026Leonid BedratyukRepresentation Learning Neural Representation Geometry
Aitchison Embeddings for Learning Compositional Graph Representations
May 1, 2026Nikolaos Nakis, Chrysoula Kosma, Panagiotis Promponas +2Neural Representation Geometry Graph Representation Learning
Human Nature Lab, Yale University, New Haven, USA · Universit´e Paris Saclay, Universit´e Paris Cit´e, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli, Gif-sur-Yvette, France · Yale Institute for Network Science, Yale University, New Haven, USA +2
Geometric and dynamical analysis of attractor boundaries and storage limits in kernel Hopfield networks
May 1, 2026Akira TamamoriDynamical Systems Neural Representation Geometry
Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan
Do Sparse Autoencoders Capture Concept Manifolds?
Apr 30, 2026Usha Bhalla, Thomas Fel, Can Rager +9Neural Representation Geometry Sparse Autoencoders
aHarvard University · bNortheastern University · cTechnion IIT +1
Semantic Structure of Feature Space in Large Language Models
Apr 29, 2026Austin C. Kozlowski, Andrei BoutylineNeural Representation Geometry LLM Interpretability
Knowledge Lab · University of Chicago · Department of Sociology +1
Monitoring Neural Training with Topology: A Footprint-Predictable Collapse Index
Apr 28, 2026Alexander KalinowskiRepresentation Collapse Neural Representation Geometry
SUNY Empire
Geometric Analysis of Self-Supervised Vision Representations for Semantic Image Retrieval
Apr 27, 2026Esteban Rodríguez-Betancourt, Edgar Casasola-MurilloNeural Representation Geometry Self-Supervised Visual Representation Learning
Posgrado en Computación e Informática Universidad de Costa Rica · Escuela de Ciencias de la Computación Universidad de Costa Rica
Representational Curvature Modulates Behavioral Uncertainty in Large Language Models
Apr 27, 2026Jack King, Evelina Fedorenko, Eghbal A. HosseiniNeural Representation Geometry Autoregressive Language Modeling
Brain and Cognitive Sciences Massachusetts Institute of Technology Cambridge, MA 02139 · Google DeepMind
Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory
Apr 22, 2026Akira TamamoriNeural Representation Geometry Hopfield Networks
Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan
Rethinking Intrinsic Dimension Estimation in Neural Representations
Apr 22, 2026Rickmer Schulte, David RügamerIntrinsic Dimensionality Neural Representation Geometry
LMU Munich, MCML
ILDR: Geometric Early Detection of Grokking
Apr 22, 2026Shreel GolwalaNeural Network Generalization Early Stopping
Virginia Tech
Geometric Decoupling: Diagnosing the Structural Instability of Latent
Apr 20, 2026Yuanbang Liang, Zhengwen Chen, Yu-Kun LaiNeural Representation Geometry Latent Diffusion Models
School of Computer Science and Informatics, Cardiff University, Cardiff, United Kingdom
Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling
Apr 20, 2026Yidi YuanNeural Representation Geometry Language Model Post-Training
Home Team Science and Technology Agency
The Umwelt Representation Hypothesis: Rethinking Universality
Apr 20, 2026Victoria Bosch, Rowan Sommers, Adrien Doerig +1Neural Representation Geometry
Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany. · Department of Psychology and Education, Freie Universität Berlin, Berlin, Germany. · Bernstein Center for Computational Neuroscience, Berlin, Germany
ATLAS: Constitution-Conditioned Latent Geometry and Redistribution Across Language Models and Neural Perturbation Data
Apr 19, 2026Gareth Seneque, Lap-Hang Ho, Nafise Erfanian Saeedi +2LLM Alignment Neural Representation Geometry
Australian Broadcasting Corporation
A Wasserstein Geometric Framework for Hebbian Plasticity
Apr 17, 2026Ulrich TanNeural Representation Geometry Hebbian Learning
Gating Enables Curvature: A Geometric Expressivity Gap in Attention
Apr 16, 2026Satwik Bathula, Anand A. JoshiInformation Geometry Neural Representation Geometry
Department of Electrical and Computer Engineering University of Southern California
A Complete Symmetry Classification of Shallow ReLU Networks
Apr 15, 2026Pranavkrishnan RamakrishnanShallow Neural Networks Neural Representation Geometry
Neural parametric representations for thin-shell shape optimisation
Apr 8, 2026Xiao Xiao, Fehmi CirakNeural Representation Geometry
School of Ocean and Civil Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China. · Department of Engineering, University of Cambridge, Cambridge, CB2 1PZ, United Kingdom.
Understanding Latent Diffusability via Fisher Geometry
Apr 3, 2026Jing Gu, Morteza Mardani, Wonjun Lee +2Information Geometry Neural Representation Geometry
School of Mathematics, University of Minnesota · NVIDIA · Department of Mathematics, The Ohio State University +1
Similarity of Neural Network Representations in Superposition
Mar 31, 2026Sunny Liu, Habon Issa, André Longon +4Feature Superposition Representational Similarity Analysis
Cold Spring Harbor Laboratory · UC San Diego · Anthropic +2
Differential privacy representation geometry for medical image analysis
Mar 1, 2026Soroosh Tayebi Arasteh, Marziyeh Mohammadi, Sven Nebelung +1Neural Representation Geometry Healthcare
Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany · INDA Institute, RWTH Aachen University, Aachen, Germany
Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models
Feb 27, 2026Arnas Uselis, Andrea Dittadi, Seong Joon OhNeural Representation Geometry Visual Representation Learning
Tübingen AI Center, University of Tübingen · Helmholtz · Technical University of Munich
Disentangling Geometry, Performance, and Training in Language Models
Feb 24, 2026Atharva Kulkarni, Jacob Mitchell Springer, Arjun Subramonian +1LLM Evaluation Neural Representation Geometry
University of Southern California · 2Carnegie Mellon Univer · University of California, Los Angeles.
Symmetry in language statistics shapes the geometry of model representations
Feb 16, 2026Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2Representation Learning Neural Representation Geometry
1UC Berkeley · 2EPFL · 3Johns Hopkins +1
Revisiting the Platonic Representation Hypothesis: An Aristotelian View
Feb 16, 2026Fabian Gröger, Shuo Wen, Maria BrbićRepresentational Similarity Analysis Neural Representation Geometry
1EPFL · University of Basel · 3HSLU
Multi-Way Representation Alignment
Feb 5, 2026Akshit Achara, Tatiana Gaintseva, Mateo Mahaut +5Neural Representation Geometry Latent Space Alignment
King’s College London, London, United Kingdom · Queen Mary University of London, London, United Kingdom · Universitat Pompeu Fabra, Barcelona, Spain +5
The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence
Jan 27, 2026Yichao Cai, Zhen Zhang, Yuhang Liu +1Cross-Modal Learning Contrastive Learning
Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre
Geometric Stability: The Missing Axis of Representations
Jan 14, 2026Prashant C. RajuRepresentational Similarity Analysis Neural Representation Geometry
Output Embedding Centering for Stable LLM Pretraining
Jan 5, 2026Felix Stollenwerk, Anna Lokrantz, Niclas HertzbergLanguage Model Pretraining Neural Representation Geometry
AI Sweden
Probing Spectrum-Like Organization of States of Mind in Transformer Representation Spaces
Dec 23, 2025Sophie ZhaoTransformer Interpretability Neural Representation Geometry
School of Computer Science Georgia Institute of Technology
TetraSDF: Analytic Isosurface Extraction with Multi-resolution Tetrahedral Grid
Nov 20, 2025Seonghun Oh, Youngjung Uh, Jin-Hwa Kim3D Mesh Generation Neural Representation Geometry
Yonsei University, Republic of Korea · NAVER AI Lab, Republic of Korea · Seoul National University, Republic of Korea

May 11, 2026, cs.LGJ/K move · Enter open · S save

The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws

Eslam Zaher, Maciej Trzaskowski, Quan Nguyen, Fred Roosta

ARC Training Centre for Information Resilience (CIRES) · School of Mathematics and Physics, University of Queensland · Institute for Molecular Bioscience, University of Queensland+2

Neural Representation Geometry Sparse Autoencoders

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) operationalise the linear representation hypothesis: they reconstruct model activations as sparse linear combinations of interpretable dictionary atoms, on the implicit assumption that activation space is well approximated by a globally linear structure. Their reconstruction error varies sharply across layers in ways that existing scaling laws, fitted at single layers, do not explain. We argue that this variation is the empirical trace of a geometric mismatch: where the activation manifold is curved and its intrinsic dimension varies across layers, no sparse linear dictionary can match it uniformly, and the SAE's width-sparsity scaling becomes a layer-dependent function of manifold structure rather than a single universal law. We conduct the first cross-layer SAE scaling study, fitting and regressing on 844 residual-stream Gemma Scope SAE checkpoints across 68 layers of Gemma 2 2B and 9B. Stage 1 fits a per-layer scaling-law surface; Stage 2 regresses the fitted parameters and the derived per-layer width exponents on four layerwise geometric summaries. We find that manifold geometry predicts the per-layer width exponent in both models, and that the same regression coefficients learnt on one model predict the other model's per-layer exponents under cross-model transfer, indicating a transferable geometric law. At the showcase layers where richer width grids permit identification of the asymptotic floor, we find that the fitted floor tracks the layerwise geometric ordering: higher curvature and intrinsic dimension correspond to higher floor, consistent with the irreducible second-order residual that any sparse linear approximation of a curved manifold must leave behind. SAEs thus encounter not a finite-resource ceiling but a geometry-dependent wall, set by the manifold they are trying to reconstruct.

Momentum

Related topics

Latest papers 181