Representation Geometry in Language Models

Overview

This field studies the shapes and relationships formed by token embeddings and hidden activations, and how those structures change across layers, contexts, and training. Researchers use linear probes, representational similarity analysis, spectral and manifold methods, sparse autoencoders, and causal interventions such as activation patching and steering. These tools are applied to questions about reasoning, multilingual transfer, retrieval, safety, and reliability, using settings that range from controlled synthetic tasks to benchmarks such as GSM8K and MTEB. A central open tension is that information being decodable or geometrically similar does not guarantee that a model uses it causally; useful structure may also be local, nonlinear, or sensitive to measurement choices.

Latest papers

CardsList

446 papers35 papers in the last four weeks, down 58% on the four weeks before. 0.3% of all new papers.

  1. Structure--Residual Inductive Biases for Language Model Pretraining

    Nov 17, 2026Hai Huang, Bartosz Żuk, Rassul Magauin +2Language Model PretrainingAutoregressive Language Modeling

  2. Can AI Mature a Model's Geometry? Training-Free Weight Baking in LLMs

    Nov 16, 2026Amin Golnari, Saeid SaneiLanguage Model EditingRepresentation Geometry in Language Models

  3. A phase diagram for training and inference in language models

    Nov 15, 2026Zhuo-Yang Song, Zeyu Li, Qing-Hong Cao +2Representation Geometry in Language ModelsTransformer Training Dynamics

  4. A Bag of Concepts? Testing Role-Filler Binding in the Verbalizable Workspace

    Nov 14, 2026Reya Dawlah, Elizabeth McMillin, Michael Shihong Zhang +4LLM InterpretabilityRepresentation Geometry in Language Models

  5. The Geometry of Conceptors in Neural-Activation Spaces

    Nov 14, 2026Ilias Triantafyllopoulos, Young Min Cho, Ren Tao +8Representation Geometry in Language ModelsConcept Localization in Language Models

  6. Geometrically Disentangling Concept Learning from the Language Modeling Loss

    Nov 13, 2026Yupei Wang, Neil Mallinar, Mikhail Belkin +1LLM InterpretabilityConcept Learning

  7. Dancing Tokens on the Vocabulary Network from LLM Generation

    Nov 13, 2026Yuan-Chao HuLLM InterpretabilityRepresentation Geometry in Language Models

  8. Where Does Alignment Emerge and What Does It Mean? A Closer Look at the Platonic Representation Hypothesis

    Nov 9, 2026Julia Herbst, Simon Roschmann, Quentin Bouniot +1Platonic Representation HypothesisCross-Modal Alignment

  9. Improving Confidence Estimation in Language Models with Linear Representations of Ambiguity

    Nov 9, 2026Themistoklis Nikas, Nishant Subramani, Aaron MuellerConfidence Estimation in Language ModelsLLM Uncertainty Estimation

  10. Spectral Rank and Step-Direction Dynamics in Language-Model Post-Training

    Nov 9, 2026Khurram Khalil, Khaza Anuarul HoqueTransformer Training DynamicsRL Post-Training

  11. Linear Probes to Bayesian Manifolds: Geometry of Arithmetic in Large Language Models

    Nov 8, 2026Anshul Kumar, Neha Prasad, Manoj Kumar +2LLM InterpretabilityRepresentation Geometry in Language Models

  12. On the Geometric Bottlenecks of Cross-Lingual Knowledge Editing

    Nov 7, 2026Aastha A K Verma, Anwoy Chatterjee, Sparsh Jain +1Language Model EditingRepresentation Geometry in Language Models

  13. Persona Perspectives: Adopting Viewpoints Reorganizes Representations in LLMs

    Nov 4, 2026Austin C. Kozlowski, Avital Mintz, Madison LeeLLM InterpretabilityRepresentation Geometry in Language Models

  14. Editing Large Language Models with Geometry-Aware Regularization

    Nov 4, 2026Bingqing Liu, Wei Liu, Jun Wang +3Sequential Knowledge EditingKnowledge Editing

  15. Sweep Less, Steer More: Using Activation Steering Geometry for Reduced Hyperparameter Search

    Nov 1, 2026Nishkal Hundia, Swastik Agrawal, Navita Goyal +1Language Model SteeringActivation Steering

  16. TopoSteer: Topological Layer Selection for Activation Steering in LLMs

    Nov 1, 2026Tuc Nguyen, Marcos Gutiérrez-del-Pozo, Eduardo Paluzo-Hidalgo +3Language Model SteeringActivation Steering

  17. Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions

    Oct 30, 2026Ning E, Changde Du, Yizhuo Lu +1LLM AlignmentMultimodal Large Language Models

  18. A Low-Rank Space Captures Linear Alignment Between LLM and Human Neural Activity During Naturalistic Language Listening

    Oct 29, 2026Zach Paris, Emily FinnNeural EncodingRepresentation Geometry in Language Models

  19. Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization

    Oct 28, 2026Frederick Riemenschneider, Anette FrankCross-Lingual Representation AlignmentMultilingual Language Models

  20. Five Architectures, One Shared Space Cross-Model Representational Alignment Without Weight Merging

    Oct 24, 2026Oladimeji Anthonio, Dimeji Olawuyi, Oloruntoba Ajayi +1Latent Space AlignmentRepresentation Geometry in Language Models

  21. Generalization properties of the token perceptron: the role of embedding and vocabulary

    Oct 23, 2026Dario Bocchi, Matteo Negri, Flavio Nicoletti +1Language Model GeneralizationRepresentation Geometry in Language Models

  22. A Mathematical Theory of the Jacobian Lens

    Oct 22, 2026Mitchell Ostrow, Daoyuan Qian, Toni Liu +1LLM InterpretabilityRepresentation Geometry in Language Models