Representation Geometry in Language Models

Follow this topic Ask about this topic

This field studies the shapes and relationships formed by token embeddings and hidden activations, and how those structures change across layers, contexts, and training. Researchers use linear probes, representational similarity analysis, spectral and manifold methods, sparse autoencoders, and causal interventions such as activation patching and steering. These tools are applied to questions about reasoning, multilingual transfer, retrieval, safety, and reliability, using settings that range from controlled synthetic tasks to benchmarks such as GSM8K and MTEB. A central open tension is that information being decodable or geometrically similar does not guarantee that a model uses it causally; useful structure may also be local, nonlinear, or sensitive to measurement choices.

Latest papers

417 papers78 papers in the last four weeks, up 34% on the four weeks before. 0.5% of all new papers.

Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness
Jul 2, 2026Aria Masoomi, Mahsa Bazzaz, Adel Javanmard +1LLM Evaluation Representation Geometry in Language Models
1Northeastern University · University of Southern California · 3Google Research
Ember: A Kilobyte-State Optimizer for Token Interfaces
Jul 1, 2026Kathan ShahLanguage Model Pretraining LLM Fine-Tuning
Preprint.
A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
Jul 1, 2026Siyi Wang, James Bailey, Ting DangLanguage Model Steering Conditional Flow Matching
The University of Melbourne, Australia · Monash University, Australia
NeuroCogMap Reveals Cognitive Organization of Large Language Models
Jul 1, 2026Zhongxiang Sun, Haolang Lu, Qiang Ma +11Cognitive Modeling LLM Interpretability
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China · The University of Hong Kong, Hong Kong, China +5
DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning
Jul 1, 2026Hengyu Fu, Tianyu Guo, Zixuan Wang +5Recurrent Transformers Multi-Hop Reasoning
University of California, Berkeley
SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models
Jun 30, 2026Jian Gu, Aldeida Aleti, Chunyang Chen +1Mechanistic Interpretability Representation Geometry in Language Models
Monash University · Technical University of Munich · Chongqing University
Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts
Jun 29, 2026Huanping Xiao, Yingji LiGender Bias in Language Models Representation Disentanglement
College of Foreign Languages and Cultures, Jilin University, Changchun, Jilin Province, China · College of Computer Science and Technology, Jilin University, Changchun, Jilin Province, China
Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings
Jun 28, 2026V. S. Raghu ParupudiSemantic Textual Similarity Neural Representation Geometry
University of California, San Diego
Invariant Reasoning Directions in Latent Trajectories of Language Models
Jun 28, 2026Arun Vignesh Malarkkan, Manan Roy Choudhury, Utkarsh Byahut +3Reasoning Consistency in Language Models Implicit Reasoning in Language Models
Arizona State University, Tempe, Arizona.
Search for Truth from Reasoning: A Dynamic Representation Editing Framework for Steering LLM Trajectories
Jun 26, 2026Tianlong Wang, Yuhang Wang, Weibin Liao +5Language Model Steering Representation Geometry in Language Models
National Engineering Research Center of Software Engineering, Peking University · School of Computer Science, Peking University · Key Laboratory of High Confidence Software Technologies, Ministry of Education +1
The Geometry of Updates: Fisher Alignment at Vocabulary Scale
Jun 25, 2026John SweeneyTask Transferability Estimation Transfer Learning
Sideplane AI
Where Do Models Find Happiness? Emotion Vectors in Open-Source LLMs
Jun 25, 2026Sinie van der Ben, Raphaël Baur, Yannick Metz +1LLM Interpretability Valence-Arousal Modeling
Department of Computer Science, ETH, Zurich, Switzerland.
Structure Before Collapse: Transient semantic geometry in next-token prediction
Jun 25, 2026Yize Zhao, Isabel Papadimitriou, Christos ThrampoulidisAutoregressive Language Modeling Next-Token Prediction
Department of Electrical and Computer Engineering, The University of British Columbia · Department of Linguistics, The University of British Columbia
Reasoning as Attractor Dynamics: Latent Memory Retrieval via Gibbs-Weighted Energy Minimization
Jun 23, 2026Kanishk AwadhiyaLLM Inference Representation Geometry in Language Models
Independent Researcher
Evidence for feature-specific error correction in LLMs
Jun 23, 2026Francisco Ferreira da Silva, Stefan HeimersheimLLM Interpretability Language Model Robustness
Pivotal Research.
Perfect Detection, Failed Control: The Geometry of Knowing vs. Steering in Language Models
Jun 23, 2026Cosimo Galeone, Anna Ettorre, Minsu Park +2Mechanistic Interpretability Causal Interventions in Language Models
Alomana, Grottaglie, Italy
Do LLM Embedding Spaces Recover Expert Structure?
Jun 22, 2026Yixuan Zhu, Zhenke Duan, Fanghen LiLLM Fine-Tuning Mental Health
Zhongnan University of Economics and Law
Abstract representational geometry supports inference in large language models
Jun 22, 2026Yunan Zeng, Yuwang WangLLM Inference LLM Interpretability
College of Future Information Technology, Fudan University, Shanghai, 200433, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China
Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs
Jun 21, 2026Wenhao Yu, Zhicong Lu, Bo Lv +4Knowledge Editing Model Editing
School of Computer Science and Technology, Tianjin University · Kexin Technology · University of Chinese Academy of Sciences +2
ROMEVA: Geometry-Preserving Vocabulary Expansion for Roman Urdu Language Models
Jun 21, 2026Mahnoor Khan, Afsheen Asif, Milhan Afzal Khan +2Multilingual Language Models Word Embeddings
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan · Department of Computer Science, University of Agriculture Faisalabad, Faisalabad, Pakistan
From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models
Jun 18, 2026Jiaxu Zuo, Mu You, Kaixin Lan +5LLM Interpretability Automated Essay Scoring
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Institute of International Language Services Studies, Macau Millennium College · School of Data Science and Information Engineering, Guizhou Minzu University
What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis
Jun 18, 2026Xinghao Chen, Chak Tou Leong, Wenjin Guo +3Continuous Latent Reasoning Process Supervision
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China.
GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs
Jun 18, 2026Yu DengCausal Interventions in Language Models Representation Geometry in Language Models
Code-Switching Reveals Language Anchoring in Multilingual LLMs
Jun 18, 2026Jeonghyun Park, Seunghyun Yoon, Yonghyun Jun +1Multilingual Language Models Code-Switching
Chung-Ang University, Seoul, Korea · Adobe Research, USA
Tracking Representation Dynamics in Large Language Models with Persistent Homology
Jun 17, 2026Naman Malhotra, Jay Ambadkar, Abhinav Gupta +4LLM Alignment LLM Fine-Tuning
Department of Mathematics, Imperial College London, UK · Department of Computing, Imperial College London, UK
RepSelect: Robust LLM Unlearning via Representation Selectivity
Jun 15, 2026Filip Sondej, Yushi Yang, Adam MahdiLLM Safety Representation Geometry in Language Models
Independent · University of Oxford
The Value Axis: Language Models Encode Whether They're on the Right Track
Jun 15, 2026Nick Jiang, Isaac Kauvar, Jack LindseyConfidence Estimation in Language Models Language Model Calibration
1Stanford University · 2Anthropic
Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures
Jun 15, 2026Xueping GaoKernel Methods Representation Geometry in Language Models
Alibaba Cloud
Data-Driven Decoding of Russell's Circumplex Model of Affect
Jun 15, 2026Amdjed Belaref, Samir Sadok, Zineb Noumir +1Multimodal Embedding Representation Geometry in Language Models
Alten, France · CentraleSup´elec IETR UMR CNRS 6164, France
High-Dimensional Random Projection for Activation Steering in Language Models
Jun 13, 2026Minh-Hieu Pham, Bach Do, Laziz Abdullaev +2Language Model Steering Representation Geometry in Language Models
Hanoi University of Science and Technology

Jul 2, 2026J/K move · Enter open · S save

Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness

Aria Masoomi, Mahsa Bazzaz, Adel Javanmard, Vahab Mirrokni

1Northeastern University · University of Southern California · 3Google Research

LLM Evaluation Representation Geometry in Language Models LLM Mathematical Reasoning LLM Reasoning Reasoning Trace Analysis

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning enables large language models (LLMs) to solve complex problems by generating intermediate reasoning steps. While much attention has been paid to the length and content of these reasoning chains, far less is known about their internal geometry. We study the geometry of CoT trajectories in the hidden state space of transformer models, formalizing each reasoning chain as a discrete curve in

\mathbb{R}^d

and characterizing it through spectral, positional, and kinematic geometric functionals. We introduce the effective dimension

d_ρ

as a measure of trajectory complexity and show theoretically that trajectories with flatter eigenvalue spectra correspond to harder tasks, as they explore more of the hidden dimensions. Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial and current hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete, and may inform future early-stopping strategies. Experimentally, on mathematical reasoning problems from the MATH500 dataset,

d_ρ

achieves

0.93

AUC in distinguishing easy from hard problems, while kinematic features potentially can predict correctness from only the first

20\%

of generated tokens. These correctness signatures transfer across questions of varying difficulty, establishing that the shape of a model's internal reasoning trajectory is a principled window into both task hardness and solution quality.