Representation Geometry in Language Models

Follow this topic Ask about this topic

Momentum

48 papers in the last four weeks, up 118% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 221

All topics

Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
May 1, 2026Sheridan Feucht, Tal Haklay, Usha Bhalla +9Numerical Reasoning in Language Models LLM Interpretability
aNortheastern University · bTechnion IIT · cHarvard University +1
Escaping Mode Collapse in LLM Generation via Geometric Regulation
May 1, 2026Xin Du, Kumiko Tanaka-IshiiLanguage Model Steering Decoder-Only Language Models
Department of Communications and Computer Engineering, Waseda University, Tokyo, Japan · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China · Department of Computer Science and Engineering, Waseda University, Tokyo, Japan
Representational Curvature Modulates Behavioral Uncertainty in Large Language Models
Apr 27, 2026Jack King, Evelina Fedorenko, Eghbal A. HosseiniNeural Representation Geometry Autoregressive Language Modeling
Brain and Cognitive Sciences Massachusetts Institute of Technology Cambridge, MA 02139 · Google DeepMind
Convergent Evolution: How Different Language Models Learn Similar Number Representations
Apr 22, 2026Deqing Fu, Tianyi Zhou, Mikhail Belkin +2Fourier Feature Embeddings Frequency-Domain Feature Learning
University of Southern California · UC San Diego
Cell-Based Representation of Relational Binding in Language Models
Apr 21, 2026Qin Dai, Benjamin Heinzerling, Kentaro InuiLLM Interpretability Representation Probing
1Tohoku University · 2RIKEN AIP · 3MBZUAI
Exploring Concreteness Through a Figurative Lens
Apr 20, 2026Saptarshi Ghosh, Tianyu JiangFigurative Language Understanding LLM Interpretability
University of Cincinnati
Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
Apr 16, 2026Aimen BoukhariRepresentation Geometry Masked Language Modeling
´Ecole Nationale Sup´erieure d’Informatique (ESI), Algiers , Algeria
How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing
Feb 25, 2026Javier MarínTransformer Interpretability LLM Interpretability
Groundlens-dev
Symmetry in language statistics shapes the geometry of model representations
Feb 16, 2026Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2Representation Learning Neural Representation Geometry
1UC Berkeley · 2EPFL · 3Johns Hopkins +1
From Directions to Regions: Decomposing Activations in Language Models via Local Geometry
Feb 2, 2026Or Shafran, Shaked Ronen, Omri Fahn +3Language Model Steering LLM Interpretability
Blavatnik School of Computer Science and AI, Tel Aviv University, Israel · New York University, New York, NY, USA · Goodfire.
Functional Subspace, where language models can use vector algebra to solve problems
Feb 2, 2026Jung H. Lee, Sujith VijayanLLM Interpretability In-Context Learning
Pacific Northwest National Laboratory Seattle, WA · School of Neuroscience Virginia Tech Blacksburg, VA

May 1, 2026, cs.AIJ/K move · Enter open · S save

Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts

Sheridan Feucht, Tal Haklay, Usha Bhalla, Daniel Wurgaft+8

aNortheastern University · bTechnion IIT · cHarvard University+1

Numerical Reasoning in Language Models LLM Interpretability Mechanistic Interpretability Causal Abstraction Representation Geometry in Language Models LLM Mathematical Reasoning

Open paper AskPDF arXiv

Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs.

Momentum

Related topics

Latest papers 221