Representation Geometry in Language Models

Follow this topic Ask about this topic

Momentum

48 papers in the last four weeks, up 118% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 221

All topics

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs
Aug 31, 2026Deniz Bayazit, Badr AlKhamissi, Antoine BosselutMultilingual Language Models Multilingual Language Model Evaluation
EPFL
Geometry of Divergence: Tracking Hidden-State Trajectories for Adaptive Multi-Turn Reasoning
Aug 31, 2026Jie Liang, Zhengxin Yu, Hamid Nasiri +1Long-Horizon LLM Agents Representation Geometry in Language Models
School of Computing and Communications, Lancaster University, UK · School of Computing and Engineering, University of Huddersfield, UK
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization
Aug 31, 2026Junjie Yao, Liangkai Hang, Zhi-Qin John XuLanguage Model Pretraining Word Embeddings
School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China · Institute of Natural Sciences, School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China
Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees
Aug 30, 2026Shulei WangLanguage Model Pretraining Next-Token Prediction
Department of Statistics, University of Illinois at Urbana-Champaign, 605 E. Springfield Ave., Champaign, IL 61820
The Depth Flow of Token Representations Is Nonlinear and Does Not Descend Its Own Density
Aug 30, 2026Alexandre QuemyDynamical Systems Representation Geometry in Language Models
Hother Labs
A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddings
Aug 30, 2026Alexandre QuemyIntrinsic Dimensionality Neural Representation Geometry
Hother Labs
How Language Models Organize and Structure Moral Knowledge
Aug 27, 2026Orion Reblitz-RichardsonMoral Reasoning Representation Geometry in Language Models
Distiller Labs
Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations
Aug 27, 2026Anjishnu Mukherjee, Ziwei Zhu, Antonios AnastasopoulosLanguage Model Pretraining Multilingual Language Models
George Mason University
Falsehood and Impossibility Are Different Directions in an AI's Representation of Language
Aug 13, 2026Yoon Pyo LeeRepresentation Geometry in Language Models
Department of Nuclear, Plasma, and Radiological Engineering, University of Illinois Urbana-Champaign, Urbana, Illinois, USA.
Geometric and Behavioral Stratification in Transformer Residual Streams
Aug 12, 2026Nelson GudaTransformer Interpretability Transformer
Reading the Gate, Not the Interference: Output-Side Interference Measurement Does Not Track Merge Collapse
Aug 12, 2026Chencheng ZhuTask Vector Merging Task Arithmetic
UNSW Sydney
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
Aug 12, 2026Kai Yang, Jingwei Xu, Wanyu Wang +4RL for Language Model Reasoning Constrained RL
School of Computer Science, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · NovaCore +3
Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders
Aug 11, 2026Nikolai Bolik, Lennart Stöpler, Artur AndrzejakRepresentation Learning LLM Interpretability
Heidelberg University
Mapping and Measuring the Behavioral Evolution of Large Language Models
Aug 11, 2026Dong Qiao, Chris Ding, Jicong FanLLM Evaluation Representation Geometry in Language Models
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So
Aug 10, 2026Mark OskinTransformer Interpretability Transformer Attention
School of Computer Science and Engineering University of Washington
One Adapter Pair per Model: A Universal Activation Interface for Language Models
Aug 10, 2026Su-Hyeon Kim, Jiwan Mun, Yo-Sub HanRepresentation Geometry in Language Models Language Model Probing
Department of Artificial Intelligence, Yonsei University · Department of Computer Science, Yonsei University
Reproducing and Stress-Testing Two Approaches to LLM Reasoning Reliability: Test-Time Probability Aggregation and Logic-Representation Editing
Aug 9, 2026Minhan Cho, Jimin KweonLLM Evaluation Reasoning Consistency in Language Models
Remember & Company AI Lab · DSAIL, Dept. of Applied AI · Sungkyunkwan University +2
HoloAegis: Frozen Representation, Topological Inference --- Minimally Parametric Safety Manifolds and Their Capability Boundaries for LLM Guardrails
Aug 9, 2026Tak Ho Alex Li, Kaijie Liu, Lik-Hang Lee +3Language Model Safety Evaluation LLM Guardrails
Department of Mathematics, Hong Kong Baptist University, Hong Kong 999077 · Guangdong-Hong Kong-Macao Institute of ESG and New Quality Productive Forces, School of Optoelectronic Engineering, Guangdong Polytechnic Normal University, Guangzhou 510665 · Guangdong Institute of Digital Industry, Guangzhou 510000 +3
Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation
Aug 7, 2026Donald Loveland, Liam Collins, Bhuvesh Kumar +2Large Language Model-Based Recommendation Representation Geometry in Language Models
Snap Inc. Bellevue, WA, USA · University of Michigan Ann Arbor, MI, USA
Transformers Struggle to Use Their Emergent World Models: Revisiting the Tower of Hanoi, and the Illusion of Thinking
Aug 7, 2026Devin Pereira, Willem ZuidemaLLM World Models LLM Planning
1Artificial Intelligence Program, University of Amsterdam · ELLIS Unit Amsterdam · Institute for Logic Language and Computation, University of Amsterdam
Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models
Aug 6, 2026Lavanya Nigam, Ishaan Bansal, Aryan Sood +2Masked Diffusion Models Diffusion Language Model Decoding
Indian Institute of Technology Roorkee, Roorkee, Uttarakhand, India
Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs
Aug 6, 2026Hamed Damirchi, Ignacio Meza De la Jara, Damith Ranasinghe +2Language Model Error Detection LLM Interpretability
Australian Institute for Machine Learning · Adelaide University · Naval Group Pacific +1
Scaling Inherently Interpretable Language Models
Aug 6, 2026Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail +7Language Model Steering LLM Interpretability
Guide Labs Team
Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility
Aug 4, 2026Jo-Ku Cheng, Nikolaos Aletras, Marco ValentinoLanguage Model Pretraining Representation Geometry in Language Models
School of Computer Science, University of Sheffield, United Kingdom
The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner
Aug 4, 2026Simon Lam-MuirCompositional Reasoning Representation Geometry in Language Models
Prime Calibre, Australia
Language Models Encode the Contextual Truth of Propositions
Aug 4, 2026Rupak Sarkar, Pritika Ramu, Rachel RudingerLLM Sycophancy Knowledge Conflicts in Language Models
University of Maryland, College Park
A Graph Signal Processing Perspective on Numerical Sequence Representations in LLM In-Context Learning
Aug 4, 2026Jiajun Bao, Zihao Qi, Toni J. B. Liu +4Numerical Reasoning in Language Models In-Context Learning
1Cornell University · 2Goodfire AI · 3Imperial College London
ChaosProbe: A Neurochaotic Lens on Frozen Transformer Input-Embedding Spaces
Aug 3, 2026Kunal Kumar Pant, Nithin NagarajLanguage Model Fingerprinting Representation Probing
Department of Electronics and Communication Engineering (ECE) Amrita Vishwa Vidyapeetham Kollam, Kerala, India · Complex Systems Programme National Institute of Advanced Studies (NIAS) IISc Campus, Bengaluru Karnataka, India
Rewriting or Reweighting? A Geometric Account in Language Models
Aug 3, 2026Juntong Wang, Shengkun Yang, Xiyuan Wang +1Language Model Post-Training Representation Geometry in Language Models
Institute for Artificial Intelligence, Peking University, Beijing, China · School of Intelligence Science and Technology, Peking University, Beijing, China · School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China
Divergent large language model predictions from convergent representations in ambiguous word pairs
Aug 3, 2026K. Jack Scott, Narun Pat, Veronica LiesaputraTransformer Interpretability Decoder-Only Language Models
Department of Psychology, University of Otago, Dunedin, New Zealand · School of Computing, University of Otago, Dunedin, New Zealand

Aug 31, 2026, cs.CLJ/K move · Enter open · S save

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs

Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut

EPFL

Multilingual Language Models Multilingual Language Model Evaluation Representation Geometry in Language Models Language Model Probing

Open paper AskPDF arXiv

Latent language identification is often used to argue that multilingual language models route computation through language-specific states, such as English pivots. However, existing probes infer latent language from different signals, such as the geometry of hidden states or what can be decoded from intermediate representations. Since such claims shape conclusions about how models share and route information across languages, we ask whether these probes measure the same phenomenon or expose distinct aspects of multilingual computation. We study this question across model families, training regimes, domains, tasks, checkpoints, and up to 27 languages. We find that identification probes systematically disagree: the GMM-based representation probe, which draws evidence from hidden state geometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased signals. These differences track model multilinguality and training progression, but are comparatively stable across domains. Our results suggest a more cautious interpretation of latent language identification, where current probes expose different aspects of multilingual processing, rather than directly revealing a single internal lingua franca.

Momentum

Related topics

Latest papers 221