Representation Geometry in Language Models

Ask about this topic

Overview

This field studies the shapes and relationships formed by token embeddings and hidden activations, and how those structures change across layers, contexts, and training. Researchers use linear probes, representational similarity analysis, spectral and manifold methods, sparse autoencoders, and causal interventions such as activation patching and steering. These tools are applied to questions about reasoning, multilingual transfer, retrieval, safety, and reliability, using settings that range from controlled synthetic tasks to benchmarks such as GSM8K and MTEB. A central open tension is that information being decodable or geometrically similar does not guarantee that a model uses it causally; useful structure may also be local, nonlinear, or sensitive to measurement choices.

Latest papers

CardsList

417 papers78 papers in the last four weeks, up 34% on the four weeks before. 0.5% of all new papers.

Feb 28, 2026

A Gauge Theory of Superposition: Toward a Sheaf-Theoretic Atlas of Neural Representations

We develop a discrete gauge-theoretic framework for superposition in large language models (LLMs) that replaces the single-global-dictionary premise with a sheaf-theoretic atlas of local semantic charts. Contexts are clustered into a stratified context complex; each chart carries a local feature space and a local information-geometric metric (Fisher/Gauss-Newton) identifying predictively consequential feature interactions. This yields a Fisher-weighted interference energy and three measurable obstructions to global interpretability: (O1) local jamming (active load exceeds Fisher bandwidth), (O2) proxy shearing (mismatch between geometric transport and a fixed correspondence proxy), and (O3) nontrivial holonomy (path-dependent transport around loops). We prove and instantiate four results on a frozen open LLM (Llama-3.2-3B Instruct) using WikiText-103, a C4-derived English web-text subset, and the-stack-smol. (A) After constructive gauge fixing on a spanning tree, each chord residual equals the holonomy of its fundamental cycle, making holonomy computable and gauge-invariant. (B) Shearing lower-bounds a data-dependent transfer mismatch energy, turning DshearD_{\mathrm{shear}} into an unavoidable failure bound. (C) We obtain non-vacuous certified jamming/interference bounds with high coverage and zero violations across seeds and hyperparameters. (D) Bootstrap and sample-size experiments show stable estimation of DshearD_{\mathrm{shear}} and DholD_{\mathrm{hol}}, with improved concentration on well-conditioned subsystems.

Hossein Javidnia

School of Computing, Dublin City University, Ireland

LLM InterpretabilityRepresentation Geometry in Language ModelsNeural Representation Geometry
AskPDF
Feb 27, 2026

Universal Conceptual Structure in Neural Translation: Probing NLLB-200's Multilingual Geometry

Do neural machine translation models learn language-universal conceptual representations, or do they merely cluster languages by surface similarity? We investigate this question by probing the representation geometry of Meta's NLLB-200, a 200-language encoder-decoder Transformer, through six experiments that bridge NLP interpretability with cognitive science theories of multilingual lexical organization. Using the Swadesh core vocabulary list embedded across 135 languages, we find that the model's embedding distances significantly correlate with phylogenetic distances from the Automated Similarity Judgment Program (ρ=0.13ρ= 0.13, p=0.020p = 0.020), demonstrating that NLLB-200 has implicitly learned the genealogical structure of human languages. We show that frequently colexified concept pairs from the CLICS database exhibit significantly higher embedding similarity than non-colexified pairs (U=42656U = 42656, p=1.33×10−11p = 1.33 \times 10^{-11}, d=0.96d = 0.96), indicating that the model has internalized universal conceptual associations. Per-language mean-centering of embeddings improves the between-concept to within-concept distance ratio by a factor of 1.19, providing geometric evidence for a language-neutral conceptual store analogous to the anterior temporal lobe hub identified in bilingual neuroimaging. Semantic offset vectors between fundamental concept pairs (e.g., man to woman, big to small) show high cross-lingual consistency (mean cosine = 0.84), suggesting that second-order relational structure is preserved across typologically diverse languages. We release InterpretCognates, an open-source interactive toolkit for exploring these phenomena, alongside a fully reproducible analysis pipeline.

Kyle Elliott Mathewson

University of Alberta

Multilingual Language ModelsRepresentation Geometry in Language ModelsCross-Lingual Consistency in Language ModelsCross-Lingual Representation Alignment
AskPDF
Feb 26, 2026

Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA

Large Language Models (LLMs) obey consistent scaling laws -- empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these laws are descriptive rather than prescriptive: they characterize typical training, not optimal training. Surprisingly few works have successfully challenged the data-efficiency bounds implied by these laws -- which is our primary focus. To that end, we introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear. Building on this principle, we propose a novel Semantic Tube Prediction (STP) task, a JEPA-style regularizer that confines hidden-state trajectories to a tubular neighborhood of the geodesic. STP generalizes JEPA to language without requiring explicit multi-view augmentations. We show this constraint improves signal-to-noise ratio, and consequently preserves diversity by preventing trajectory collisions during inference. Empirically, STP allows LLMs to match baseline accuracy with 16×\times less training data on the NL-RX-SYNTH dataset, directly violating the data term of Chinchilla-style scaling laws and demonstrating that principled geometric priors can surpass brute-force scaling. Code is available at https://github.com/galilai-group/llm-jepa#stp.

Hai Huang, Yann LeCun, Randall Balestriero

Atlassian · NYU · Brown

Efficient Language Model TrainingLLM TrainingJoint-Embedding Predictive ArchitectureRepresentation Geometry in Language ModelsLanguage Model Scaling Laws
AskPDF
Feb 25, 2026

Causality ≠\neq Invariance: Function and Concept Vectors in LLMs

Do large language models (LLMs) represent concepts abstractly, i.e., independent of input format? We revisit Function Vectors (FVs), compact representations of in-context learning (ICL) tasks that causally drive task performance. Across multiple LLMs, we show that FVs are not fully invariant: FVs are nearly orthogonal when extracted from different input formats (e.g., open-ended vs. multiple-choice), even if both target the same concept. We identify Concept Vectors (CVs), which carry more stable concept representations. Like FVs, CVs are composed of attention head outputs; however, unlike FVs, the constituent heads are selected using Representational Similarity Analysis (RSA) based on whether they encode concepts consistently across input formats. While these heads emerge in similar layers to FV-related heads, the two sets are largely distinct, suggesting different underlying mechanisms. Steering experiments reveal that FVs excel in-distribution, when extraction and application formats match (e.g., both open-ended in English), while CVs generalize better out-of-distribution across both question types (open-ended vs. multiple-choice) and languages. Our results show that LLMs do contain abstract concept representations, but these differ from those that drive ICL performance.

Gustaw Opiełka, Hannes Rosenbusch, Claire E. Stevenson

University of Amsterdam

Representation Geometry in Language ModelsIn-Context Learning
AskPDF
Feb 25, 2026

Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory

This thesis addresses two persistent and closely related challenges in modern deep learning, reliability and efficiency, through a unified framework grounded in Spectral Geometry and Random Matrix Theory (RMT). As deep networks and large language models continue to scale, their internal behavior becomes increasingly opaque, leading to hallucinations, fragile generalization under distribution shift, and growing computational and energy demands. By analyzing the eigenvalue dynamics of hidden activations across layers and inputs, this work shows that spectral statistics provide a compact, stable, and interpretable lens on model behavior, capable of separating structured, causal representations from noise-dominated variability. Within this framework, the first contribution, EigenTrack, introduces a real-time method for detecting hallucinations and out-of-distribution behavior in large language and vision-language models. EigenTrack transforms streaming activations into spectral descriptors such as entropy, variance, and deviations from the Marchenko-Pastur baseline, and models their temporal evolution using lightweight recurrent classifiers, enabling early detection of reliability failures before they appear in model outputs while offering interpretable insight into representation dynamics. The second contribution, RMT-KD, presents a principled approach to compressing deep networks via random matrix theoretic knowledge distillation. By interpreting outlier eigenvalues in activation spectra as carriers of task-relevant information, RMT-KD progressively projects networks onto lower-dimensional subspaces through iterative self-distillation, yielding significantly more compact and energy-efficient models while preserving accuracy and dense, hardware-friendly structure.

Davide Ettori

LLM CompressionRepresentation Geometry in Language ModelsKnowledge DistillationOOD DetectionHallucination Detection in VLMsRandom Matrix Theory
AskPDF
Feb 25, 2026

How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing

When a decoder-only transformer is forced to process matched correct and incorrect single-token continuations of a factual query, the two pathways through hidden-state space diverge: displacement vectors from the query-only representation keep near-equal magnitude but rotate apart, with angular separation growing through mid-depth before late layers resolve an asymmetric outcome. A logit-lens preference in the incorrect run falls far below the equal-probability prior (roughly 11.5x more mass on the incorrect token than the correct one). We read this pattern, rotational divergence then late-layer asymmetric commitment, as the geometric signature of the model externally appearing to reject a wrong continuation, while staying explicit that it is observational, not causal: the incorrect run could equally reflect the model conforming to the token it is forced to carry, which only a random-token control can settle. It holds across six decoder-only transformers spanning four architecture families (Llama, Mistral, Gemma, StableLM) from 1B to 13B parameters; a seventh (Qwen2 1.5B) is flat under our protocol, plausibly a tokenizer artefact, leaving an emergence threshold open. Linear probes recover the distinction at intermediate depth, and cross-domain transfer is structurally asymmetric, a financial-medical corridor transferring far better than transport pairs. Where single-layer activation patching is cleanly interpretable (LLaMA-2 13B, Mistral 7B) it yields no layer band of consistent recovery; a third model (StableLM-2 1.6B) recovers uniformly above the ceiling, which we diagnose as a code-path artefact and exclude. Under this scoped null the late-layer asymmetry is not localized to a single component, fitting a distributed-by-trajectory account rather than single-layer localized recall. We document this with forced-completion probing across seven models, three domains, and 300 queries.

Javier Marín

Groundlens-dev

Transformer InterpretabilityLLM InterpretabilityFactual Knowledge in Language ModelsRepresentation Geometry in Language ModelsLanguage Model Probing
AskPDF
Feb 24, 2026

Frayed RoPE and Long Inputs: A Geometric Perspective

Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models, which, while effective, causes performance breakdown when input length exceeds training length. Prior analyses assert (rightly) that long inputs cause channels to rotate ``out of distribution,'' but it is not clear how extra rotation relates to or causes pathological behavior. Through empirical and theoretical analysis we advance a unified geometric understanding of attention behavior with RoPE. We find that attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink tokens: placeholders that allow attention heads to avoid token mixing when not required. RoPE applied to longer inputs damages this key/query cluster separation, producing pathological behavior by inhibiting sink token functionality. From this geometric perspective, we propose RoPE-ID (In Distribution), a straightforward modification that allows attention layers to generalize to longer inputs out of the box: apply RoPE with high frequency to a subset of channels. We demonstrate the effectiveness of RoPE-ID for extended inputs using 1B and 3B parameter Transformers on the LongBench and RULER information retrieval benchmarks.

Davis Wertheimer, Aozhong Zhang, Derrick Liu +2

IBM Research · University at Albany, SUNY

Rotary Positional EmbeddingsLong-Context Language ModelingLong-Context ModelingRepresentation Geometry in Language ModelsAttention Sinks
AskPDF
Feb 23, 2026

Emergent Manifold Separability during Reasoning in Large Language Models

Chain-of-Thought (CoT) prompting significantly improves reasoning in Large Language Models, yet the temporal dynamics of the underlying representation geometry remain poorly understood. We investigate these dynamics by applying Manifold Capacity Theory (MCT) to two compositional reasoning tasks: a controlled Boolean logic tree that supports deep mechanistic analysis, and a natural-language eligibility task in which the model has to extract attributes from prose, compare them to thresholds, and compose the local decisions through a fixed evaluation tree. MCT lets us quantify the linear separability of latent representations without the confounding factors of probe training. On both tasks, and across several open-weight models, reasoning manifests as a transient geometric pulse: concept manifolds are untangled into linearly separable subspaces immediately prior to computation and rapidly compressed thereafter. This behavior diverges from standard linear probe accuracy, which remains high long after computation, suggesting a fundamental distinction between information that is merely retrievable and information that is geometrically prepared for processing. We interpret this phenomenon as Dynamic Manifold Management, a mechanism where the model dynamically modulates representational capacity to optimize the bandwidth of the residual stream throughout the reasoning chain.

Chanwoo Chun, Alexandre Polo, SueYeon Chung

Department of Physics, Harvard University · Center for Data Science, New York University · Kempner Institute, Harvard University +1

Representation Geometry in Language ModelsCoT ReasoningCompositional Reasoning
AskPDF
Feb 20, 2026

On the Semantic and Syntactic Information Encoded in Proto-Tokens for One-Step Text Reconstruction

Autoregressive large language models (LLMs) generate text token-by-token, requiring n forward passes to produce a sequence of length n. Recent work, Exploring the Latent Capacity of LLMs for One-Step Text Reconstruction (Mezentsev and Oseledets), shows that frozen LLMs can reconstruct hundreds of tokens from only two learned proto-tokens in a single forward pass, suggesting a path beyond the autoregressive paradigm. In this paper, we study what information these proto-tokens encode and how they behave under reconstruction and controlled constraints. We perform a series of experiments aimed at disentangling semantic and syntactic content in the two proto-tokens, analyzing stability properties of the e-token, and visualizing attention patterns to the e-token during reconstruction. Finally, we test two regularization schemes for "imposing" semantic structure on the e-token using teacher embeddings, including an anchor-based loss and a relational distillation objective. Our results indicate that the m-token tends to capture semantic information more strongly than the e-token under standard optimization; anchor-based constraints trade off sharply with reconstruction accuracy; and relational distillation can transfer batch-level semantic relations into the proto-token space without sacrificing reconstruction quality, supporting the feasibility of future non-autoregressive seq2seq systems that predict proto-tokens as an intermediate representation.

Ivan Bondarenko, Egor Palkin, Fedor Tikunov

Non-Autoregressive Text GenerationRepresentation Geometry in Language Models
AskPDF
Feb 19, 2026

Understanding Unreliability of Steering Vectors in Language Models: Geometric Predictors and the Limits of Linear Approximations

Steering vectors are a lightweight method for controlling language model behavior by adding a learned bias to the activations at inference time. Although effective on average, steering effect sizes vary across samples and are unreliable for many target behaviors. In my thesis, I investigate why steering reliability differs across behaviors and how it is impacted by steering vector training data. First, I find that higher cosine similarity between training activation differences predicts more reliable steering. Second, I observe that behavior datasets where positive and negative activations are better separated along the steering direction are more reliably steerable. Finally, steering vectors trained on different prompt variations are directionally distinct, yet perform similarly well and exhibit correlated efficacy across datasets. My findings suggest that steering vectors are unreliable when the latent target behavior representation is not effectively approximated by the linear steering direction. Taken together, these insights offer a practical diagnostic for steering unreliability and motivate the development of more robust steering methods that explicitly account for non-linear latent behavior representations.

Joschka Braun

Eberhard Karls University of Tübingen Faculty of Science

Language Model SteeringActivation SteeringLLM ReliabilityRepresentation Geometry in Language Models
AskPDF
Feb 19, 2026

ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization

Understanding what features are encoded by learned directions in LLM activation space requires identifying inputs that strongly activate them. Feature visualization, which optimizes inputs to maximally activate a target direction, offers an alternative to costly dataset search approaches, but remains underexplored for LLMs due to the discrete nature of text. Furthermore, existing prompt optimization techniques are poorly suited to this domain, which is highly prone to local minima. To overcome these limitations, we introduce ADAPT, a hybrid method combining beam search initialization with adaptive gradient-guided mutation, designed around these failure modes. We evaluate on Sparse Autoencoder latents from Gemma 2 2B, proposing metrics grounded in dataset activation statistics to enable rigorous comparison, and show that ADAPT consistently outperforms prior methods across layers and latent types. Our results establish that feature visualization for LLMs is tractable, but requires design assumptions tailored to the domain.

João N. Cardoso, Arlindo L. Oliveira, Bruno Martins

Instituto Superior T´ecnico · INESC-ID

LLM InterpretabilityPrompt OptimizationRepresentation Geometry in Language Models
AskPDF
Feb 19, 2026

Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomy

The black-box nature of Large Language Models necessitates novel evaluation frameworks that transcend surface-level performance metrics. This study investigates the internal neural representations of cognitive complexity using Bloom's Taxonomy as a hierarchical lens. By analyzing high-dimensional activation vectors from different LLMs, we probe whether different cognitive levels, ranging from basic recall (Remember) to abstract synthesis (Create), are linearly separable within the model's residual streams. Our results demonstrate that linear classifiers achieve approximately 95% mean accuracy across all Bloom levels, providing strong evidence that cognitive level is encoded in a linearly accessible subspace of the model's representations. These findings provide evidence that the model resolves the cognitive difficulty of a prompt early in the forward pass, with representations becoming increasingly separable across layers.

Bianca Raimondi, Maurizio Gabbrielli

Department of Computer Science and Engineering, University of Bologna, Bologna, Italy

LLM InterpretabilityRepresentation Geometry in Language ModelsLanguage Model Probing
AskPDF
Feb 17, 2026

The Geometric Anatomy of Capability Acquisition in Transformers

Neural networks gain capabilities during training, but the internal changes that precede capability acquisition are not well understood. In particular, the relationship between geometric change and behavioral change, and the effect of task difficulty and model scale on that relationship, is unclear. We track geometric measures and linear probes across six transformer sizes (405K--151M parameters), eight algorithmic tasks (144 task×\timeslevel×\timesmodel combinations), and three Pythia language models (160M--2.8B). Across all settings, representations first collapse to a low-dimensional state, then recover, and only then does behavioral performance improve. Linear probes show that the model's hidden states already contain task-relevant information before the model can act on it. The collapse floor is task-specific, the collapse propagates top-down through the network, and of the geometric measures tested, only \rankme reliably precedes capability acquisition for hard tasks. Whether this precursor is detectable depends on task difficulty relative to model capacity. For hard tasks, there is a clear gap: geometry changes first, behavior follows. For easy tasks, the model learns so quickly that both happen simultaneously and no precursor is detectable. On Pythia-2.8B, a logical deduction task that is genuinely hard for the model shows a precursor gap of ∼{\sim}49K training steps, while easy benchmarks show none. This suggests that geometric patterns observed in small proxy models can persist at larger scale when the task remains difficult relative to model capacity.

Jayadev Billa

San Jose, CA, USA

Representation Geometry in Language ModelsRepresentation Collapse
AskPDF
Feb 17, 2026

Recursive Concept Evolution for Compositional Reasoning in Large Language Models

Large language models achieve strong performance on many complex reasoning tasks, yet their accuracy degrades sharply on benchmarks that require compositional reasoning, including ARC-AGI-2, GPQA, MATH, BBH, and HLE. Existing methods improve reasoning by expanding token-level search through chain-of-thought prompting, self-consistency, or reinforcement learning, but they leave the model's latent representation space fixed. When the required abstraction is not already encoded in this space, performance collapses. We propose Recursive Concept Evolution (RCE), a framework that enables pretrained language models to modify their internal representation geometry during inference. RCE introduces dynamically generated low-rank concept subspaces that are spawned when representational inadequacy is detected, selected through a minimum description length criterion, merged when synergistic, and consolidated via constrained optimization to preserve stability. This process allows the model to construct new abstractions rather than recombining existing ones. We integrate RCE with Mistral-7B and evaluate it across compositional reasoning benchmarks. RCE yields 12-18 point gains on ARC-AGI-2, 8-14 point improvements on GPQA and BBH, and consistent reductions in depth-induced error on MATH and HLE.

Sarim Chaudhry

Purdue University

Compositional ReasoningTest-Time AdaptationRepresentation Geometry in Language ModelsCompositional Representation Learning
AskPDF
Feb 17, 2026

PERSONA: Dynamic and Compositional Inference-Time Personality Control via Activation Vector Algebra

Current methods for personality control in Large Language Models rely on static prompting or expensive fine-tuning, failing to capture the dynamic and compositional nature of human traits. We introduce PERSONA, a training-free framework that achieves fine-tuning level performance through direct manipulation of personality vectors in activation space. Our key insight is that personality traits appear as extractable, approximately orthogonal directions in the model's representation space that support algebraic operations. The framework operates through three stages: Persona-Base extracts orthogonal trait vectors via contrastive activation analysis; Persona-Algebra enables precise control through vector arithmetic (scalar multiplication for intensity, addition for composition, subtraction for suppression); and Persona-Flow achieves context-aware adaptation by dynamically composing these vectors during inference. On PersonalityBench, our approach achieves a mean score of 9.60, nearly matching the supervised fine-tuning upper bound of 9.61 without any gradient updates. On our proposed Persona-Evolve benchmark for dynamic personality adaptation, we achieve up to 91% win rates across diverse model families. These results provide evidence that aspects of LLM personality are mathematically tractable, opening new directions for interpretable and efficient behavioral control.

Xiachong Feng, Liang Zhao, Weihong Zhong +5

Harbin Institute of Technology · The University of Hong Kong

Controllable Text GenerationPersonality Modeling in Language ModelsActivation SteeringRepresentation Geometry in Language Models
AskPDF
Feb 17, 2026

The Information Geometry of Softmax: Probing and Steering

This paper concerns the question of how AI systems encode semantic structure into the geometric structure of their representation spaces. The motivating observation is that the natural geometry of these representation spaces should reflect the way models use representations to produce behavior. We focus on the important special case of representations that define softmax distributions. In this case, we argue that the natural geometry is information geometry. Our focus is on the role of information geometry on semantic encoding and the linear representation hypothesis. As an illustrative application, we develop "dual steering", a method for robustly steering representations to exhibit a particular concept using linear probes. We prove that dual steering optimally modifies the target concept while minimizing changes to off-target concepts. Empirically, we find that dual steering enhances the controllability and stability of concept manipulation.

Kiho Park, Todd Nief, Yo Joong Choe +1

University of Chicago · INSEAD

Representation Geometry in Language ModelsInformation GeometryRepresentation SteeringLanguage Model SteeringLinear Probing
AskPDF
Feb 16, 2026

Symmetry in language statistics shapes the geometry of model representations

The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational manifolds originate in the statistical symmetries of natural language.

Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2

1UC Berkeley · 2EPFL · 3Johns Hopkins +1

Representation LearningNeural Representation GeometryWord EmbeddingsRepresentation Geometry in Language ModelsLatent Space Geometry
AskPDF
Feb 16, 2026

Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts

Large Language Models (LLMs) have achieved remarkable progress, with Parameter-Efficient Fine-Tuning (PEFT) emerging as a key technique for downstream task adaptation. However, existing PEFT methods mainly operate in Euclidean space, fundamentally limiting their capacity to capture complex geometric structures inherent in language data. While alternative geometric spaces, like hyperbolic geometries for hierarchical data and spherical manifolds for circular patterns, offer theoretical advantages, forcing representations into a single manifold type ultimately limits expressiveness, even when curvature parameters are learnable. To address this, we propose Mixture of Space (MoS), a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations. Building on this scheme, we develop MoSLoRA, which extends Low-Rank Adaptation (LoRA) with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context. Furthermore, to address the computational overhead of frequent manifold switching, we develop a lightweight routing mechanism. Moreover, we provide empirical insights into how curvature optimization impacts training stability and model performance. Our experiments across diverse benchmarks demonstrate that MoSLoRA consistently outperforms strong baselines, achieving up to 5.6% improvement on MATH500 and 15.9% on MAWPS.

Buze Zhang, Jinkai Tao, Zilang Zeng +4

School of Software Engineering, Xi’an Jiaotong University, China · School of Information, Central University of Finance and Economics, Beijing, China · Department of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China +2

Parameter-Efficient Fine-TuningRepresentation Geometry in Language ModelsLow-Rank Adaptation
AskPDF
Feb 10, 2026

StructLens: A Structural Lens for Language Models via Maximum Spanning Trees

Language exhibits inherent structures, a property that explains both language acquisition and language change. Given this characteristic, we expect language models to manifest their own internal structures as well. While interpretability research has investigated how models compute representations mechanistically through attention patterns and Sparse AutoEncoders, the organization of the resulting representations is overlooked. To address this gap, we introduce StructLens, a framework to analyze representations through a holistic structural view. StructLens constructs maximum spanning trees based on the semantic representations in residual streams, inspired by tree representation in dependency parsing, and provides summaries of token relationships in representation space. We analyze how contiguous tokens are also nearby in representation space and find that middle layers show the strongest local-span organization. Moreover, analysis of pre-training checkpoints reveals that smaller local units become detectable earlier in pre-training, and larger units later. Our findings demonstrate that StructLens provides insights into how models organize token representations across layers and training. Our code is available at https://github.com/naist-nlp/structlens.

Haruki Sakajo, Frederikus Hudi, Yusuke Sakai +2

Nara Institute of Science and Technology (NAIST)

LLM InterpretabilityRepresentation Geometry in Language Models
AskPDF
Feb 2, 2026

From Directions to Regions: Decomposing Activations in Language Models via Local Geometry

Activation decomposition methods in language models are tightly coupled to geometric assumptions on how concepts are realized in activation space. Existing approaches search for individual global directions, implicitly assuming linear separability, which overlooks concepts with nonlinear or multi-dimensional structure. In this work, we leverage Mixture of Factor Analyzers (MFA) as a scalable, unsupervised alternative that models the activation space as a collection of Gaussian regions with their local covariance structure. MFA decomposes activations into two compositional geometric objects: the region's centroid in activation space, and the local variation from the centroid. We train large-scale MFAs for Llama-3.1-8B and Gemma-2-2B, and show they capture complex, nonlinear structures in activation space. Moreover, evaluations on localization and steering benchmarks show that MFA outperforms unsupervised baselines, is competitive with supervised localization methods, and often achieves stronger steering performance than sparse autoencoders. Together, our findings position local geometry, expressed through subspaces, as a promising unit of analysis for scalable concept discovery and model control, accounting for complex structures that isolated directions fail to capture.

Or Shafran, Shaked Ronen, Omri Fahn +3

Blavatnik School of Computer Science and AI, Tel Aviv University, Israel · New York University, New York, NY, USA · Goodfire.

Language Model SteeringLLM InterpretabilityGaussian Mixture ModelsLanguage ModelingRepresentation Geometry in Language ModelsConcept Localization in Language Models
AskPDF
Feb 2, 2026

Language Steering for Multilingual In-Context Learning

If multilingual language model representations contain separable language-associated structure, then shifting between languages may require only simple activation-space interventions. To test this hypothesis, we study multilingual in-context learning (ICL) as a controlled setting: few-shot demonstrations are provided in English while the test query is in a target language, creating a clear cross-lingual transfer challenge. We propose language vectors, which are computed as mean activation differences between parallel source and target language examples at a particular layer and applied as inference-time offsets to the model's hidden states. Across three tasks spanning 19 languages and three models, our method improves over English few-shot prompting without any parameter updates. Beyond performance, the vectors encode interpretable linguistic structure: closely related languages form tight clusters, and vectors transfer effectively across tasks and languages, with closely related languages' vectors recovering more of the gain than distant ones. Together, these findings suggest that language occupies structured directions in multilingual activation space, and that inference-time interventions on English demonstrations can narrow the cross-lingual ICL gap.

Neeraja Kirtane, Kuan-Hao Huang

Texas A&M University

Multilingual Language ModelsCross-Lingual TransferIn-Context LearningActivation SteeringRepresentation Geometry in Language Models
AskPDF
Feb 2, 2026

Functional Subspace, where language models can use vector algebra to solve problems

Large language models (LLMs) were invented for natural language tasks such as translation, but they have proved that they can perform highly complex functions across domains. Additionally, they have been thought to develop new skills without being trained on them. These learning capabilities lead to LLMs adoption in a wide range of domains. Thus, it is imperative that we understand their operating mechanisms and limitations for proper diagnostics and repair. The earlier studies proposed that high level concepts are encoded as linear directions in LLMs activation space and that the geometry of embeddings have semantic meanings. Inspired by these studies, we hypothesize that LLMs may use subspaces and vector algebra in subspaces to perform tasks. To address this hypothesis, we analyze LLMs' functional modules and residual streams collected from LLMs engaging in in-context learning (ICL), one of the emergent abilities. Our analyses suggest that 1) LLMs can create subspaces, where evidence can be accumulated and 2) ICL tasks can be solved via simple algebraic operations in subspaces.

Jung H. Lee, Sujith Vijayan

Pacific Northwest National Laboratory Seattle, WA · School of Neuroscience Virginia Tech Blacksburg, VA

LLM InterpretabilityIn-Context LearningRepresentation Geometry in Language Models
AskPDF
Jan 31, 2026

From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs

We investigate the extent to which an LLM's hidden-state geometry can be recovered from its behavior in psycholinguistic experiments. Across eight instruction-tuned transformer models, we run two experimental paradigms -- similarity-based forced choice and free association -- over a shared 5,000-word vocabulary, collecting 17.5M+ trials to build behavior-based similarity matrices. Using representational similarity analysis, we compare behavioral geometries to layerwise hidden-state similarity and benchmark against FastText, BERT, and cross-model consensus. We find that forced-choice behavior aligns substantially more with hidden-state geometry than free association. In a held-out-words regression, behavioral similarity (especially forced choice) predicts unseen hidden-state similarities beyond lexical baselines and cross-model consensus, indicating that behavior-only measurements retain recoverable information about internal semantic geometry. Finally, we discuss implications for the ability of behavioral tasks to uncover hidden cognitive states.

Louis Schiekiera, Max Zimmer, Christophe Roux +2

Computational Modelling Lab, Humboldt-Universität zu Berlin, Berlin, Germany · Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany · Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany +1

Representation Geometry in Language ModelsComputational PsycholinguisticsLanguage Model Probing
AskPDF
Nov 27, 2025

From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures

Studying the shape of text embedding spaces enhances our understanding of model behavior and how geometry affects downstream task performance. While a range of topological and geometric metrics have been explored, they are typically analyzed in isolation, and the connections between them remain largely unknown. In this paper, we present Unified Topological Signatures (UTS) for characterizing the structure of embedding spaces. We evaluate our method across a wide range of text embedding models and retrieval datasets, finding that it outperforms individual metrics in downstream performance prediction. Through several experiments, we demonstrate that the signatures characterize embedding spaces more accurately, improve the prediction of model-specific properties and biases, and reveal novel insights into representational similarity that challenge the global claims of the Platonic Representation Hypothesis.

Florian Rottach, William Rudman, Bastian Rieck +2

University of Tübingen · Boehringer Ingelheim GmbH, Biberach (Riss), Germany · The University of Texas at Austin +1

Representation Geometry in Language ModelsText Embedding EvaluationTopological Data AnalysisRepresentational Similarity Analysis
AskPDF
Oct 8, 2025

Contextual Representation Flow Across Depth in State-Space and Transformer Architectures

State Space Models (SSMs) have emerged as efficient, linear-time alternatives to Transformer-Based Models (TBMs) for long-context processing. However, their different mechanisms raise an important question: how does contextual representation flow across depth differ between these model families and their hybrids? We examine this question across twenty-one pretrained models, combining token-wise spectral diversity, layer-wise variation on seven datasets, and answer-location probing on two retrieval tasks. This joint analysis reveals that TBMs retain most of their token diversity at the output, whereas SSMs reach richer intermediate diversity but retain less of it on average, with late-layer compression. Comparisons with reinitialized weights reveal that these profiles depend on pretrained weights, while hybrids combine strong answer-location accessibility with architecture-dependent geometry, showing that high accessibility need not require the steadiest representation flow. Finally, we conduct theoretical case studies connecting these contrasts to mechanism-level differences: query-time token mixing in TBMs, repeated recurrent updates in SSMs, and preserved attention updates in hybrids.

Nhat M. Hoang, Do Xuan Long, Cong-Duy Nguyen +3

Nanyang Technological University, Singapore · National University of Singapore

State Space ModelsState Space Language ModelsTransformerRepresentation Geometry in Language ModelsLong-Context Modeling
AskPDF
Jul 16, 2025

Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility

Language models (LMs) are used for a diverse range of tasks, from question answering to writing fantastical stories. In order to reliably accomplish these tasks, LMs must be able to discern the modal category of a sentence (i.e., whether it describes something that is possible, impossible, completely nonsensical, etc.). However, recent studies have called into question the ability of LMs to categorize sentences according to modality (Michaelov et al., 2025; Kauf et al., 2023). In this work, we identify linear representations that discriminate between modal categories within a variety of LMs, or modal difference vectors. Analysis of modal difference vectors reveals that LMs have access to more reliable modal categorization judgments than previously reported. Furthermore, we find that modal difference vectors emerge in a consistent order as models become more competent (i.e., through training steps, layers, and parameter count). Notably, we find that modal difference vectors identified within LM activations can be used to model fine-grained human categorization behavior. This potentially provides a novel view into how human participants distinguish between modal categories, which we explore by correlating projections along modal difference vectors with human participants' ratings of interpretable features. In summary, we derive new insights into LM modal categorization using techniques from mechanistic interpretability, with the potential to inform our understanding of modal categorization in humans.

Michael A. Lepori, Jennifer Hu, Ishita Dasgupta +3

Department of Computer Science, Brown University · Google DeepMind · Department of Cognitive & Psychological Sciences, Brown University

Representation Geometry in Language Models
AskPDF
May 30, 2025

Localizing Persona Representations in LLMs

We present a study on how and where personas -- defined by distinct sets of human characteristics, values, and beliefs -- are encoded in the representation space of large language models (LLMs). Using a range of dimension reduction and pattern recognition methods, we first identify the model layers that show the greatest divergence in encoding these representations. We then analyze the activations within a selected layer to examine how specific personas are encoded relative to others, including their shared and distinct embedding spaces. We find that, across multiple pre-trained decoder-only LLMs, the analyzed personas show large differences in representation space only within the final third of the decoder layers. We observe overlapping activations for specific ethical perspectives -- such as moral nihilism and utilitarianism -- suggesting a degree of polysemy. In contrast, political ideologies like conservatism and liberalism appear to be represented in more distinct regions. These findings help to improve our understanding of how LLMs internally represent information and can inform future efforts in refining the modulation of specific human traits in LLM outputs. Warning: This paper includes potentially offensive sample statements.

Celia Cintas, Miriam Rateike, Erik Miehling +2

IBM Research Africa · IBM Research Europe

LLM InterpretabilityPersonality Modeling in Language ModelsRepresentation Geometry in Language Models
AskPDF