A Survey on the Linear Representation Hypothesis

Sewoong Lee; Marc E. Canby; Ikhyun Cho; Julia Hockenmaier

cs.AISep 19, 2026

A Survey on the Linear Representation Hypothesis

Authors: Sewoong Lee, Marc E. Canby, Ikhyun Cho, Julia Hockenmaier

Organizations: Siebel School of Computing and Data Science The Grainger College of Engineering University of Illinois Urbana-Champaign

Abstract

The term "linear representation hypothesis" (LRH) has appeared across diverse subfields of artificial intelligence, neuroscience, and cognitive science. But previous works have not consistently treated the LRH as a falsifiable scientific hypothesis; we analyze these inconsistencies and examine their implications for how prior theoretical and methodological results should be interpreted. Based on this analysis, we argue that claims regarding linear representations become well-defined only through careful examination of the model, representation location, feature definition, and evaluation dataset. We therefore propose a more rigorous formalization of the LRH that makes these dependencies explicit and allows the hypothesis to be evaluated as a falsifiable scientific claim. Finally, we identify some non-trivial open problems that warrant further attention from the research community.

Figures & tables

Year	New papers	Cumulative
1976	1	1
2001	1	2
2007	1	3
2022	2	5
2023	11	16
2024	149	165

Table 1: Number of papers explicitly employing the term “linear representation hypothesis” over time, based on Google Scholar search results (counts as of December 2025). The early history is discussed in Section 2.1 . The sharp increase after 2022 was triggered by Elhage et al. (2022) .

Representation	Model	Layer	Feature	Source
Linear representation	OthelloGPT	${\color[rgb]{0.75,0.75,0.75}\bullet}\bullet{\color[rgb]{0.75,0.75,0.75}\bullet}$	Board state of Othello	Nanda et al. (2023)
	GPT2-XL, GPT-J, OPT, LLaMA3 8B	${\color[rgb]{0.75,0.75,0.75}\bullet}\bullet{\color[rgb]{0.75,0.75,0.75}\bullet}$	Negative-to-positive sentiment shift, detoxification	Turner et al. (2023)
	Claude 3 Sonnet	$\circ\bullet\circ$	The Golden Gate Bridge, implementing addition functions in code, security vulnerability, gender bias, sycophancy, deception, harmful behavior	Templeton et al. (2024)
	GPT2-small, Pythia 1.4B, Pythia 2.8B	${\color[rgb]{0.75,0.75,0.75}\bullet}\bullet{\color[rgb]{0.75,0.75,0.75}\bullet}$	Movie review sentiment	Tigges et al. (2024)
	LLaMA2 7B	$\circ\circ\bullet$	Gender, translation, morphology, plurality, country–capital; part holonym/meronym not observed	Park et al. (2024c)
	LLaMA2 7B	$\circ\circ\bullet$	Gender, translation, morphology, plurality, country–capital	Jiang et al. (2024)

Table 2: Summary of studies providing empirical evidence for, or counterexamples to, the linear representation hypothesis (LRH). The Representation column indicates whether the LRH holds in the study; when it does not hold, the observed non-linear representation is specified. In the Layer column,

\bullet\circ\circ

denotes the initial (non-contextualized) embedding layer,

\circ\bullet\circ

intermediate layers, and

\circ\circ\bullet

the final layer before unembedding. Gray circles (

{\color[rgb]{0.75,0.75,0.75}\bullet}

) indicate that the corresponding layer was also examined but exhibited weaker linear representations.

Context of LRH Usage	Papers (may appear in multiple rows)
Type 1. Theoretical analyses supporting LRH	Park et al. (2024c) ; Jiang et al. (2024) ; Park et al. (2025) ; Liu et al. (2026)
Type 2. Empirical evidence: Supporting examples	Nanda et al. (2023) ; Turner et al. (2023) ; Templeton et al. (2024) ; Tigges et al. (2024) ; Park et al. (2024c) ; Jiang et al. (2024) ; Gurnee and Tegmark (2024) ; Nguyen and Leng (2025) ; Park et al. (2025) ; Kim et al. (2025) ; Agarwal et al. (2025)
Type 3. Empirical evidence: Counterexamples	Csordás et al. (2024) ; Engels et al. (2025a) ; Modell et al. (2025) ; Kantamneni and Tegmark (2025) ; Lei and Cooper (2025a) ; Walker et al. (2025)
Type 4. Empirical analysis of when LRH holds	Merullo et al. (2025)
Type 5. Statistical hypothesis testing of LRH	Reizinger et al. (2025) ; Guo et al. (2025)
Type 6. Critical discussions of LRH limitations	Marshall et al. (2024) ; Wattenberg and Viégas (2024) ; Tan et al. (2024) ; Canby et al. (2025) ; Muraleedharan (2025) ; Geiger et al. (2025) ; Tiblias et al. (2025)

Table 3: Contexts in which the term “linear representation hypothesis (LRH)” is used across the literature. A substantial fraction of prior work employs the LRH primarily as a methodological assumption ( Type 8 ) to justify linear analysis tools, such as linear probing, sparse autoencoder (SAE), and steering vector. As argued in Section 5.3 , when such assumptions are invoked across different layers or feature definitions, the resulting methodological justification may not necessarily transfer across studies.

Linear Representation Hypothesis

The Linear Representation Hypothesis holds that concepts and attributes are encoded as directions in a model’s activation space.

See all 16 papers

The Linear Representation Hypothesis Needs a Group Action
Sep 22, 2026Louie Hong Yao, Yuhao Li, Shengchao LiuLinear Representation Hypothesis Neural Network Interpretability
The Chinese University of Hong Kong
From Plato to Aristotle: Toward a General Theory of Representation Hypotheses
Oct 14, 2026Dai Shi, Andi Han, Xiaoyu Li +2Representation Alignment Platonic Representation Hypothesis
University of Cambridge · University of Sydney · University of New South Wales
Representation Alignment Rests on Linear Structure
May 22, 2026Kiril Bangachev, Guy Bresler, Yury PolyanskiyLinear Representation Hypothesis Cross-Modal Alignment
Department of Electrical Engineering and Computer Science · Massachusetts Institute of Technology

Sep 22, 2026J/K move · Enter open · S save

The Linear Representation Hypothesis Needs a Group Action

Louie Hong Yao, Yuhao Li, Shengchao Liu

The Chinese University of Hong Kong

Linear Representation Hypothesis Neural Network Interpretability

Open paper AskPDF arXiv

To make claims about representations that generalize beyond a particular trained model, we need to specify when two representations should count as equivalent. The Linear Representation Hypothesis is often discussed without making this equivalence explicit. Different notions of equivalence preserve different structures, so metrics, probes, and interventions that appear to study the same representation may in fact correspond to different hypotheses. We therefore argue that the Linear Representation Hypothesis is not one hypothesis but a family of claims distinguished by representation equivalence. We formalize this idea using group actions, specifying the representation object, the procedure that produces it, and the property ultimately asserted, while accounting for equivalences imposed by the model architecture. This framework clarifies how assumptions can change across metrics, reading points, and analysis stages, and we use it to audit common representation quantities and recent interpretability analyses.

Abstract

Linear Representation Hypothesis

Explore similar work

The Linear Representation Hypothesis Needs a Group Action

From Plato to Aristotle: Toward a General Theory of Representation Hypotheses

Representation Alignment Rests on Linear Structure