Analysing the Linearity of Linguistic Relations in Language Model Embedding Spaces

Vasudevan Nedumpozhimana; Fathima Thekkekara; John Kelleher

cs.CLSep 18, 2026

Analysing the Linearity of Linguistic Relations in Language Model Embedding Spaces

Authors: Vasudevan Nedumpozhimana, Fathima Thekkekara, John Kelleher

Organizations: ADAPT Research Centre Trinity College Dublin, Ireland · Indian Institute of Technology Bombay Mumbai, India

Abstract

We propose a framework to analyse how strongly different linguistic relations are linearly encoded in language model embedding spaces. We formalise linear encoding via a constrained linear approximation over related and unrelated word pairs and apply this to an extended BATS dataset covering inflectional, derivational, lexicographic, and encyclopedic relations in GloVe, RoBERTa, and ModernBERT. Our experiments show near-perfect linear encodings for inflectional and derivational relations, but substantially higher errors for lexicographic and encyclopedic relations, especially for one-to-many and many-to-many associations. We also find that RoBERTa and ModernBERT generally encode relations more linearly than GloVe. These results indicate that our framework can reveal which relational structures are most linearly accessible in embeddings, offering a compact tool for probing and comparing relational geometry across models.

Figures & tables

Relations	# Related pairs			# Unrelated pairs			Avg error of approximation
	Min	Max	Avg	Min	Max	Avg	GloVe	RoBERTa	ModernBERT
Inflectional	100	131	108.6	9900	17030	11765.2	0	0	0
Derivational	101	202	124.7	9999	20200	13909.8	0	0	0
Encyclopedic	104	284	177.2	2751	11539	7544.9	0.4704	0.4685	0.4549
Lexicographic	209	1988	965.1	12814	197900	63923.3	0.9201	0.8139	0.8360

Table 1: Statistics of dataset and linear approximation errors (macro average) of BATS relations.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

Relational Linear Properties in Language Models: An Empirical Investigation
May 21, 2026Giovanni Valer, Luigi Gresele, Marco Bronzini +1LLM Interpretability Factual Knowledge in Language Models
Fondazione Bruno Kessler, Trento, Italy · University of Bologna, Bologna, Italy · University of Pisa, Pisa, Italy +2
Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis
Mar 18, 2026Andor Diera, Ansgar ScherpLLM Interpretability
Ulm University, Ulm, Germany
Linear Representations of Hierarchical Concepts in Language Models
Apr 9, 2026Masaki Sakata, Benjamin Heinzerling, Takumi Ito +2Hierarchical Representation Learning LLM Interpretability
Tohoku University · RIKEN · Langsmith Inc. +2

May 21, 2026J/K move · Enter open · S save

Relational Linear Properties in Language Models: An Empirical Investigation

Giovanni Valer, Luigi Gresele, Marco Bronzini, Emanuele Marconato

Fondazione Bruno Kessler, Trento, Italy · University of Bologna, Bologna, Italy · University of Pisa, Pisa, Italy+2

LLM Interpretability Factual Knowledge in Language Models Language Model Probing

Open paper AskPDF arXiv

Linear properties are ubiquitous in the representations of language models; however, testing them experimentally remains a challenging task. This work focuses on relational linearity: the hypothesis that, for a fixed relation (e.g., "plays"), the unembedding of an object (e.g., "trumpet") can be predicted from the embedding of its subject (e.g.,"Miles Davis") by a linear map. We present an experimental method to test the formulation of relational linearity by Marconato et al. (2025). Specifically, we introduce a probing method, based on Kullback-Leibler divergence, to evaluate this property and examine its variation across layers and paraphrased relational queries. It is also more efficient than previous work; for example, it avoids the crude Jacobian approximations used in Linear Relational Embeddings by Hernandez et al. (2024). Our findings across four datasets show that relational linearity varies across models, exhibits layer-wise patterns consistent with prior observations about linguistic information in model representations, and is differently affected by changes in how the relation is phrased.

Abstract

Representation Geometry in Language Models

Explore similar work

Relational Linear Properties in Language Models: An Empirical Investigation

Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis

Linear Representations of Hierarchical Concepts in Language Models