Predicting Linear Relational Decodability Without the Relation

Jonas Rohweder; Adrian Mülthaler; Faridoun Mehri; Sebastian Gerstner; Hinrich Schütze

cs.CLOct 19, 2026

Predicting Linear Relational Decodability Without the Relation

Authors: Jonas Rohweder, Adrian Mülthaler, Faridoun Mehri, Sebastian Gerstner, Hinrich Schütze

Organizations: Center for Information and Language Processing (CIS), LMU Munich · Ubiquitous Knowledge Processing Lab (UKP Lab), TU Darmstadt · Munich Center for Machine Learning (MCML) · relAI – Konrad Zuse School of Excellence in Reliable AI

Abstract

Linear relational embeddings (LREs) approximate a language model's subject-to-object mapping with an affine function. Their faithfulness varies enormously across relations, from near zero for person_father to the ceiling for country_capital_city. We investigate the factors that contribute to this variation and demonstrate that it can be significantly predicted from a readout that is not aware of the underlying relation. To do so, we propose a novel form of the Jacobian lens, the entity lens. Whereas the Jacobian lens is estimated across all tokens and concepts, the entity lens EL_t is trained to predict what the model will output for an entity of type t under the neutral prompt "{} is known for"; e.g., we train ELs for countries and persons. After estimating an entity lens, we measure how highly it ranks the model's answer for a relation without using any relation-specific information in estimation and prediction. Across 28 relations of the LRE dataset on Olmo-3-7B this relation-agnostic score correlates with LRE faithfulness at $ρ= 0.7355$ (p < .01). This suggests that linear decodability is largely a property of what a subject's representation already carries, rather than of the relation being decoded. Moving the unit of analysis from the relation to the individual fact, we conclude that new relation datasets and analysis methods are needed to address several problems in existing datasets, most prominently semantically weak first tokens (e.g., most predicted universities' first token is "University") and reference competition (e.g., the LLM confuses obscure actor names with competing non-actor entities that have the same last name). Our full pipeline is available at https://github.com/jonasrohw/entity-lens

Figures & tables

Figure 1: LRE faithfulness against the relation-agnostic CEL lens score, 28 relations. The lens sees no relation template, no objects, and no (subject, object) pairs, and uses one fixed layer throughout.

relation	prom.	gold $\in$ lens	$n$
city_in_country	0.96	0.71	24
person_plays_pro_sport	0.69	0.36	318
landmark_in_country	0.56	0.30	836
star_constellation	0.48	0.06	361
person_university	0.32	0.00	91
country_capital_city	0.29	0.42	24

Table 1: Object prominence, measured by the model’s own free generation and therefore independently of the lens, against how often the lens surfaces the gold object in its top five. Spearman

\rho=0.570

(

p=0.033

n=14

reference set	CEL	NEL	CEL	NEL
	median $\log_{10}$ rank of $m_{e}$		median rank of “French”
8 non-French	1.62	1.81	47	64
8 from other relations	1.88	1.72	102	70
4 French, 4 non-French	1.93	1.79	115	82
8 French	2.72	1.71	665	55

Table 2: The reference set’s effect on the two lenses, on person_native_language . All four reference sets are scored on the same 61 French subjects. Left: median

\log_{10}

rank of the model’s own answer

m_{e}

. Right: median rank of the token “French” itself. CEL degrades as the reference entities come to share the subject’s answer; NEL, which takes no difference, does not.

		subject term			controls
relation	$n$	own	foreign	NEL	anchor alone	wrong donor
country_capital_city	22	12	12	7	0	0
country_largest_city	22	12	8	4	0	0
person_plays_position	24	19	21	18	7	6
person_plays_pro_sport	24	22	21	21	6	3

Table 3: Recombining an LRE’s anchor with different subject terms,

\beta=3

. Counts are how often the decoded top-1 is the relation’s gold object. Own is the unmodified operator; foreign uses the paired relation’s subject term; NEL is the (relation-agnostic) noncontrastive entity lens; anchor alone drops the subject term; wrong donor uses another subject type’s NEL. The model answers all 22 countries correctly, but the operator itself reproduces only 12 (Appendix E ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Contrastive entity lens	Noncontrastive entity lens	Logit lens	Differential logit lens	Attribute rate
OLMo 3 7B	$\phantom{-}0.7355$	$\phantom{-}\mathbf{0.7650}$	$\phantom{-}0.2056$	$\phantom{-}0.4390$	$-0.4016$
GPT-J 6B ∗	$\phantom{-}0.4346$	$\phantom{-}0.4196$	$\phantom{-}\mathbf{0.4669}$	$\phantom{-}0.2020$	$-0.1035$
Mistral 7B	$\phantom{-}0.6561$	$\phantom{-}\mathbf{0.6711}$	$\phantom{-}0.6581$	$\phantom{-}0.5186$	$-0.2202$
Gemma 2 9B	$\phantom{-}\mathbf{0.6889}$	$\phantom{-}0.6868$	$\phantom{-}0.4728$	$\phantom{-}0.3436$	$-0.5515$
Qwen3 8B Base ∗	$\phantom{-}0.6647$	$\phantom{-}0.6232$	$\phantom{-}\mathbf{0.7908}$	$\phantom{-}0.5952$	$-0.0557$
Qwen3.8 27B ∗	$\phantom{-}0.5539$	$\phantom{-}\mathbf{0.5717}$	$\phantom{-}0.3404$	$\phantom{-}0.4492$	$-0.0099$

Appendix

Table 4: Entity lenses and readout baselines across seven models. Spearman correlation

\rho

with LRE faithfulness.

Marginal frequencies
Frequency measure	Spearman $\rho$	$p$ -value
Subject frequency, $c_{s}$	$\phantom{-}0.1380$	$0.484$
Object frequency, $c_{o}$	$\phantom{-}0.5712$	$0.001$
Co-occurrence
Joint frequency, $c_{so}$	$\phantom{-}0.4568$	$0.015$
Mean of $\log(1+c_{so})$	$\phantom{-}0.4191$	$0.026$

Appendix

Table 5: Frequency baselines for OLMo 3 7B. Correlations with LRE faithfulness using document counts from OLMo’s three published training mixes combined.

Qwen3 8B Base: prompt wording and prefix
Neutral prompt	$\rho$
{s} is known for	$\phantom{-}0.3430$
{s}	$\phantom{-}0.2479$
{s} is	$\phantom{-}0.2362$
{s} can be described as	$\phantom{-}0.4297$
Information about {s}:	$\phantom{-}0.6998$

Appendix

Table 6: Neutral-prompt variations. Spearman

\rho

between contrastive entity lens scores and LRE faithfulness, using the evaluation setup of Table 4 .

\dagger

: leading newline added to address high subject-state norms at position zero.

relation	$n$	own	foreign	NEL	currency	anchor alone	wrong donor
country_capital_city	22	12	12	7	6	0	0
country_largest_city	22	12	8	4	5	0	0
person_plays_position	24	19	21	18	–	7	6
person_plays_pro_sport	24	22	21	21	–	6	3

Appendix

Table 7: All arms,

\beta=3

, each subject read with the others as references. Currency : a subject term estimated under the official-currency prompt, which shares nothing with either country relation.

arm	capital (22)	largest (22)	position (24)	sport (24)
own	10.7 (10–14)	11.3 (10–12)	18.3 (17–20)	22.0 (21–23)
foreign	9.7 (9–11)	9.3 (8–10)	17.3 (15–21)	20.4 (16–23)
NEL	5.2 (1–8)	4.6 (4–5)	14.7 (7–20)	21.5 (16–24)
anchor alone	0.0 (0–1)	0.0 (0–0)	7.0 (7–7)	4.5 (1–7)

Appendix

Table 8: Mean (range) hits over 500 anchors, each estimated from eight references drawn from outside the test set; subject terms fixed.

relation	$k$	faith.	$A_{r}$
country_language ∗	1.7	1.00	$-1.22$
name_birthplace	3.9	0.97	$-1.32$
name_religion	6.2	0.97	$-1.40$
person_plays_pro_sport	63.6	0.95	$-1.23$
landmark_on_continent	189.4	0.92	$-2.43$
country_largest_city ∗	1.0	0.92	$-0.81$

Appendix

Table 9: All 28 relations, sorted by LRE faithfulness.

k

is the mean number of subjects per distinct object;

A_{r}

is the relation-agnostic lens score, higher meaning closer agreement with the model. † rescored for the numeric-object artifact (Appendix D ); ∗ fewer than ten test items, so faithfulness is coarsely quantized.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 776 papers

Relational Linear Properties in Language Models: An Empirical Investigation
May 21, 2026Giovanni Valer, Luigi Gresele, Marco Bronzini +1LLM Interpretability Factual Knowledge in Language Models
Fondazione Bruno Kessler, Trento, Italy · University of Bologna, Bologna, Italy · University of Pisa, Pisa, Italy +2
Knowledge That Is Linear Before Fine-Tuning Generalizes After It
Oct 15, 2026Faridoun Mehri, Hinrich SchützeLanguage Model Generalization Factual Knowledge in Language Models
Center for Information and Language Processing, LMU Munich Munich Center for Machine Learning (MCML)
Analysing the Linearity of Linguistic Relations in Language Model Embedding Spaces
Sep 18, 2026Vasudevan Nedumpozhimana, Fathima Thekkekara, John KelleherRepresentation Geometry in Language Models Word Embeddings
ADAPT Research Centre Trinity College Dublin, Ireland · Indian Institute of Technology Bombay Mumbai, India

May 21, 2026J/K move · Enter open · S save

Relational Linear Properties in Language Models: An Empirical Investigation

Giovanni Valer, Luigi Gresele, Marco Bronzini, Emanuele Marconato

Fondazione Bruno Kessler, Trento, Italy · University of Bologna, Bologna, Italy · University of Pisa, Pisa, Italy+2

LLM Interpretability Factual Knowledge in Language Models Language Model Probing

Open paper AskPDF arXiv

Linear properties are ubiquitous in the representations of language models; however, testing them experimentally remains a challenging task. This work focuses on relational linearity: the hypothesis that, for a fixed relation (e.g., "plays"), the unembedding of an object (e.g., "trumpet") can be predicted from the embedding of its subject (e.g.,"Miles Davis") by a linear map. We present an experimental method to test the formulation of relational linearity by Marconato et al. (2025). Specifically, we introduce a probing method, based on Kullback-Leibler divergence, to evaluate this property and examine its variation across layers and paraphrased relational queries. It is also more efficient than previous work; for example, it avoids the crude Jacobian approximations used in Linear Relational Embeddings by Hernandez et al. (2024). Our findings across four datasets show that relational linearity varies across models, exhibits layer-wise patterns consistent with prior observations about linguistic information in model representations, and is differently affected by changes in how the relation is phrased.

Abstract

LLM Interpretability

Explore similar work

Relational Linear Properties in Language Models: An Empirical Investigation

Knowledge That Is Linear Before Fine-Tuning Generalizes After It

Analysing the Linearity of Linguistic Relations in Language Model Embedding Spaces