World Properties without World Models: Distributional Associations and the Interpretation of Decoding Results from Language Models

Elan Barenholtz

cs.CLMar 4, 2026

World Properties without World Models: Distributional Associations and the Interpretation of Decoding Results from Language Models

Authors: Elan Barenholtz

Organizations: Department of Psychology & Center for Complex Systems and Brain Sciences Florida Atlantic University

Abstract

A growing literature shows that variables can be linearly decoded from the activations of large language models (LLMs). These range from properties of the world, such as the locations of cities and the lifetimes of historical figures, to emotions and pain. Such findings are often taken as evidence that language models go beyond surface text statistics and form internal models of the world. We show that static word embeddings (fixed, context-insensitive representations learned from corpus statistics) of the same or matched stimuli support much of the same decoding. Across four published cases (place, time, pain and emotion), static vectors predict coordinates and year of death (R^2 = 0.42-0.59), separate pain from matched control sentences (held-out AUC 0.85-0.88), and classify twelve emotions in stories written to avoid naming them (AUC 0.84-0.88). Because static embeddings assign each word a single, context-independent vector, these results are a lower bound on what word associations alone can support. The LLMs retain clear advantages on representational tests, and causal and behavioral findings remain outside the scope of the baseline. On the original authors' entities, where we reproduce their Llama-2 results, the transformer's advantage lies mostly in placing historical figures in the right century and places in the right country, coarse sorting that richer word associations would be expected to improve; within those groups every representation orders items poorly. Static vectors for disambiguated Wikipedia entities, which carry the associations of a particular place or person rather than of the words in its name, close most of the remaining gap, matching Pythia-2.8B on coordinates and Llama-2-7B on year of death. These results indicate that decodability alone cannot distinguish a representation of a property from information already available in fixed distributional associations.

Figures & tables

Figure 1 : Held-out places from Gurnee and Tegmark (2024) that have a Wikipedia2Vec entity vector ( $n=5{,}984$ ), at their true locations and as decoded by linear probes on Wikipedia2Vec’s word vectors averaged over the name, on its static entity vectors, and on Llama-2-7B (layer 24), as in Gurnee and Tegmark (2024) ’s Figure 1 (which shows Llama-2-70B); color marks the true continent. Panel titles give $R^{2}$ for latitude / longitude over all places; four places in Antarctica are not drawn. The entity vectors recover a blurred version of the layout that Llama-2 recovers more precisely (median error 2,450 km against 760 km; Table 2 ).

	Places (latitude / longitude)			Figures (year of death)
		Within-group $r$			Within-group $r$
Representation	$R^{2}$	Continent	Country	$R^{2}$	Century	Decade
GloVe	.47 / .53	.33 / .14	.06 / .05	.50	.05	.00
Word2Vec	.54 / .57	.39 / .18	.08 / .06	.42	.05	.01
Pythia-2.8B, layer 0	.50 / .57	.36 / .15	.11 / .07	.34	.06	.00
Pythia-2.8B, selected	.76 / .79	.64 / .33	.20 / .12	.70	.10	.03

Table 1 : Transformers and static embeddings on the entities released by Gurnee and Tegmark (2024) , using their train–test split and scoring all representations on the held-out items covered by GloVe (

n=6{,}690

places,

7{,}316

figures); static values therefore differ slightly from Table 8 . Within-group

r

correlates predictions with targets after removing group means in the test set. “Input mean” averages the model’s input embeddings over all tokens of the name. “Selected” is the layer chosen on validation data within the training partition (Section 3.5 ).

	Places			Figures
Representation	$R^{2}$ (lat / lon)	Within $r$	Median error	$R^{2}$	Within $r$	Median error
GloVe, word average	.48 / .55	.07 / .05	3,480	.54	.04	175
fastText, word average	.59 / .62	.12 / .08	3,038	.62	.07	160
Wikipedia2Vec, word average	.61 / .66	.13 / .09	2,879	.65	.07	150
Wikipedia2Vec, entity	.79 / .79	.25 / .11	2,448	.80	.14	109
Pythia-2.8B	.77 / .81	.21 / .13	1,192	.74	.10	124

Table 2 : Word-level and entity-level static vectors against the transformers on the released entities that have a Wikipedia2Vec entity vector (authors’ split;

n_{\text{test}}=5{,}984

places and

4{,}707

figures). Within-group

r

is within country for places and within century for figures; the median error is in kilometers or years. Transformers at the selected layer.

Figure 2 : Words whose GloVe vectors are most similar to place names that lie farther south, north, west or east. For each of 15,923 words (the 20,000 most frequent GloVe words with at least four letters, excluding a list of country names, demonyms and geographic terms and any word that is part of at least three place names), we correlate the word’s cosine similarity to the averaged name vector of each of the 33,500 released places that have a GloVe vector with the place’s latitude or longitude; shown are the twelve strongest correlations at each end. The words are selected for extreme correlations from the whole vocabulary, so the figure is descriptive and implies no significance.

	Held-out AUC				$z$ on S2 direction
Representation	S2	S1	S2 vs. Arousal	S2 vs. Random	Numb	Sadness
Keyword baseline	$0.727$	$0.669$	—	—	—	—
GloVe	$0.850\pm 0.009$	$0.773\pm 0.011$	$0.664$	$0.815$	$-0.07$	$-0.09$
Word2Vec	$0.867\pm 0.007$	$0.802\pm 0.012$	$0.693$	$0.857$	$-0.38$	$-0.48$
fastText	$0.878\pm 0.006$	$0.785\pm 0.013$	$0.712$	$0.883$	$-0.67$	$-0.50$
25 LLMs	$0.91$ – $1.00$	$0.85$ – $0.94$	all separate a	all separate a	$-0.4$ – $+0.3$	—

Table 3 : The pain-axis tests on static word embeddings, using the v2 stimuli of Tagliabue et al. (2026) . Held-out AUC for pain against the S2 or S1 controls and, for the S2 direction, the standalone Arousal and Random sets (mean

\pm

s.d. over 20 fold assignments);

z

: mean projection of the standalone Numb and Sadness sets on the S2 direction, referenced to the S2 sentences (pain

\approx+0.7

, controls

\approx-0.7

). LLM values are the authors’ held-out estimates; the keyword baseline is from pain-axis-review contributors (2026) .

	Replication stories (12 emotions)		GoEmotions (27 emotions)
	AUC	12-way acc.	all text	emotion words removed
GloVe, story direction	$0.838$ [.82, .86]	$0.415$ [.36, .48]	$0.791$	$0.783$
GloVe, word direction	$0.752$ [.72, .78]	$0.277$ [.24, .32]	—	—
Word2Vec, story direction	$0.873$ [.85, .89]	$0.484$ [.41, .55]	$0.804$	$0.794$
Word2Vec, word direction	$0.779$ [.75, .81]	$0.330$ [.27, .39]	—	—
fastText, story direction	$0.876$ [.85, .90]	$0.511$ [.45, .57]	$0.818$	$0.811$

Table 4 : Emotion directions on averaged static word vectors. Stories come from an independent replication corpus ( Ogunlana, 2026 ) generated under the prompt of Sofroniew et al. (2026) ; topics are held out, and brackets give 95% intervals from resampling held-out topics. The word direction uses the emotion label’s static vector and no stories. GoEmotions ( Demszky et al., 2020 ) has 27 emotions in human-written text; the last column excludes every comment containing an emotion word or synonym.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

		Places		Figures
Model	Condition	$R^{2}$	Within-country $r$	$R^{2}$	Within-century $r$
Pythia-2.8B	Full activations	.76 / .79	.20 / .12	.70	.10
	Group removed	.01 / .00	.47 / .49	.00	.09
	Group + static removed	.00 / .00	.37 / .34	.00	.00
	Group + random removed	.01 / .00	.46 / .49	.00	.09
	Full + static appended	.76 / .79	.21 / .13	.71	.10

Appendix

Table 5 : Removing group membership from the activations (released entities, training-selected layers; Appendix A.3 ).

R^{2}

for latitude / longitude and year of death, and within-country (places) or within-century (figures)

r

of a probe refitted after each removal. The last row gives the within-group

r

after removing the static component beyond group membership minus that after the matched random removal, with 95% intervals from 1,000 resamples of test-set countries (273) or centuries (29).

Places, continent held out (median error, lat. / long.)
	GloVe	Word2Vec	Pythia-2.8B	Llama-2-7B
Africa	24.7 / 32.9	23.6 / 28.4	17.5 / 25.8	19.5 / 16.9
Americas	19.1 / 141.9	16.0 / 144.9	18.9 / 142.3	26.6 / 152.5
Asia	12.0 / 76.9	9.7 / 90.2	9.0 / 63.2	6.5 / 70.3
Europe	37.8 / 38.7	38.1 / 32.4	39.7 / 26.0	34.4 / 23.6
Oceania	61.2 / 168.2	59.6 / 176.1	45.4 / 181.0	54.4 / 147.8

Appendix

Table 6 : Grouped holdouts on the released entities (transformers at the fixed layer 19). Median absolute error for each held-out group (degrees of latitude / longitude; years), the pooled

R^{2}

over all held-out predictions, and, for comparison,

R^{2}

and median error under random five-fold splits of the same items.

	Places (lat / lon)		Figures (year)
Representation	With entity vector	Without	With	Without
GloVe, word average	.48 / .55	.34 / .28	.53	.40
fastText, word average	.59 / .62	.43 / .34	.61	.47
Pythia-2.8B	.77 / .81	.59 / .57	.73	.60
Llama-2-7B	.89 / .90	.70 / .71	.82	.68

Appendix

Table 7 :

R^{2}

on held-out items with and without a Wikipedia2Vec entity vector, for probes trained on all training items (authors’ split, GloVe-covered items; transformers at the selected layer). Test items: 5,995 covered and 695 uncovered places, 4,710 covered and 2,606 uncovered figures. Counts differ slightly from Table 2 , which also requires a Wikipedia2Vec word vector.

	World places ( $n_{\text{test}}\approx 6{,}900$ )			Historical figures ( $n_{\text{test}}\approx 7{,}300$ )
Representation	Latitude	Longitude	Both	Death year	Coverage
GloVe	$0.469$	$0.526$	$0.497$	$0.496$	85% / 98%
Word2Vec	$0.547$	$0.573$	$0.560$	$0.421$	88% / 97%
fastText	$0.586$	$0.594$	$0.590$	$0.574$	92% / 98%
Llama-2-7B	—	—	$0.881$	$0.785$	—
Llama-2-13B	—	—	$0.896$	$0.804$	—

Appendix

Table 8 : The type-level baseline on the entity datasets released by Gurnee and Tegmark (2024) , using their designated train–test split and their metric (out-of-sample

R^{2}

; for places, the mean over the two coordinates, as in their evaluation code). LLM values are their reported figures for linear probes at 60% layer depth (their Table 2). Coverage is the share of entity names with at least one in-vocabulary token; each representation is scored on the test items it covers.

	S2 $\times$ fear			S2 $\times$ negative emotion			S2 $\times$ sadness
Representation	Original	Neutral	Controls	Original	Neutral	Controls	Original
GloVe	$+0.05$	$+0.61$	$+0.11$	$+0.23$	$+0.76$	$+0.08$	$+0.19$
Word2Vec	$+0.03$	$+0.54$	$+0.13$	$+0.19$	$+0.71$	$+0.10$	$+0.10$
fastText	$-0.01$	$+0.61$	$+0.03$	$+0.14$	$+0.71$	$+0.06$	$+0.11$
LLMs (mean)	$+0.12$	$+0.58$	$+0.08$	$+0.21$	$+0.77$	$+0.14$	$+0.38$

Appendix

Table 9 : Cosine of the S2 pain direction with other directions on static word embeddings, under the original construction and the two common baselines of v2 ( Tagliabue et al., 2026 ) . LLM values are the means reported in v2 (23–25 models).

Figure 3 : The representational tests reported for a “pain axis” in 25 LLMs ( Tagliabue et al., 2026 ) , run on averaged static word vectors of the authors’ own sentences. A : held-out separation of pain from controls (S2, first person; five-fold cross-validation over sentence sets, mean $\pm$ s.d. over 20 fold assignments). The keyword baseline is the best of several surface-lexical classifiers reported by an independent review ( pain-axis-review contributors, 2026 ) . LLM values are each model’s best layer. B : cosine of the pain direction with fear, negative-emotion and negative-world-state directions; bars span the LLMs’ per-model range.

Category	Words	$k$	Var.	Drop (null)	$z$	Drop (null)	$z$
				Latitude		Longitude
Cardinal directions	16	9	0.07	.013 (.007)	1.3	.005 (.013)	$-$ 1.0
Climate and weather	27	19	0.11	.019 (.008) ∗	3.0	.004 (.008)	$-$ 1.2
Region and continent	28	18	0.14	.053 (.022) ∗	2.6	.037 (.032)	0.3
Country names	68	20	0.17	.095 (.032) ∗∗	5.9	.091 (.055) ∗	2.6
Economic terms	27	19	0.11	.008 (.008)	0.0	.006 (.009)	$-$ 0.5

Appendix

Table 10 : Drop in test

R^{2}

when the embedding subspace spanned by a word category is projected out of the GloVe name vectors of the released places, with the mean drop for 200 matched random word sets in parentheses and

z

against that null.

k

is the number of dimensions projected out and Var. the share of the place vectors’ variance they hold. ∗

p\leq.05

and ∗∗

p\leq.01

, one-sided empirical

p

over the 200 matched sets.

Explore similar work

Where Do Models Find Happiness? Emotion Vectors in Open-Source LLMs
Jun 25, 2026Sinie van der Ben, Raphaël Baur, Yannick Metz +1Valence-Arousal Estimation Emotion
Department of Computer Science, ETH, Zurich, Switzerland.
LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations
Sep 3, 2025Daniela Gottesman, Alon Gilae-Dotan, Ido Cohen +4Large Language Model Pretraining Token Co-Occurrence Graphs
Tel Aviv University · Mila – Quebec AI Institute & McGill University
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
Dec 14, 2024Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9Single-Token Output Distributions Distributional Information
Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3

Jun 25, 2026, cs.CLJ/K move · Enter open · S save

Where Do Models Find Happiness? Emotion Vectors in Open-Source LLMs

Sinie van der Ben, Raphaël Baur, Yannick Metz, Mennatallah El-Assady

Department of Computer Science, ETH, Zurich, Switzerland.

Valence-Arousal Estimation Emotion Text Corpora Poetry

Open paper AskPDF arXiv

Recent work identified emotion vectors in Claude Sonnet 4.5, which are internal representations that encode emotion concepts, causally influence behavior, and exhibit geometry mirroring human psychological structure. We test the generality of these findings in two open-weight models, Apertus-8B-Instruct-2509 and Gemma-4-E4B-it, extracting emotion contrast vectors across all layers, using two model-generated corpora. We recover valence geometry for both models, with peak PC1--valence correlations of

r = 0.76

and

r = 0.83

, approaching the

r = 0.81

reported for Claude.Beyond replication, we observe notable differences in how valence representations emerge across model depth. In Gemma-4-E4B-it, valence is strongly encoded in early layers but collapses towards later layers, whereas Apertus-8B-Instruct-2509 exhibits the opposite pattern, with valence representations absent in early layers, but emerging at mid depths. Arousal encoding, in contrast, is sensitive to the extraction corpus: both models show stronger PC2--arousal alignment with Gemma-generated stories (

r

up to

0.45

) than Apertus-generated ones (

r \leq 0.21

), suggesting arousal-relevant cues are unevenly distributed across generated corpora. We open-source our experiment code and dataset for reproducible investigation of emotion representations across language model architectures.

Abstract

Explore similar work

Where Do Models Find Happiness? Emotion Vectors in Open-Source LLMs

LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations

Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning