Everyone Was Right: Lexical Representations in Language Models Are Both Abstract and Holistic

Zachary Nicholas Houghton; Yu Zhou; Dan Pluth; Vijay K. Gurbani; Vsevolod Kapatsinski

cs.CLOct 13, 2026

Everyone Was Right: Lexical Representations in Language Models Are Both Abstract and Holistic

Authors: Zachary Nicholas Houghton, Yu Zhou, Dan Pluth, Vijay K. Gurbani, Vsevolod Kapatsinski

Organizations: University of Oregon · Vail Systems, Inc.

Abstract

Lexical representations can encode both abstract information and information specific to particular contexts. We investigate how these forms of information are organized in language models by decomposing contextual word representations into item, class, and word-class interaction components. Across six pretrained transformers and two linguistic categories, we find that the interaction is consistently present, grows across layers, and contains information associated with both item and class. Unlike the nearly orthogonal item and class effects, the interaction overlaps with both and encodes class information beyond the shared class effect. An interaction also emerges in artificial-language learners even when the input contains only additive structure. Together, these findings show that lexical representations combine abstract information with structure specific to particular word-class combinations.

Figures & tables

Figure 1: Structure of our model. Every form is observed 24 times at each class, each time in a different context; that variation across contexts is what the measurement later splits in two.

Figure 2: The four item-by-class means formed by two items in two classes. The arrow is

\gamma(i,c)

Figure 3: Ground-truth effect size of each component in the language (x) against its size in the MLP’s representations (y). Points are runs; lines are fitted values from the size model (Equation 3 ) with 95% intervals, colored by hidden width.

Figure 4: Fraction of the representation (ignoring context) accounted for by each component, summing to 1, against relative depth from the embedding layer (0) to the final layer (1). Each line is one model; color is the training corpus, linetype the hidden width, shading the 95% interval.

model	accuracy	null
part of speech
BabyLM 1.3B	0.90 [0.88, 0.91]	0.51 [0.49, 0.54]
Pythia 1.4B	0.94 [0.92, 0.95]	0.50 [0.48, 0.53]
grammatical role
BabyLM 1.3B	0.80 [0.77, 0.83]	0.50 [0.47, 0.53]
Pythia 1.4B	0.84 [0.81, 0.86]	0.52 [0.49, 0.55]

Table 1: Accuracy of recovering the class from the interaction, at the deepest layer of the largest model in each family, compared to the null; brackets are bootstrapped intervals.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

factor	levels
language
item weight	0, 0.5, 1, 2
class weight	0, 0.5, 1, 2
interaction weight	0, 0.5, 1, 2
context weight	0.5, 2, 6
interaction rank	4, 16, 64, 177

Appendix

Table 2: The Experiment 1 language parameters. The first five rows define each artificial language, the next two the learner MLP model. Item and class ranks are held fixed so that the interaction’s rank is the only one that varies. Crossing every level gives 38,400 cells; the three languages in which all three effect weights are zero contain no item-by-class structure at all and are excluded, leaving the reported total.

Figure 5: Fit and recovery against hidden width, for languages whose planted structure is more than 30% interaction. Left: Kullback-Leibler divergence from the learner to the ground-truth distribution, so zero is a perfect fit. Right: the interaction share measured in the hidden layer as a fraction of the share planted in the language, so one would be full recovery and the plotted values are the fraction of it reached. Lines are medians over runs with interquartile bands, which show how much the languages in the grid differ from one another rather than measurement error.

Figure 6: Each word in each class is observed in many contexts, and those observations are averaged to give that word’s representation in that class. Every one of these averages is a vector in

\mathbb{R}^{d}

, and every item appears in every class.

Figure 7: The three components across the grid of words and classes. An item effect is one value per word, the same in every class; a class effect is one value per class, the same for every word; and the interaction differs for every word in every class.

specified	runs	measured	flagged
0.000	6300	0.0000 [0.0000, 0.0094]	5%
0.030	900	0.0323 [0.0199, 0.0418]	100%
0.048	1800	0.0495 [0.0385, 0.0688]	100%
0.056	1800	0.0573 [0.0458, 0.0918]	100%
0.059	1800	0.0607 [0.0495, 0.1041]	100%
0.100	1800	0.1007 [0.0911, 0.1089]	100%

Appendix

Table 3: Recovery of the interaction at 24 observations per item and class in 768 or more dimensions, pooled over noise levels, item counts, and widths. Specified is the interaction share built into the generated representations and measured the median share recovered, with the 2.5th and 97.5th percentiles across runs in brackets; flagged is the share of runs whose re-split interval excludes zero.

effect	observations	median	95th pct	flagged
class effect	10	0.00000	0.0081	6.1%
class effect	24	0.00000	0.0034	6.5%
class effect	100	0.00000	0.0009	6.3%
interaction	10	0.00009	0.0644	6.2%
interaction	24	0.00004	0.0309	5.9%
interaction	100	0.00002	0.0083	4.9%

Appendix

Table 4: What the measure reports when an effect is set to exactly zero, by observations per item and class, pooled over every other axis of the validation grid. Median and 95th pct summarize the measured share of that effect across runs, and flagged is the share of runs whose re-split interval excludes zero, which should sit near the nominal 5%.

width d	item	class	interaction
8	3%	14%	7%
16	7%	3%	5%
32	4%	4%	10%
128	6%	4%	5%
768	2%	8%	3%
1024	3%	5%	3%

Appendix

Table 5: How often a component set to exactly zero is flagged, by width and by effect.

width d	measured	analytic chance
8	0.17501 [0.06900, 0.40018]	0.25000
16	0.10678 [0.03744, 0.20884]	0.12500
32	0.05609 [0.01864, 0.10454]	0.06250
128	0.01714 [0.00450, 0.02611]	0.02344
768	0.00261 [0.00079, 0.00472]	0.00391
1024	0.00217 [0.00065, 0.00330]	0.00293

Appendix

Table 6: Item-class overlap when the two effects are constructed orthogonal, at 24 observations per item and class. Measured is the median overlap across the runs at that width with the 2.5th and 97.5th percentiles in brackets; chance is the overlap expected if the component were randomly oriented,

r/d

for a target subspace of

r

dimensions in a representation of width

d

width d	measured	chance	detected
8	0.1688 [0.0824, 0.3998]	0.12500	9%
16	0.0951 [0.0414, 0.2026]	0.06250	22%
32	0.0593 [0.0225, 0.1119]	0.03125	36%
128	0.0213 [0.0062, 0.0517]	0.00781	63%
768	0.0065 [0.0011, 0.0298]	0.00260	83%
1024	0.0056 [0.0008, 0.0331]	0.00098	82%

Appendix

Table 7: Measured item-class overlap when a fraction of the class direction is placed inside the item subspace (non-orthogonal), at 24 observations per item and class. Measured is the median overlap across the runs at that width with the 2.5th and 97.5th percentiles in brackets; chance is the overlap expected if the component were randomly oriented,

r/d

for a target subspace of

r

dimensions in a representation of width

d

; detected is the fraction of runs whose overlap falls outside that null’s 95% interval.

specified	noise	runs	measured	flagged
	1x	6300	0.0000	6%
	10x	6300	0.0000	6%
0.00	50x	6300	0.0001	6%
	1x	1800	0.1012	100%
	10x	1800	0.1002	100%
0.10	50x	1800	0.1001	100%

Appendix

Table 8: Recovery of the interaction across simulations with different amounts of noise, in 768 or more dimensions, pooled over item counts, widths and sample sizes. Specified is the interaction share built into the representations and noise the variance across contexts per observation as a multiple of the variance between words and classes, so at 1x the two are equal. Measured is the median interaction share recovered across runs and flagged the share whose re-split interval excludes zero, which should sit at 5% in the rows where nothing was planted.

property	level	runs	item	class	interaction
	under 30%	3600	0.490	0.399	0.028
	30 to 70%	2871	0.538	0.371	0.008
context in the language	over 70%	3886	0.578	0.350	0.004
	item < class	3525	0.428	0.480	0.017
	item = class	3427	0.569	0.359	0.015
item against class	item > class	3405	0.644	0.294	0.011

Appendix

Table 9: How two other properties of the artificial language move the composition of the item-by-class structure, under ReLU, pooled over widths. Rows within a property partition the same runs. Values are medians of the three shares over the runs in the row.

component	term	estimate	error	95% CI	P(b > 0)
item	item	1.68	0.04	[1.61, 1.75]	1.00
class	item	-1.63	0.04	[-1.70, -1.56]	0.00
interaction	item	0.09	0.04	[0.02, 0.16]	0.99
item	class	-2.67	0.04	[-2.75, -2.59]	0.00
class	class	2.77	0.04	[2.69, 2.85]	1.00
interaction	class	0.24	0.04	[0.16, 0.31]	1.00

Appendix

Table 10: Coefficient values of the size model (Equation 3 ) with error and 95% CIs, ReLU arm.

component	term	estimate	error	95% CI	P(b > 0)
item	item	2.36	0.04	[2.29, 2.43]	1.00
class	item	-2.29	0.03	[-2.35, -2.22]	0.00
interaction	item	0.48	0.04	[0.40, 0.55]	1.00
item	class	-3.52	0.04	[-3.60, -3.45]	0.00
class	class	3.66	0.04	[3.58, 3.74]	1.00
interaction	class	0.08	0.04	[-0.00, 0.16]	0.97

Appendix

Table 11: Coefficient values of the size model (Equation 3 ) with error and 95% CIs, linear arm.

Figure 8: Coefficient estimates from the size model (Equation 3 ), one panel per measured component. The upper block holds the planted effect sizes and the lower the architecture terms, which are on a much smaller scale and so are drawn on their own axis. Points are posterior medians with 95% intervals.

measure	term	estimate	error	95% CI	P(b > 0)
item_class	item	-0.51	0.04	[-0.59, -0.42]	0.00
int_margins	item	-0.96	0.04	[-1.05, -0.88]	0.00
item_class	class	-1.62	0.05	[-1.71, -1.53]	0.00
int_margins	class	-0.62	0.05	[-0.71, -0.53]	0.00
item_class	interaction	0.31	0.04	[0.23, 0.39]	1.00
int_margins	interaction	-0.57	0.04	[-0.65, -0.49]	0.00

Appendix

Table 12: Every coefficient of the overlap model (Equation 3 ), with Estimate, Estimated error, and 95% CIs.

Figure 9: Model results for the size model (Equation 3 ). Columns are the three measured components, rows are the planted interaction rank, and lines are hidden widths, with 95% intervals. Holding rank fixed within a row is what separates the width effect from the width-by-rank term; the near-identical rows show how little the rank changes it.

Figure 10: Two overlap measurements as a function of width of the hidden layer. Left: the item effect projected onto the class direction. Right: the interaction projected onto the span of both margins. The dashed line is chance.

Figure 11: Fraction of the representation by each component (such that they sum to 1), against hidden width, with one line per rank of the interaction planted in the language and 95% intervals.

class	item types	median tokens
part of speech (noun / verb)	78	18 / 18
grammatical role (subject / object)	49	15 / 19

Appendix

Table 13: Stimuli for the two linguistic categories, with the two classes of each in parentheses. Item types is how many word forms occur in both of them at least ten times; forms occurring in only one are excluded. Median tokens is how often a typical form occurs in each of its two classes, in the order they are listed.

family	model	params	width	layers	training data	tokens
	opt-babylm-125m	125M	768	12	BabyLM (child-available)	~3B
	opt-babylm-350m	350M	1024	24	BabyLM (child-available)	~3B
BabyLM	opt-babylm-1.3B	1.3B	2048	24	BabyLM (child-available)	~3B
	pythia-160m	160M	768	12	the Pile	~300B
	pythia-410m	410M	1024	24	the Pile	~300B
Pythia	pythia-1.4b	1.4B	2048	24	the Pile	~300B

Appendix

Table 14: The six models, paired by width. Training-token counts are approximate; the OPT-BabyLM models train for 20 epochs over a 100M-word corpus, which is roughly 150M tokens, so their effective token exposure is about twenty times that.

	interaction		class effect		item-class overlap
model	std	raw	std	raw	std	raw
BabyLM 125M	0.095	0.095	0.072	0.074	0.0052	0.0053
BabyLM 350M	0.122	0.122	0.066	0.068	0.0056	0.0061
BabyLM 1.3B	0.104	0.104	0.050	0.051	0.0035	0.0036
Pythia 160M	0.231	0.228	0.078	0.089	0.0058	0.0081
Pythia 410M	0.244	0.242	0.098	0.123	0.0067	0.0195

Appendix

Table 15: Part of speech at the deepest layer, measured with and without the per-dimension rescaling. Sizes are shares of the item-by-class structure; overlaps are the raw fractions, not multiples of chance.

class	model	item-class overlap	its null	interaction overlap	its null
	BabyLM 125M	0.0052 (4.2×)	0.0012	0.178 (3.5×)	0.051
	BabyLM 350M	0.0056 (5.9×)	0.0010	0.166 (4.3×)	0.039
	BabyLM 1.3B	0.0035 (7.6×)	0.0005	0.106 (4.8×)	0.022
	Pythia 160M	0.0058 (4.6×)	0.0013	0.231 (4.2×)	0.055
	Pythia 410M	0.0067 (6.9×)	0.0010	0.239 (6.1×)	0.039
part of speech	Pythia 1.4B	0.0055 (11.3×)	0.0005	0.209 (9.7×)	0.021

Appendix

Table 16: Overlap measurement and the null values at the final layer of each model. Each overlap is the fraction of one component lying inside another’s subspace: the item effect inside the class subspace, the interaction inside the span of the two main effects. Each is followed by the overlap divided by its own null, so 4.2x means the overlap is 4.2 times what random orientation gives.

		item effect		class effect		interaction
class	model	hidden	output	hidden	output	hidden	output
	BabyLM 125M	0.832	0.751	0.069	0.142	0.099	0.107
	BabyLM 350M	0.810	0.721	0.064	0.146	0.126	0.134
	BabyLM 1.3B	0.843	0.757	0.049	0.123	0.108	0.120
	Pythia 160M	0.688	0.577	0.077	0.143	0.235	0.280
	Pythia 410M	0.656	0.619	0.095	0.121	0.248	0.260

Appendix

Table 17: Each component’s share of the item-by-class structure at the final hidden layer and after the unembedding, measured on identical tokens. Both members of a pair are point estimates from one run, which is what makes them comparable to each other; they therefore sit about 0.003 from the re-split medians quoted elsewhere.

class	model	word	class effect	interaction
	BabyLM 125M	36.3	3.1	4.2
	BabyLM 350M	31.2	2.5	4.7
	BabyLM 1.3B	25.8	1.5	3.2
	Pythia 160M	22.4	2.5	7.5
	Pythia 410M	20.3	3.0	7.6
part of speech	Pythia 1.4B	18.8	2.4	6.8

Appendix

Table 18: How much of a token’s hidden state each component accounts for, at the deepest layer, as the percentage reduction in squared error against guessing the grand mean.

POS (noun/verb)
model	word-built	context	null
BabyLM 125M	0.85 [0.82, 0.88]	0.15	-0.00
Pythia 160M	0.86 [0.83, 0.87]	0.14	-0.00
BabyLM 350M	0.89 [0.86, 0.91]	0.11	-0.00
Pythia 410M	0.81 [0.77, 0.83]	0.19	-0.00
BabyLM 1.3B	0.89 [0.87, 0.91]	0.11	-0.00

Appendix

Table 19: How much of the deepest-layer interaction is attributable to the left context, by model and category. Word-built is the share requiring word-dependent processing, with its bootstrap interval; context is the share an item-independent carry-over of the context predicts; null mismatches items before the comparison.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 506 papers

Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language
May 23, 2026Mathis Immertreu, Achim Schilling, Thomas Kinfe +1Predictive Representation Learning Representation Geometry in Language Models
Cognitive Computational Neuroscience Group, Friedrich-Alexander-Universität Erlangen–Nürnberg (FAU), Germany. · Mannheim Center for Neuromodulation and Neuroprosthetics (MCNN), University Hospital Mannheim, University Heidelberg, Germany.
Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning
Aug 1, 2026Zachary Nicholas Houghton, Vsevolod KapatsinskiMemorization in Language Models Language Modeling
University of Oregon · Vail Systems, Inc.
On the Persistent Effects of Lexicality in Large Language Models
Jun 1, 2026Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani +3LLM Interpretability Language Modeling
Dalhousie University Halifax, NS · University of Kentucky Lexington, KY · Carnegie Mellon University Pittsburgh, PA

May 23, 2026J/K move · Enter open · S save

Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language

Mathis Immertreu, Achim Schilling, Thomas Kinfe, Patrick Krauss

Cognitive Computational Neuroscience Group, Friedrich-Alexander-Universität Erlangen–Nürnberg (FAU), Germany. · Mannheim Center for Neuromodulation and Neuroprosthetics (MCNN), University Hospital Mannheim, University Heidelberg, Germany.

Predictive Representation Learning Representation Geometry in Language Models Successor Representations

Open paper AskPDF arXiv

Neural language models are typically trained on next-token prediction, although linguistic structure spans multiple temporal scales. Successor representations (SRs) make this horizon explicit by encoding discounted distributions over future states. Here, we ask whether such predictive representations can recover not only word classes, but also finer functional and construction-like structure from natural language. A residual network trained on WikiText-103 predicts SR distributions at three horizons without part-of-speech supervision. At the shortest horizon, unsupervised clustering robustly recovers nouns, verbs, and adjectives, while directed inter-cluster transitions reproduce familiar syntactic asymmetries. At finer resolutions and across 13 part-of-speech categories, the same geometry reveals semantic-functional groupings that cross category boundaries and directed relations tracing candidate date, measurement, and title-name constructions. Part-of-speech agreement declines as the predictive horizon lengthens. These results suggest that word classes are coarse regions within a richer predictive geometry in which categorical and construction-like linguistic structure emerge from future-word distributions.

Abstract

Language Modeling

Explore similar work

Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

On the Persistent Effects of Lexicality in Large Language Models