The Lattice Representation Hypothesis of Large Language Models

Bo Xiong

cs.AIMar 1, 2026

The Lattice Representation Hypothesis of Large Language Models

Authors: Bo Xiong

Organizations: Stanford University, United States

Abstract

We propose the Lattice Representation Hypothesis of large language models: a symbolic backbone that grounds conceptual hierarchies and logical operations in embedding geometry. Our framework unifies the Linear Representation Hypothesis with Formal Concept Analysis (FCA), showing that linear attribute directions with separating thresholds induce a concept lattice via half-space intersections. This geometry enables symbolic reasoning through geometric meet (intersection) and join (union) operations, and admits a canonical form when attribute directions are linearly independent. Experiments on WordNet sub-hierarchies provide empirical evidence that LLM embeddings encode concept lattices and their logical structure, revealing a principled bridge between continuous geometry and symbolic abstraction. Datasets and code are open available at https://github.com/xiongbo010/lattice-representation-hypothesis.

Figures & tables

Figure 1: How LLMs encode conceptual structure. (a) Humans represent concepts as symbols and compose them using logical operators. (b) Under the standard extensional view, LLMs encode concepts as directions, where subsumption is interpreted through the relative orientation of vectors. (c) Under our intensional view, a concept is represented as the intersection of half-spaces defined by its attributes, and compositional semantics emerges through region intersection and union.

Figure 2: How FCA connects to the linear lattice geometry of LLMs. (a) A formal context describing which objects satisfy which attributes . (b) The discrete concept lattice constructed exactly from this formal context. (c) The corresponding lattice geometry encoded in LLM embeddings, where each attribute is represented as a linear direction and each object as a point, and concept composition (meet and join) emerges as intersection or union of half-spaces.

Figure 3: Distribution of projection lengths for positive and negative objects onto the directions of the first eight attributes (sorted alphabetically) in the WN-Animal dataset.

Model		WN-Animal			WN-Plant			WN-Food			WN-Event			WN-Cognition
Model		Pre.	Rec.	F1	Pre.	Rec.	F1	Pre.	Rec.	F1	Pre.	Rec.	F1	Pre.	Rec.	F1
LLaMA3.1-8B	Random	49.7	49.6	45.3	50.1	50.1	47.3	50.0	50.0	46.4	50.2	50.2	48.6	50.2	50.2	50.1
	Mean	63.7	69.3	63.7	63.7	67.2	63.3	67.2	71.4	68.1	63.7	65.1	63.9	68.4	68.4	68.4
	Linear	81.4	84.0	82.5	81.6	82.4	82.4	79.7	80.6	80.1	71.4	71.6	71.5	75.0	74.9	75.0
Gemma-7B	Random	49.8	49.7	45.3	50.1	50.1	47.3	50.1	50.1	46.3	49.4	49.3	47.8	50.1	50.1	50.1
	Mean	53.5	55.2	50.1	53.5	54.4	51.3	53.7	55.1	51.2	53.3	53.8	52.2	56.4	56.4	56.3

Table 1: Evaluation of recovering concept–attribute relations (formal context). Bold denotes the best across models, shading denotes the best within model. All metrics are percentages.

Model		WN-Animal			WN-Plant			WN-Food			WN-Event			WN-Cognition
Model		Pre.	Rec.	F1	Pre.	Rec.	F1	Pre.	Rec.	F1	Pre.	Rec.	F1	Pre.	Rec.	F1
LLaMA3.1-8B	Random	52.3	43.2	47.3	51.1	49.5	47.6	25.0	50.0	33.3	50.2	50.2	50.2	49.8	49.9	49.8
	Mean	68.4	65.0	66.7	64.2	63.3	63.8	61.5	58.5	55.7	60.3	57.9	59.1	58.2	55.4	56.8
	Linear	75.9	78.3	77.1	71.2	70.0	70.4	78.1	75.9	75.4	69.1	67.5	68.3	70.4	68.9	69.6
Gemma-7B	Random	54.0	51.2	50.6	52.7	50.3	49.5	53.4	50.8	39.1	49.8	50.1	49.9	49.4	49.7	49.5
	Mean	66.2	61.0	63.4	62.4	59.8	60.9	50.7	50.7	50.6	56.3	55.0	55.6	54.1	52.8	53.4

Table 2: Evaluation of partial order inference from projection-based concept representations. Bold denotes the best across models, shading denotes the best within model. All metrics are percentages.

Panel of Figure 4 — Figure 4: Quantitative evaluation (MRR) of concept algebra for Meet ( left ) and Join ( right ) operators.

A	B	A $\vee$ B	A $\wedge$ B
dog	wolf	predator, animal, canine, meat-eater, hunter, wild, mammal, quadruped, pet	dog, hound, puppy, terrier, mutt, beagle, retriever, spaniel, shepherd, pooch
cat	lion	predator, feline, animal, meat-eater, beast, carnivore, hunter, whiskers, mammal, wild	cat, kitten, tiger, panther, leopard, tomcat, feline, cheetah, tabby, lynx
sparrow	robin	avian, songbird, fowl, feathered, finch, beak, chirp, nest, perching, small	sparrow, robin, songbird, warbler, finch, canary, thrush, chickadee, pipit, titmouse
horse	zebra	equid, hoofed, animal, ungulate, mammal, quadruped, stallion, beast, herbivore	horse, pony, stallion, mare, foal, filly, mustang, gelding, thoroughbred, colt
carrot	parsnip	root, edible, produce, food, green, vegetable, crunchy, fresh, garden, plant	carrot, parsnip, radish, beet, turnip, tuber, rootcrop, veg, sprout, crop
eagle	falcon	animal, raptor, bird, predator, creature, wingspan, talon, flyer, sky, sharp-eyed	eagle, falcon, hawk, osprey, kestrel, buzzard, kite, harrier, condor, bird of prey

Table 3: Top-10 terms related to the join and meet of selected WordNet concept pairs.

Panel of Figure 5 — Figure 5: (a) PCA-based visualization of attribute directions in WN-Animal (top 20 most frequent attributes); (b) Performance of LLaMA-3 models of different sizes across WordNet domains.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Category	WN-Animal	WN-Plant	WN-Food	WN-Event	WN-Cognition
#Objects	7342	7704	2506	1009	2802
#Attributes	100	145	184	60	107
#Hypernyms	7473	8051	2628	1079	3003

Appendix

Table 4: Statistics of the five WordNet-derived datasets.

Figure 6: Correlation analysis of attribute directions in the WordNet-Animal dataset.

Symbolic Reasoning in Language Models

See all 18 papers

Abstract representational geometry supports inference in large language models
Jun 22, 2026Yunan Zeng, Yuwang WangLLM Inference LLM Interpretability
College of Future Information Technology, Fudan University, Shanghai, 200433, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China
Laguerre Geometry for Interpreting Large Language Models
Jul 12, 2026Chunwei Ma, Russell WolfingerTransformer Interpretability LLM Interpretability
JMP Statistical Discovery, Cary, NC, USA
Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Models
May 10, 2026Aojie Yuan, Zhiyuan SuLLM Interpretability Representation Geometry in Language Models
University of Southern California · 2Duke University

Jun 22, 2026J/K move · Enter open · S save

Abstract representational geometry supports inference in large language models

Yunan Zeng, Yuwang Wang

College of Future Information Technology, Fudan University, Shanghai, 200433, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China

LLM Inference LLM Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

A defining feature of human intelligence is the ability to adapt to changing environments by inferring latent task structure from sparse observations. Neuroscientific research indicates that this capability relies on the hippocampus constructing abstract representations, expressed as low-dimensional, approximately orthogonal manifolds in neural state space. However, the internal mechanisms of large language models (LLMs) remain largely opaque, making it unclear whether they form comparable abstract representations or instead rely on task-specific statistical regularities when performing comparable reasoning tasks. Here we adapt a contextual reversal-learning paradigm to a text-based setting and compare humans and LLMs at both the Behavioural and representational levels. We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus. Notably, this representational geometry is not uniformly distributed but is organized hierarchically across model depth: whereas lower layers show early, stable encoding of stimulus identity, higher layers form a hippocampal-like functional band enriched for abstract context geometry associated with inference. Furthermore, complementary intervention experiments mechanistically implicate geometry in reasoning: task-sequence language modelling induces geometric disentanglement, whereas geometric regularization of higher layers increases the emergence of generalizable inference. Together, these findings establish abstract representational geometry as a mechanistic principle supporting inference in large language models.

Abstract

Symbolic Reasoning in Language Models

Explore similar work

Abstract representational geometry supports inference in large language models

Laguerre Geometry for Interpreting Large Language Models

Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Models