The Geometry of Knowledge Accessibility in Large Language Models

Lihu Chen

cs.CLOct 2, 2026

The Geometry of Knowledge Accessibility in Large Language Models

Authors: Lihu Chen

Organizations: Imperial College London

Abstract

Large language models (LLMs) contain broad knowledge, but they cannot access all of it reliably. We study this problem through knowledge accessibility, which describes whether the knowledge needed for a query can be recalled from the model. We find that knowledge accessibility has a simple geometric structure in the model's representation of the query alone, before any generation. More accessible queries are closer to a center in the representation space, while less accessible queries are farther away. This geometry reveals a knowledge boundary that separates more accessible queries from less accessible ones. Accessibility consistently decreases with distance from the center, and this distance-based ordering transfers across datasets even when the centers differ. Controlled experiments further show that the centered geometry is more closely related to knowledge accessibility than to reasoning difficulty. The geometry also reveals when different interventions are useful. Query rewriting helps more for accessible queries, chain-of-thought reasoning helps more near the boundary, and retrieval gives larger gains beyond the boundary. These findings not only provide a new geometric view of how knowledge is organized in language models, but also suggest a useful pre-generation signal for adaptive inference.

Figures & tables

Table 1

Panel of Figure 1 — Figure 1: Layer-wise comparison of different geometry hypotheses across four datasets and two language models. Peak performance is reported in the legend.

Figure 2: AUROC on TriviaQA with Llama-8B.

Group	K / R	$n$	Dist.	Position
Success	✓ / ✓	312	7.85	Interior
Reasoning Failure	✓ / ✗	311	5.33	Interior
Knowledge Failure	✗ / ✗	281	1.05	Exterior
Recovered Success	✗ / ✓	96	5.29	Interior

Table 3: MQuAKE groups by absolute distance to the knowledge boundary, where K and R denote knowledge accessibility ( single-hop ) and reasoning ( multi-hop ) success, respectively.

Panel of Figure 3 — Figure 3: Behavior of knowledge accessibility for Llama-8B on Trivia QA. (a) Accessibility decreases with distance from the center. (b) Answerability is least stable near the boundary. (c) Consistency decreases with distance from the center. Results for Qwen14B are provided in B.2 in Appendix.

Figure 4: Transferability of the distance and accuracy relation for Llama-8B. Accuracy generally decreases with distance across target datasets. Results for Qwen-14B are provided in Figure B.3 in Appendix.

Panel of Figure 5 — Figure 5: Intervention effects across the knowledge boundary on TriviaQA.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1: Learned norm coefficient

\alpha

in Linear+Norm across layers on TriviaQA with Llama-8B. The line shows the mean over five random seeds and the shaded region shows

\pm 1

standard deviation.

Panel of Figure 9 — Figure B.2: Behavior of knowledge accessibility for Qwen-14B on TriviaQA (counterpart of Figure 3 ). (a) Accessibility decreases with distance from the center. (b) Correctness is least stable near the boundary. (c) Consistency decreases with distance from the center.

Figure B.3: Transferability of the distance and accuracy relation for Qwen-14B (counterpart of Figure 4 ). Accuracy generally decreases with distance across target datasets.

Figure B.4: Cross-dataset diagnostics for Llama-8B (layer 18). (a) AUROC retention, (b) distance–accuracy correlation, (c) distance between learned centers, and (d) correlation between learned dimension weights. The distance-based ordering transfers across datasets even though the centers and dimension weights are dataset-specific.

Factual Knowledge in Language Models

Factual knowledge in language models concerns what facts they store, how reliably they retrieve them, and how that knowledge can be tested or changed.

See all 195 papers

Abstract representational geometry supports inference in large language models
Jun 22, 2026Yunan Zeng, Yuwang WangLLM Inference LLM Interpretability
College of Future Information Technology, Fudan University, Shanghai, 200433, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China
Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs
Nov 8, 2025Renfei Zhang, Manasa Kaniselvan, Rylan Schaeffer abd Niloofar MireshghallahRL for Language Model Reasoning Factual Knowledge in Language Models
Carnegie Mellon University · MIT · Stanford University
Improving Parametric Knowledge Access in Reasoning Language Models
Feb 25, 2026Melody Ma, John HewittFactual Knowledge in Language Models Knowledge-Intensive QA
Columbia University

Jun 22, 2026J/K move · Enter open · S save

Abstract representational geometry supports inference in large language models

Yunan Zeng, Yuwang Wang

College of Future Information Technology, Fudan University, Shanghai, 200433, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China

LLM Inference LLM Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

A defining feature of human intelligence is the ability to adapt to changing environments by inferring latent task structure from sparse observations. Neuroscientific research indicates that this capability relies on the hippocampus constructing abstract representations, expressed as low-dimensional, approximately orthogonal manifolds in neural state space. However, the internal mechanisms of large language models (LLMs) remain largely opaque, making it unclear whether they form comparable abstract representations or instead rely on task-specific statistical regularities when performing comparable reasoning tasks. Here we adapt a contextual reversal-learning paradigm to a text-based setting and compare humans and LLMs at both the Behavioural and representational levels. We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus. Notably, this representational geometry is not uniformly distributed but is organized hierarchically across model depth: whereas lower layers show early, stable encoding of stimulus identity, higher layers form a hippocampal-like functional band enriched for abstract context geometry associated with inference. Furthermore, complementary intervention experiments mechanistically implicate geometry in reasoning: task-sequence language modelling induces geometric disentanglement, whereas geometric regularization of higher layers increases the emergence of generalizable inference. Together, these findings establish abstract representational geometry as a mechanistic principle supporting inference in large language models.

Abstract

Factual Knowledge in Language Models

Explore similar work

Abstract representational geometry supports inference in large language models

Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

Improving Parametric Knowledge Access in Reasoning Language Models