Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds

Md. Faiyaz Abdullah Sayeedi

cs.CLAug 18, 2026

Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds

Authors: Md. Faiyaz Abdullah Sayeedi

Organizations: BRAC University, Bangladesh

Abstract

Large Language Models (LLMs) demonstrate remarkable multi-hop reasoning capabilities over long contexts, yet the internal mechanisms enabling these distant cognitive leaps remain poorly understood. Traditional attention-based interpretability often fails to capture true semantic proximity due to routing artifacts like attention sinks. In this paper, we bypass attention weights to directly analyze the dynamic geometry of the hidden state manifold, proving that deep LLM latent spaces natively organize into Small-World networks. By sparsifying the continuous similarity matrices of long-context representations into unweighted graphs, we trace the connectivity between highly disjoint semantic anchors across two distinct architectures. Our findings reveal a sharp topological phase transition: while early syntactic layers remain entirely fractured, deep reasoning layers abruptly compress massive conceptual distances into highly navigable pathways strictly bounded by the "Six Degrees of Separation" limit (=< 6 semantic hops). Furthermore, we demonstrate the practical efficacy of this framework by applying it to zero-shot hallucination detection within Retrieval-Augmented Generation (RAG) using the RAGognize dataset. We show that factually grounded generations maintain structural integrity with their source context (approximately 3 hops), whereas hallucinations induce severe topological collapse. Ultimately, this work mathematically formalizes how transformers execute abstract reasoning and provides a novel, strictly geometric signature for evaluating factual reliability.

Figures & tables

Figure 1: Illustration of Topological Compression. A 3D PCA projection of an LLM’s deep latent manifold. The highlighted Small-World pathway bridges fundamentally disjoint anchors, Mosquito (ecology) and Jailbreak (cybersecurity), across a dense cloud of unrelated context tokens (gray). The model compresses this massive conceptual distance into just 5 hops by leveraging polysemous intermediate concepts (e.g., Virus).

	Qwen2.5-1.5B (Layer 24)		Phi-3-Mini (Layer 30)
Threshold ( $\tau$ )	Deep $C_{rate}$	Deep $L_{avg}$	Deep $C_{rate}$	Deep $L_{avg}$
0.85	0.0%	$\infty$	12.3%	4.93
0.83	2.0%	6.00	25.8%	4.19
0.81	7.5%	4.93	42.0%	3.60
0.79	15.5%	4.42	52.1%	2.95
0.77	30.0%	4.40	58.9%	2.50

Table 1: Phase Transition Sweep for Semantic Anchors (

\geq 200

token separation).

C_{rate}

denotes Connectivity Rate.

L_{avg}

denotes Average Hops. Early layers uniformly failed to connect.

Figure 2: The Phase Transition Boundary for Qwen2.5-1.5B. Early syntax layers completely fail to connect the semantic anchors, whereas the deep reasoning layers exhibit a sudden Small-World emergence at $\tau=0.81$ .

Figure 3: Phase Transition Boundary for Phi-3-Mini. The deep layer sharply transitions into a highly connected state, proving the universality of the topological snap across different pretraining paradigms.

Figure 4: Average Semantic Hops for Qwen2.5-1.5B. Once the phase boundary is crossed, the context window is consistently compressed into fewer than 6 hops.

Figure 5: Average Semantic Hops for Phi-3-Mini. The model natively operates well beneath the Six Degrees limit, compressing 200+ tokens of physical distance into $\approx 3$ to $5$ hops.

Layer (Qwen2.5-1.5B)	$C_{rate}$	$L_{avg}$
Layer 4 (Early Syntax)	0.0%	–
Layer 12 (Mid Syntax)	0.8%	–
Layer 16 (Transitional)	12.3%	7.91
Layer 22 (Solidifying)	44.7%	5.62
Layer 24 (Deep Reasoning)	92.4%	3.42

Table 2: Layer-Wise Topological Evolution (

\tau=0.81

L_{avg}

is omitted (–) for early layers where extreme sparsity prevents the formation of viable pathways.

Physical Separation ( $\delta$ )	$C_{rate}$	$L_{avg}$
$\delta=10$ tokens	95.8%	3.84
$\delta=50$ tokens	94.1%	4.15
$\delta=100$ tokens	92.4%	4.78
$\delta=250$ tokens	88.6%	5.31

Table 3: Impact of Anchor Separation (

\delta

) on Deep Layer. The semantic routing efficiency (

L_{avg}

) remains remarkably stable and strictly bounded within the

\leq 6

hop limit, independent of physical sequence distance.

Model	Prompt Config.	$C_{rate}$	$L_{avg}$
Qwen2.5-1.5B	Answerable (Factual)	92.4%	3.42
Qwen2.5-1.5B	Unanswerable (Hallucinated)	14.8%	8.75
Phi-3-Mini	Answerable (Factual)	96.1%	2.81
Phi-3-Mini	Unanswerable (Hallucinated)	18.2%	7.33

Table 4: Latent Topological Signatures on the RAGognize Dataset Framework (

\tau=0.81

). Hallucinated outputs in the unanswerable configuration exhibit severe topological fracturing.

C_{rate}

: Connectivity Rate.

L_{avg}

: Average Path Length.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	AUROC	F1 Score	Precision	Recall
ROUGE-L	0.68	0.62	0.65	0.59
PPL	0.74	0.69	0.71	0.67
Ours	0.89	0.84	0.86	0.82

Appendix

Table 5: Hallucination Detection Performance on Qwen2.5-1.5B (RAGognize Test Set). The topological approach significantly outperforms standard lexical and statistical confidence metrics.

Long-Context Reasoning in Language Models

See all 40 papers

Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning
Mar 1, 2026Hamed Damirchi, Ignacio Meza De la Jara, Ehsan Abbasnejad +3LLM Interpretability Representation Geometry in Language Models
Australian Institute for Machine Learning, Adelaide University · Monash University · Concordia University
When Do Hallucinations Arise? A Graph Perspective on the Evolution of Path Reuse and Path Compression
Apr 4, 2026Xinnan Dai, Kai Yang, Cheng Luo +3Hallucination in Language Models Knowledge Conflicts in Language Models
Michigan State University · University of Michigan · Independent Researcher
Geometric Latent Reasoning Induces Shorter Generations in LLMs
Jun 1, 2026Shashi Kumar, Yacouba Kaloga, Petr Motlicek +2Efficient Language Model Reasoning Continuous Latent Reasoning
Idiap Research Institute, Switzerland · EPFL, Switzerland · BUT, Czech Republic

Mar 1, 2026J/K move · Enter open · S save

Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning

Hamed Damirchi, Ignacio Meza De la Jara, Ehsan Abbasnejad, Afshar Shamsi+2

Australian Institute for Machine Learning, Adelaide University · Monash University · Concordia University

LLM Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

Existing explainability methods for Large Language Models (LLMs) typically treat hidden states as static points in activation space, assuming that correct and incorrect inferences can be separated using representations from an individual layer. However, these activations are saturated with polysemantic features, leading to linear probes learning surface-level lexical patterns rather than underlying reasoning structures. We introduce Truth as a Trajectory (TaT), which models the transformer inference as an unfolded trajectory of iterative refinements, shifting analysis from static activations to layer-wise geometric displacement. By analyzing displacement of representations across layers, TaT uncovers geometric invariants that distinguish valid reasoning from spurious behavior. We evaluate TaT across dense and Mixture-of-Experts (MoE) architectures on benchmarks spanning commonsense reasoning, question answering, and toxicity detection. Without access to the activations themselves and using only changes in activations across layers, we show that TaT effectively mitigates reliance on static lexical confounds, outperforming conventional probing, and establishes trajectory analysis as a complementary perspective on LLM explainability.

Abstract

Long-Context Reasoning in Language Models

Explore similar work

Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning

When Do Hallucinations Arise? A Graph Perspective on the Evolution of Path Reuse and Path Compression

Geometric Latent Reasoning Induces Shorter Generations in LLMs