Large Language Models (LLMs) demonstrate remarkable multi-hop reasoning capabilities over long contexts, yet the internal mechanisms enabling these distant cognitive leaps remain poorly understood. Traditional attention-based interpretability often fails to capture true semantic proximity due to routing artifacts like attention sinks. In this paper, we bypass attention weights to directly analyze the dynamic geometry of the hidden state manifold, proving that deep LLM latent spaces natively organize into Small-World networks. By sparsifying the continuous similarity matrices of long-context representations into unweighted graphs, we trace the connectivity between highly disjoint semantic anchors across two distinct architectures. Our findings reveal a sharp topological phase transition: while early syntactic layers remain entirely fractured, deep reasoning layers abruptly compress massive conceptual distances into highly navigable pathways strictly bounded by the "Six Degrees of Separation" limit (=< 6 semantic hops). Furthermore, we demonstrate the practical efficacy of this framework by applying it to zero-shot hallucination detection within Retrieval-Augmented Generation (RAG) using the RAGognize dataset. We show that factually grounded generations maintain structural integrity with their source context (approximately 3 hops), whereas hallucinations induce severe topological collapse. Ultimately, this work mathematically formalizes how transformers execute abstract reasoning and provides a novel, strictly geometric signature for evaluating factual reliability.
Figures & tables
Figure 1: Illustration of Topological Compression. A 3D PCA projection of an LLM’s deep latent manifold. The highlighted Small-World pathway bridges fundamentally disjoint anchors, Mosquito (ecology) and Jailbreak (cybersecurity), across a dense cloud of unrelated context tokens (gray). The model compresses this massive conceptual distance into just 5 hops by leveraging polysemous intermediate concepts (e.g., Virus).
Qwen2.5-1.5B (Layer 24)
Phi-3-Mini (Layer 30)
Threshold ( τ )
Deep Crate
Deep Lavg
Deep Crate
Deep Lavg
0.85
0.0%
∞
12.3%
4.93
0.83
2.0%
6.00
25.8%
4.19
0.81
7.5%
4.93
42.0%
3.60
0.79
15.5%
4.42
52.1%
2.95
0.77
30.0%
4.40
58.9%
2.50
Table 1: Phase Transition Sweep for Semantic Anchors ( ≥200 token separation). Crate denotes Connectivity Rate. Lavg denotes Average Hops. Early layers uniformly failed to connect.
Figure 2: The Phase Transition Boundary for Qwen2.5-1.5B. Early syntax layers completely fail to connect the semantic anchors, whereas the deep reasoning layers exhibit a sudden Small-World emergence at τ=0.81 .
Figure 3: Phase Transition Boundary for Phi-3-Mini. The deep layer sharply transitions into a highly connected state, proving the universality of the topological snap across different pretraining paradigms.
Figure 4: Average Semantic Hops for Qwen2.5-1.5B. Once the phase boundary is crossed, the context window is consistently compressed into fewer than 6 hops.
Figure 5: Average Semantic Hops for Phi-3-Mini. The model natively operates well beneath the Six Degrees limit, compressing 200+ tokens of physical distance into ≈3 to 5 hops.
Layer (Qwen2.5-1.5B)
Crate
Lavg
Layer 4 (Early Syntax)
0.0%
–
Layer 12 (Mid Syntax)
0.8%
–
Layer 16 (Transitional)
12.3%
7.91
Layer 22 (Solidifying)
44.7%
5.62
Layer 24 (Deep Reasoning)
92.4%
3.42
Table 2: Layer-Wise Topological Evolution ( τ=0.81 ). Lavg is omitted (–) for early layers where extreme sparsity prevents the formation of viable pathways.
Physical Separation ( δ )
Crate
Lavg
δ=10 tokens
95.8%
3.84
δ=50 tokens
94.1%
4.15
δ=100 tokens
92.4%
4.78
δ=250 tokens
88.6%
5.31
Table 3: Impact of Anchor Separation ( δ ) on Deep Layer. The semantic routing efficiency ( Lavg ) remains remarkably stable and strictly bounded within the ≤6 hop limit, independent of physical sequence distance.
Model
Prompt Config.
Crate
Lavg
Qwen2.5-1.5B
Answerable (Factual)
92.4%
3.42
Unanswerable (Hallucinated)
14.8%
8.75
Phi-3-Mini
Answerable (Factual)
96.1%
2.81
Unanswerable (Hallucinated)
18.2%
7.33
Table 4: Latent Topological Signatures on the RAGognize Dataset Framework ( τ=0.81 ). Hallucinated outputs in the unanswerable configuration exhibit severe topological fracturing. Crate : Connectivity Rate. Lavg : Average Path Length.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Method
AUROC
F1 Score
Precision
Recall
ROUGE-L
0.68
0.62
0.65
0.59
PPL
0.74
0.69
0.71
0.67
Ours
0.89
0.84
0.86
0.82
Appendix
Table 5: Hallucination Detection Performance on Qwen2.5-1.5B (RAGognize Test Set). The topological approach significantly outperforms standard lexical and statistical confidence metrics.
Existing explainability methods for Large Language Models (LLMs) typically treat hidden states as static points in activation space, assuming that correct and incorrect inferences can be separated using representations from an individual layer. However, these activations are saturated with polysemantic features, leading to linear probes learning surface-level lexical patterns rather than underlying reasoning structures. We introduce Truth as a Trajectory (TaT), which models the transformer inference as an unfolded trajectory of iterative refinements, shifting analysis from static activations to layer-wise geometric displacement. By analyzing displacement of representations across layers, TaT uncovers geometric invariants that distinguish valid reasoning from spurious behavior. We evaluate TaT across dense and Mixture-of-Experts (MoE) architectures on benchmarks spanning commonsense reasoning, question answering, and toxicity detection. Without access to the activations themselves and using only changes in activations across layers, we show that TaT effectively mitigates reliance on static lexical confounds, outperforming conventional probing, and establishes trajectory analysis as a complementary perspective on LLM explainability.
Hamed Damirchi, Ignacio Meza De la Jara, Ehsan Abbasnejad +3
Australian Institute for Machine Learning, Adelaide University · Monash University · Concordia University
Reasoning hallucinations in large language models (LLMs) often appear as fluent yet unsupported conclusions that violate either the given context or underlying factual knowledge. Although such failures are widely observed, the mechanisms by which decoder-only Transformers produce them remain poorly understood. We model next-token prediction as a graph search process over an underlying graph, where entities correspond to nodes and learned transitions form edges. From this perspective, contextual reasoning is a constrained search over a sampled subgraph (intrinsic reasoning), while context-free queries rely on memorized structures in the underlying graph (extrinsic reasoning). We show that reasoning hallucinations arise from two fundamental mechanisms: Path Reuse, where memorized knowledge overrides contextual constraints during early training, and Path Compression, where frequently traversed multi-step paths collapse into shortcut edges in later training. Together, these mechanisms provide a unified explanation for reasoning hallucinations in LLMs and connected to well-known behaviors observed in downstream applications.
Xinnan Dai, Kai Yang, Cheng Luo +3
Michigan State University · University of Michigan · Independent Researcher
Large language models solve complex problems by generating lengthy chains of explicit reasoning tokens. While effective, this makes reasoning expensive, length-sensitive, and constrained to (discrete) natural language. While latent reasoning offers a continuous alternative, determining useful structures for intermediate latent states is an open challenge. In this paper, we formulate latent reasoning as a geometric path-approximation problem within the model's pretrained token-embedding space. We introduce Geometric Latent Reasoning (GLR), which uses a lightweight transition head to predict iterative direction updates in embedding space. Using textual chain-of-thought traces as anchors, GLR learns to approximate discrete reasoning trajectories while permitting continuous deviations from exact token embeddings. Evaluations on mathematical reasoning benchmarks using Qwen3 models reveal an emergent phenomenon: geometric latent reasoning induces substantially shorter generations without an explicit length objective. By replacing early explicit reasoning with continuous latent steps, models often reach correct answers using substantially fewer total generation steps. These findings suggest that continuous trajectories act as compact intermediate reasoning states, exposing a new tradeoff between latent computation budget, output length, and accuracy.
Shashi Kumar, Yacouba Kaloga, Petr Motlicek +2
Idiap Research Institute, Switzerland · EPFL, Switzerland · BUT, Czech Republic