This paper studies the local geometry of semantic meaning in large language models. Current representation analyses ask whether meanings are close in embedding space, or whether linear probes separate semantic classes, but they do not characterize the within-concept local geometry: what is invariant across realizations of a single semantic factor, what varies, and what changes when meaning is refined. This paper characterizes that geometry through two controlled probes evaluated on four architecturally diverse decoder-only LLMs (Gemma 4, Llama 3.2, Qwen 2.5, Pythia). A cross-lingual probe uses language as a perturbation to separate exact meaning matches, same-factor realizations, nearby concepts, and out-of-domain contrasts. A refinement probe separates specificity-bearing semantic narrowing from high-probability attributive elaboration and adversarial controls; both probes are validated by blind human judgments. Four findings recur across all models: under mean pooling, once dominant variance directions are removed, sentence pairs order by semantic relatedness (same factor, then same domain, then different domain) without overlap over a band of layers; semantic factors organize around an invariant core surrounded by structured realization-specific variation; refinement produces directed displacement into tighter neighborhoods, including against controls built on the same base noun; and sentence-level and token-local readouts expose complementary geometry (volume contraction at the sentence level, sharper anisotropic endpoints at token-local readout). We interpret these results as evidence that LLM semantics are organized as local anisotropic neighborhoods in which meaning contracts under refinement.
Figures & tables
Probe
Construction
Total
Semantic-neighborhood
10 domains ×4 factors ×5 languages ×6 sentences
1,200
Semantic refinement
48 chains ×5 levels ×4 templates ×4 variants
3,840
Table 1: How the two probes are constructed.
Model
Family
Hidden-state stack
Width
Pythia 2.8B
GPT-NeoX style decoder
33
2560
Llama 3.2 3B
Llama-family decoder
29
3072
Qwen 2.5 3B
Qwen-family decoder
37
2048
Gemma 4 E4B
Gemma-family decoder
43
2560
Table 2: Models analyzed. Layer counts include the input embedding state.
Figure 1: Cosine geometry for Llama 3.2 3B (top) and Qwen 2.5 3B (bottom) under raw, top-1, and top-5 deflation. Removing dominant directions reveals a clearer semantic hierarchy, consistent with the dominant-subspace masking documented in prior work ( Ethayarajh, 2019 ; Mu and Viswanath, 2018 ; Cai et al., 2021 ) .
Figure 2: Mahalanobis distance in the semantic-neighborhood probe. Left: Gemma 4 E4B raw profile. Right: all-model profiles. Exact meaning matches (C1) form the nearest category; same-factor independent realizations (C2, C3) are the farthest, exposing the full realization spread of each factor neighborhood relative to the ambient variance of the layer.
Figure 3: Euclidean distance under raw and top-10-deflated conditions for Gemma 4 E4B and Qwen 2.5 3B. In mean pooling, deflation pushes Euclidean ordering toward the Mahalanobis ordering; in last-token pooling, it does not move toward the Mahalanobis ordering.
Figure 4: Refinement versus control in Gemma 4 E4B under mean pooling. Distances from base concept S0 to later levels (left: Euclidean; right: Mahalanobis). Refinement trajectories lie above control across most depths and target levels.
Figure 5: Refinement geometry in Llama 3.2 3B, gaps from S0 . Rows: readout (top: mean pooling, sentence-level; bottom: last token, token-local). Columns: spread measure (left: top-5-deflated log-volume; right: raw participation ratio). Refinement (solid) contracts local volume more strongly than control (dashed) under both readouts; participation ratio separates the arms most clearly at token-local readout.
Figure 6: Per-position increments in log-volume (Llama 3.2 3B top, Pythia 2.8B bottom); columns show refinement, control, and their difference. Refinement-specific contraction strengthens through late-prefix positions; at the terminal period the increment is near zero in both arms.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Pair type
English
German
Exact translation
The object is hot.
Das Objekt ist heiß.
Same-facet same-factor
It has a very high temperature.
Es hat eine sehr hohe Temperatur.
Cross-facet same-factor
You would burn yourself on it.
Man würde sich daran verbrennen.
Appendix
Table 3: Examples distinguishing exact translation from same-facet and cross-facet same-factor pairs for the factor hot .
Model
Readout
Exact match dM
Same-facet dM
Cross-facet dM
Cross-facet / same-facet
Pythia 2.8B
Mean
11.54
15.36
17.18
1.12
Pythia 2.8B
Last-token
12.42
16.90
17.14
1.01
Llama 3.2 3B
Mean
9.57
13.10
17.37
1.33
Llama 3.2 3B
Last-token
12.60
13.14
16.15
1.23
Qwen 2.5 3B
Mean
10.82
13.34
16.88
1.27
Qwen 2.5 3B
Last-token
12.53
13.13
16.13
1.23
Appendix
Table 4: Mahalanobis means for exact matches, same-facet same-factor pairs, and cross-facet same-factor pairs inside the model-specific semantic band. Cross-facet pairs are consistently farther than same-facet pairs.
Category
Example ( The object is hot. → )
C1 Exact meaning, cross-language
Das Objekt ist heiß.
C2 Same factor, same language
It has a very high temperature.
C3 Same factor, cross-language
Same facet
Es hat eine sehr hohe Temperatur.
Cross-facet
Man würde sich daran verbrennen.
C4 Same domain, different factor, same language
The object is cold.
Appendix
Table 5: Examples of the seven semantic-neighborhood pair categories.
Arm
S0
S1
S2
S3
S4
Refinement
report
audit report
environmental audit report
annual environmental audit report
certified annual environmental audit report
Main control
report
written report
written dated report
written dated formal report
written dated formal official report
Refinement
chair
office chair
ergonomic office chair
adjustable ergonomic office chair
adjustable ergonomic mesh office chair
Main control
document
written document
written dated document
written dated signed document
written dated signed official document
Near-synonym control
object
large object
large big object
large big huge object
large big huge enormous object
Scope-hedge control
object
fragile object
partly fragile object
somewhat partly fragile object
broadly somewhat partly fragile object
Appendix
Table 6: Example chains from the semantic refinement probe. The first two rows share the base noun report .
Readout
Metric
Strict R (%)
B (%)
Mean
Cosine
0–5 → 19–76
0 → 0
Mean
Euclidean
0–5 → 0
0 → 0–97
Mean
Mahalanobis
0 → 0
84–100 → 93–100
Last-token
Cosine
0–12 → 0–55
0 → 0–3
Last-token
Euclidean
0–12 → 0–21
0 → 0–94
Last-token
Mahalanobis
0 → 0–3
42–97 → 58–97
Appendix
Table 7: Two orderings across metrics and readouts: range over the four models of the percentage of layers satisfying strict R and B, in raw space → at the deflation level used in the figures ( k=5 , or 8 for Pythia, under mean pooling; k=4 , or 10 for Pythia, under last-token readout). Mahalanobis uses r=128 (mean) and r=256 (last-token). Cosine almost never satisfies B and Mahalanobis almost never satisfies R. Euclidean B depends on k (Table 13 ); per-model values are released with the code.
Model
raw
k=1 – 3
k=4
k=5
k=6
k=7
k=8
k=9
k=10
Pythia 2.8B (33)
0
0
2
0
0
4
7
8
15
Llama 3.2 3B (29)
0
0
6
17
19
21
22
22
24
Qwen 2.5 3B (37)
0
0
17
28
28
28
27
27
28
Gemma 4 E4B (43)
2
0
12
8
6
9
8
15
17
Appendix
Table 8: Layers satisfying strict R under cosine similarity (mean pooling), by deflation level k . Total layer count in parentheses.
Model
Readout
k
ρr=128
ρr=256
B r=128
B r=256
B r=512
Pythia 2.8B
Mean
8
1.00
1.00
✓
✓
✓
Llama 3.2 3B
Mean
5
0.96
0.96
✓
✓
✓
Qwen 2.5 3B
Mean
5
0.96
1.00
✓
✓
✓
Gemma 4 E4B
Mean
5
1.00
1.00
✓
✓
✓
Pythia 2.8B
Last-token
10
1.00
0.96
✓
✓
✓
Llama 3.2 3B
Last-token
4
0.71
0.96
–
✓
✓
Appendix
Table 9: Rank sensitivity at normalized depth 0.5 , at each model’s deflation level. ρ : Spearman correlation of the seven-category ordering with the r=512 reference. ✓/–: B holds / fails.
Model
en
de
es
ar
zh
Pythia 2.8B
0.29 → 0.55
0.18 → 0.36
0.18 → 0.49
0.06 → 0.18
0.16 → 0.43
Llama 3.2 3B
0.30 → 0.70
0.32 → 0.60
0.28 → 0.69
0.10 → 0.44
0.12 → 0.61
Qwen 2.5 3B
0.41 → 0.80
0.39 → 0.73
0.38 → 0.79
0.24 → 0.59
0.31 → 0.71
Gemma 4 E4B
0.30 → 0.43
0.23 → 0.32
0.27 → 0.38
0.13 → 0.26
0.17 → 0.27
Mean
0.33 → 0.62
0.28 → 0.50
0.28 → 0.59
0.13 → 0.37
0.19 → 0.51
Appendix
Table 10: Per-language cross-lingual Recall@1 at normalized depth 0.5 , raw → deflated.
Quantity
Pythia 2.8B
Llama 3.2 3B
Qwen 2.5 3B
Gemma 4 E4B
Log-volume, raw
0.0007
0.0024
0.0005
< 0.0001
Log-volume, top-5 deflated
< 0.0001
< 0.0001
0.0001
< 0.0001
Participation ratio, raw
0.0004
0.164
0.412
0.005
Appendix
Table 11: Permutation p -values ( 10,000 arm-label reshuffles across chains) for the S0→S4 refinement-minus-control gap, mean pooling, at one mid-depth layer per model.
Log-volume gap
Euclidean displacement gap
Majority of bases, % layers
Model
unpaired
matched
unpaired
matched
contraction
displacement
Pythia 2.8B
2.28
2.36
3.56
6.85
100
100
Llama 3.2 3B
1.81
1.75
0.74
2.02
97
100
Qwen 2.5 3B
1.84
1.71
4.83
14.12
87
100
Gemma 4 E4B
2.60
2.07
2.72
3.24
93
100
Appendix
Table 12: Unpaired (all 20 vs. 20 chains) versus same-base matched ( 7 vs. 7 ) refinement-minus-control gaps, and the share of layers at which a majority of the seven bases favor refinement.
Model
k=5
k=6
k=7
k=8
k=9
k=10
Pythia 2.8B
0 (0)
30 (30)
82 (79)
97 (88)
97 (91)
100 (91)
Llama 3.2 3B
0 (0)
38 (34)
100 (97)
100 (97)
100 (100)
100 (100)
Qwen 2.5 3B
0 (0)
3 (3)
22 (16)
51 (41)
62 (54)
100 (95)
Gemma 4 E4B
16 (2)
33 (19)
44 (37)
67 (60)
81 (72)
81 (72)
Appendix
Table 13: Deflated Euclidean distance, mean pooling: percentage of layers satisfying B, with the percentage satisfying B + in parentheses, by deflation level k .
Model
Readout
Metric
Mean gap
AUC
Positive layers (%)
Strict CI-separated (%)
Pythia 2.8B
Mean
Euclidean
3.56
117.63
100
33
Pythia 2.8B
Mean
Mahalanobis
2.37
78.28
100
100
Llama 3.2 3B
Mean
Euclidean
0.74
21.58
100
21
Llama 3.2 3B
Mean
Mahalanobis
2.47
71.66
100
100
Qwen 2.5 3B
Mean
Euclidean
4.83
178.86
100
24
Qwen 2.5 3B
Mean
Mahalanobis
2.58
95.56
100
100
Appendix
Table 14: Refinement-minus-control displacement for the S0→S4 contrast. Positive values mean refinement moves farther than control.
Model
Readout
Condition
Mean gap
AUC
Positive layers (%)
Strict CI-separated (%)
Pythia 2.8B
Mean
Raw
2.28
75.08
97
94
Pythia 2.8B
Mean
Top-5 deflated
3.37
111.22
100
97
Llama 3.2 3B
Mean
Raw
1.81
52.52
100
66
Llama 3.2 3B
Mean
Top-5 deflated
2.58
74.95
100
97
Qwen 2.5 3B
Mean
Raw
1.84
68.20
100
62
Qwen 2.5 3B
Mean
Top-5 deflated
2.50
92.51
100
86
Appendix
Table 15: Refinement-minus-control log-volume summaries for span-4 contrasts. Positive values mean the control branch occupies the larger neighborhood, i.e., refinement is more contractive.
Model
Readout
Mean gap
AUC
Positive (%)
Negative (%)
Strict positive (%)
Strict negative (%)
Pythia 2.8B
Mean
-0.70
-23.25
55
45
42
33
Llama 3.2 3B
Mean
-0.26
-7.61
66
34
0
21
Qwen 2.5 3B
Mean
-0.37
-13.56
0
100
0
22
Gemma 4 E4B
Mean
-0.68
-29.38
0
100
0
67
Pythia 2.8B
Last-token
-0.08
-2.72
61
36
0
15
Llama 3.2 3B
Last-token
-2.63
-76.13
0
97
0
97
Appendix
Table 16: Refinement-minus-control raw participation-ratio summaries for span-4 contrasts. Negative values indicate lower effective dimensionality under refinement than under control.
Figure 7: Gap profiles Δmetric(S4−S0) for refinement (solid), control (dashed), and near-synonym (dash-dot), with 95% CI bands shaded. Top row: log-volume; bottom row: participation ratio. Columns are models. Near-synonym tracks close to refinement in log-volume for Pythia and Qwen, while Llama and Gemma show a shallower contraction. In participation-ratio terms the arm is usually near zero or positive, contrasting with refinement which is always negative.
Figure 8: Same layout as Figure 7 for scope-hedge. The separation from refinement is consistent across all four models and both metrics: scope-hedge produces a substantially smaller log-volume contraction and stays near or above zero in participation ratio throughout the network depth, whereas refinement is persistently more negative in both.
Arm
Model
S0
S1
S2
S3
S4
Refinement
Pythia 2.8B
10.5
10.6
10.8
10.9
11.1
Llama 3.2 3B
11.4
11.4
11.5
11.5
11.8
Qwen 2.5 3B
10.4
10.4
10.5
10.5
10.8
Gemma 4 E4B
11.4
11.4
11.4
11.5
11.6
Control
Pythia 2.8B
11.5
11.5
11.7
11.8
11.9
Llama 3.2 3B
12.3
12.4
12.5
12.6
12.7
Appendix
Table 17: Mean number of tokens per sentence at each refinement level, by arm and model. Token counts use each model’s own tokenizer. Growth from S0 to S4 is 0.2 – 0.8 tokens across all conditions.
Large language models often preserve meaning despite substantial changes in wording, style, and syntax, while small semantic edits can systematically alter their hidden representations. This suggests that semantic variation may be organized along recurring local directions. We propose the Invariant Atom Hypothesis: local semantic motion admits preferred sparse coordinates along directions that remain stable under meaning-preserving transformations. We learn a shared semantic frame and sparse coordinates that reconstruct semantic displacements while suppressing nuisance variation, with anchor-dependent diagonal modulation adjusting atom strengths without sample-specific rotations. Empirically, the atoms exhibit strong semantic--nuisance separation, sparse reconstruction, reproducible directions, and causal effects on model predictions. The learned geometry generalizes to unseen semantic neighborhoods and nuisance families, while local reweighting improves semantic selectivity and preserves a consistent global-to-local structure. Atom signatures also remain stable under model modification. These findings support reusable invariant directions as a sparse coordinate system for local semantic geometry in language models.
Muhammad Ahtesham, Xin Zhong
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA
Understanding whether large language models (LLMs) capture structured meaning requires examining how they represent concept relationships. In this work, we study three models of increasing scale: Pythia-70M, GPT-2, and Llama 3.1 8B, focusing on four semantic relations: synonymy, antonymy, hypernymy, and hyponymy. We combine linear probing with mechanistic interpretability techniques, including sparse autoencoders (SAE) and activation patching, to identify where these relations are encoded and how specific features contribute to their representation. Our results reveal a directional asymmetry in hierarchical relations: hypernymy is encoded redundantly and resists suppression, while hyponymy relies on compact features that are more easily disrupted by ablation. More broadly, relation signals are diffuse but exhibit stable profiles: they peak in the mid-layers and are stronger in post-residual/MLP pathways than in attention. Difficulty is consistent across models (antonymy easiest, synonymy hardest). Probe-level causality is capacity-dependent: on Llama 3.1, SAE-guided patching reliably shifts these signals, whereas on smaller models the shifts are weak or unstable. Our results clarify where and how reliably semantic relations are represented inside LLMs, and provide a reproducible framework for relating sparse features to probe-level causal evidence.
Large language models often produce judgments that resemble human responses, but it remains unclear whether they internally capture linguistic nuances such as conventional versus idiomatic meaning. A layer-wise representational analysis was conducted on two equal-sized pre-trained models, Llama-3.2-1B and Gemma-3-1B-PT, to examine whether and where these models differentiate linguistic traits from input. The analysis combined layer-wise probing with geometric distance measures, using the top embedding dimensions ranked by principal component loadings for each layer. The results show that even with equal capacity, models organize linguistic information in fundamentally different ways, suggesting the need to analyze internal representational geometry, examining not just behavioral outputs but semantic space alignment when evaluating how closely human and model language interpretation correspond.