Invariant Atoms: Sparse Coordinates of Local Semantic Geometry in Language Model Representations

Muhammad Ahtesham; Xin Zhong

cs.LGSep 29, 2026

Invariant Atoms: Sparse Coordinates of Local Semantic Geometry in Language Model Representations

Authors: Muhammad Ahtesham, Xin Zhong

Organizations: Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

Abstract

Large language models often preserve meaning despite substantial changes in wording, style, and syntax, while small semantic edits can systematically alter their hidden representations. This suggests that semantic variation may be organized along recurring local directions. We propose the Invariant Atom Hypothesis: local semantic motion admits preferred sparse coordinates along directions that remain stable under meaning-preserving transformations. We learn a shared semantic frame and sparse coordinates that reconstruct semantic displacements while suppressing nuisance variation, with anchor-dependent diagonal modulation adjusting atom strengths without sample-specific rotations. Empirically, the atoms exhibit strong semantic--nuisance separation, sparse reconstruction, reproducible directions, and causal effects on model predictions. The learned geometry generalizes to unseen semantic neighborhoods and nuisance families, while local reweighting improves semantic selectivity and preserves a consistent global-to-local structure. Atom signatures also remain stable under model modification. These findings support reusable invariant directions as a sparse coordinate system for local semantic geometry in language models.

Figures & tables

Figure 1: Invariant atoms as preferred coordinates of semantic variation. (A) Meaning-preserving inputs remain locally stable, while semantic edits induce structured motion. (B) A semantic subspace identifies where such motion occurs but not a unique coordinate system. (C) Invariant atoms provide preferred sparse directions that compose semantic change while remaining stable to meaning-preserving variation.

Figure 2: Local geometric view of invariant atoms. Representations lie on a manifold $\mathcal{M}$ whose tangent space locally decomposes into nuisance directions $\mathcal{N}_{z}$ and semantic directions $\mathcal{H}_{z}$ . Invariant atoms form preferred local directions within $\mathcal{H}_{z}$ , allowing semantic motion to be expressed as a sparse combination of reusable coordinates.

Figure 3: Learning invariant atoms from local semantic variation. SP and SC transformations define nuisance and semantic displacements around an anchor representation. Geometric objectives suppress nuisance motion, preserve SP consistency, and reconstruct semantic motion, yielding sparse and non-redundant atoms.

Model	Learned		Baselines
$\boldsymbol{(S/T\uparrow)}$	Global	Local	Recon	PCA	Random
Mistral	1.27	1.42	0.69	0.70	0.68
LLaMA	1.88	1.98	0.89	0.83	0.79
Gemma	1.59	2.12	0.62	0.65	0.59
Qwen	1.63	1.84	0.73	0.76	0.71
GLM	2.05	2.19	1.02	0.93	0.86

Table 1: Semantic selectivity of learned global and locally reweighted atom frames compared with reconstruction-only and linear baselines. Local context improves

S/T

across all seven models.

	Deletion			Injection
Model	Active	Inactive	Shuffled	Active
Mistral	0.125	0.000	$-$ 0.006	0.105
LLaMA	0.099	$-$ 0.003	$-$ 0.005	0.141
Gemma	0.101	0.005	0.008	0.084
Qwen	0.177	0.003	$-$ 0.004	0.152
GLM	0.074	0.001	$-$ 0.005	0.084

Table 2: Causal interventions (

m=32

), reported as median

R_{\mathrm{causal}}

. Deletion removes active atom contributions from an SC state; injection adds them to the anchor. Inactive atoms are norm-matched controls, while Shuffled preserves directions but disrupts learned coefficients. Positive values indicate movement toward the target distribution.

Generalization Test	Global	Local
Unseen neighborhoods: Cos transfer	$0.95$ – $0.98\times$	$0.95$ – $0.98\times$
Unseen neighborhoods: $S/T$	$0.96$ – $1.76$	$0.94$ – $1.65$
Held-out SP families: $S/T$	$1.13$ – $3.32$	$1.75$ – $5.79$

Table 3: Generalization beyond the standard within-neighborhood split. Ranges are across evaluated model families and, for held-out SP families, all nuisance family splits.

Model	PCA	Global	Local
Mistral	.292	.458	.370
LLaMA	.294	.403	.309
Gemma	.242	.345	.230
Qwen	.204	.291	.228
GLM	.286	.363	.288
DeepSeek	.248	.368	.260

Table 4: R@1 semantic retrieval using

256

-dimensional signatures.

Model	Base	Fine-tuned	Distilled
Mistral	.849	.852	.859
LLaMA	.872	.873	.870
Qwen	.866	.861	.861
GLM	.869	.866	.866
DeepSeek	.851	.851	.851
Falcon	.857	.850	.853

Table 5: SC-type classification using atom signatures after modification.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	# Anchor Groups
Animals	169
Science	167
Countries	165
Technology	133
Food	110
History	108

Appendix

Table 6: Distribution of anchor neighborhoods across semantic domains.

Model	Frame	$E_{\mathrm{rec}}\downarrow$	$\mathrm{Cos}\uparrow$	$S/T\uparrow$	$D_{\mathrm{coord}}\downarrow$	$\mathrm{EffNum}$
Mistral	Learned	0.701	0.703	1.27	1.49	21.5
	PCA	0.716	0.690	0.70	2.25	25.0
	Random	0.985	0.171	0.68	2.04	30.8
LLaMA	Learned	0.729	0.670	1.88	1.61	23.0
	PCA	0.750	0.652	0.83	2.37	26.5
	Random	0.984	0.176	0.79	2.06	30.8

Appendix

Table 7: Full comparison across seven model families. Learned atoms consistently achieve

S/T>1

while retaining competitive semantic reconstruction and generally lower coordinate drift than PCA. We report relative reconstruction error

E_{\mathrm{rec}}=\left\|{\Delta z_{\mathrm{sem}}-\Delta\hat{z}_{\mathrm{sem}}}\right\|_{2}/\left\|{\Delta z_{\mathrm{sem}}}\right\|_{2}

, averaged over held-out SC samples. This measures normalized displacement recovery.

	$E_{\mathrm{rec}}\downarrow$		$S/T\uparrow$		$D_{\mathrm{coord}}\downarrow$
Model	Full	Recon	Full	Recon	Full	Recon
Mistral	0.701	0.671	1.27	0.69	1.49	2.34
LLaMA	0.729	0.709	1.88	0.89	1.61	2.55
Gemma	0.711	0.676	1.59	0.62	2.50	4.14
Qwen	0.682	0.652	1.63	0.73	1.77	3.21
GLM	0.716	0.695	2.05	1.02	1.48	2.13

Appendix

Table 8: Full versus SAE-style Recon-Only training across architectures. Reconstruction-only learning often improves

E_{\mathrm{rec}}

but substantially reduces semantic selectivity and increases coordinate drift.

SC Type	LLaMA	Qwen	GLM	DeepSeek
Entity	0.56	0.56	0.58	0.49
Attribute	0.82	0.74	0.83	0.60
Relation	1.35	1.15	1.27	0.81
Negation	3.80	3.57	4.82	1.69
Quantity	1.52	1.53	1.76	0.91
Temporal	2.32	1.66	2.05	1.46

Appendix

Table 9:

S/T

of the learned atom frame by semantic-change type. Logical and temporal changes are generally most separable, whereas entity and attribute substitutions are more difficult because they overlap more strongly with lexical variation.

SP Style	Learned	PCA	Random
Formal	7.98	2257	208
Casual	10.55	3168	340
Question	7.13	1965	194
Technical	10.18	2599	256
Simplified	10.90	2889	292
Verbose	13.07	4897	465

Appendix

Table 10: Nuisance projection energy across SP transformation families for Qwen2.5-7B. Lower is better. Learned atoms suppress nuisance motion across all eight styles rather than specializing to a particular paraphrase template.

$k$	$E_{\mathrm{rec}}\downarrow$	$\mathrm{Cos}\uparrow$	$S/T\uparrow$	$D_{\mathrm{coord}}\downarrow$	$\mathrm{EffNum}$
LLaMA-3.1-8B
4	0.918	0.361	1.88	2.38	3.2
8	0.875	0.462	1.83	2.28	6.2
16	0.829	0.542	1.68	2.02	11.8
24	0.794	0.592	1.91	1.83	17.2
32	0.772	0.622	1.86	1.72	22.1

Appendix

Table 11: Representative sparsity sweeps. Reconstruction improves as more atoms are available, while

S/T

remains stable over a broad range. Effective support remains substantially below the full set of 256 atoms. Sparsity-sweep models are independently trained with random orthonormal initialization; the

k{=}32

entry may therefore differ from the corresponding row in Table 7 , which uses PCA initialization.

Model	Comparison	Mean $\|\cos\|$	Median	$\mathbf{\%>0.9}$	Subspace
Mistral	Learned–Learned	0.711	0.820	36.4	0.845
	Learned–PCA	0.287	0.161	0.4	0.656
	Learned–Random	0.045	0.044	0.0	0.213
LLaMA	Learned–Learned	0.747	0.869	43.8	0.867
	Learned–PCA	0.300	0.159	1.0	0.701
	Learned–Random	0.045	0.045	0.0	0.213

Appendix

Table 12: Cross-seed reproducibility of the global atom frame after optimal permutation and sign alignment. Each model is trained five times from a common PCA initialization, yielding ten learned–learned comparisons. Learned–PCA measures residual alignment with the initialization, while Learned–Random provides a chance-level reference. Across all seven models, independently optimized frames recover both highly similar semantic subspaces and substantially aligned individual atom directions.

	$S/T\uparrow$		$D_{\mathrm{coord}}\downarrow$
Objective	Mistral	Qwen	Mistral	Qwen
Full	1.27	1.63	1.49	1.77
Recon-Only	0.69	0.73	2.34	3.21
No- $L_{\mathrm{nui}}$	1.24	1.54	1.51	1.92
No- $L_{\mathrm{stress}}$	1.26	1.58	1.50	1.81
No- $L_{\mathrm{nui}}$ / $L_{\mathrm{stress}}$	0.69	0.74	2.40	3.25

Appendix

Table 13: Objective ablations on Mistral and Qwen. Removing both transformation-aware invariance terms reproduces the Recon-Only regime, whereas removing either term alone causes only moderate degradation.

Model	Condition	$m{=}1$	$m{=}4$	$m{=}8$	$m{=}16$	$m{=}32$
Mistral	Active	0.013	0.042	0.071	0.100	0.125
	Inactive	0.000	0.001	0.000	0.002	0.000
	Shuffled	-0.014	-0.007	-0.002	-0.002	-0.006
LLaMA	Active	0.015	0.037	0.049	0.078	0.099
	Inactive	0.000	-0.001	-0.002	0.000	-0.003
	Shuffled	-0.017	-0.003	-0.001	-0.005	-0.005

Appendix

Table 14: Full deletion dose response for the global atom frame. Values are median

R_{\mathrm{causal}}

. Active atom effects grow systematically with the number of intervened atoms, whereas norm-matched inactive atoms and coefficient-shuffled controls remain near zero.

	Global		Local
Model	Deletion	Injection	Deletion	Injection
Mistral	0.125	0.105	0.105	0.084
LLaMA	0.099	0.141	0.080	0.125
Gemma	0.101	0.084	0.063	0.064
Qwen	0.177	0.152	0.109	0.104
GLM	0.074	0.084	0.052	0.054

Appendix

Table 15: Causal effects for global and locally scaled atom frames at

m=32

. Values are median

R_{\mathrm{causal}}

for active-atom interventions. Both frames support bidirectional intervention, while the global frame generally produces stronger effects.

		$E_{\mathrm{rec}}\downarrow$		$\mathrm{Cos}\uparrow$		$S/T\uparrow$
Model	Frame	Seen	Unseen	Seen	Unseen	Seen	Unseen
Mistral	Global	0.709	0.723	0.695	0.680	1.29	1.17
	Local	0.695	0.714	0.708	0.689	1.43	1.10
LLaMA	Global	0.733	0.747	0.665	0.650	1.85	1.68
	Local	0.722	0.740	0.676	0.658	2.01	1.54
Gemma	Global	0.718	0.732	0.679	0.665	1.67	1.30

Appendix

Table 16: Generalization to group-disjoint semantic neighborhoods. Entire anchor-centered groups are withheld from atom learning. Reconstruction remains stable on unseen groups, while semantic selectivity transfers strongly for most models.

		Global			Local
Model	Split	Train	Held	Transfer	Train	Held	Transfer
Mistral	A	1.86	2.42	1.30	2.52	3.60	1.43
	B	2.01	1.63	0.81	2.76	2.23	0.81
	C	2.07	1.29	0.62	2.87	1.78	0.62
LLaMA	A	2.63	3.32	1.26	3.40	4.78	1.41
	B	2.80	2.37	0.85	3.67	3.11	0.85

Appendix

Table 17: Generalization to SP families excluded during training. Entries report

S/T

on observed and held-out nuisance families and their ratio. Split A holds out question/passive, Split B casual/technical, and Split C simplified/verbose.

Model	Direction	Frame	Seen $S/T$	Unseen $S/T$	Transfer
Mistral	M $\rightarrow$ Q	Global	1.23	1.46	1.19
		Local	1.62	1.87	1.15
	Q $\rightarrow$ M	Global	4.09	2.02	0.49
		Local	7.60	4.48	0.59
LLaMA	M $\rightarrow$ Q	Global	1.74	2.10	1.21
		Local	2.10	2.47	1.17

Appendix

Table 18: Bidirectional cross-generator generalization.

M

and

Q

denote Mistral- and Qwen-generated perturbation sets. Transfer is the ratio between unseen-generator and seen-generator

S/T

		Mistral		LLaMA		Gemma		Qwen
Method	Dim.	R@1	mAP	R@1	mAP	R@1	mAP	R@1	mAP
Raw	$d$	.440	.519	.419	.508	.384	.472	.287	.363
PCA	256	.292	.366	.294	.381	.242	.329	.204	.273
Global	256	.458	.552	.403	.510	.345	.447	.291	.388
Local	256	.370	.462	.309	.413	.230	.324	.228	.319
Local+ $L_{\mathrm{cons}}(.05)$	256	.375	.468	.313	.415	.231	.325	.228	.319

Appendix

Table 19: Retrieval results for Mistral, LLaMA, Gemma, and Qwen. Bold marks the strongest

256

-dimensional method for each model. Raw uses the native hidden dimension

d

; all other methods use

256

coordinates.

		GLM		DeepSeek		Falcon
Method	Dim.	R@1	mAP	R@1	mAP	R@1	mAP
Raw	$d$	.411	.505	.407	.491	.403	.498
PCA	256	.286	.375	.248	.330	.258	.344
Global	256	.363	.464	.368	.466	.314	.414
Local	256	.288	.386	.260	.353	.241	.333
Local+ $L_{\mathrm{cons}}(.05)$	256	.291	.389	.261	.355	.241	.332

Appendix

Table 20: Retrieval results for GLM, DeepSeek, and Falcon. Global is the strongest unmodified

256

-dimensional signature on all three models, substantially outperforming PCA at the same dimensionality.

	Mistral		LLaMA		Qwen		GLM
Style	Raw	Global	Raw	Global	Raw	Global	Raw	Global
Formal	.698	.681	.699	.611	.578	.505	.663	.555
Casual	.430	.422	.460	.422	.219	.228	.445	.366
Question	.680	.671	.654	.629	.552	.500	.679	.611
Technical	.423	.433	.403	.360	.333	.292	.398	.329
Simplified	.486	.469	.440	.369	.254	.223	.400	.307

Appendix

Table 21: R@1 by SP style for representative models. Global often provides its largest relative gains on harder nuisance transformations such as verbose, passive, and imperative reformulations.

Model	Signature	Base $\rightarrow$ FT		Base $\rightarrow$ Distill
		$\cos\uparrow$	$L_{2}\downarrow$	$\cos\uparrow$	$L_{2}\downarrow$
Mistral	Raw	.770	41.6	.932	22.5
	Global	.811	1.89	.950	.97
	Local	.817	.99	.954	.50
LLaMA	Raw	.906	26.8	.953	18.9
	Global	.942	1.11	.976	.72

Appendix

Table 22: Signature drift under fine-tuning and distillation. Atom signatures exhibit high cosine stability and substantially smaller Euclidean drift than raw hidden representations.

Model	Signature	Dim.	Base	Fine-tuned	Distilled
Mistral	Raw	4096	.878	.882	.887
	PCA	256	.849	.852	.855
	Global	256	.849	.852	.859
	Local	256	.833	.843	.843
	TopK	256	.809	.822	.823
LLaMA	Raw	4096	.892	.893	.891

Appendix

Table 23: Zero-shot SC-type classification after model modification. The classifier is trained only on base-model signatures and evaluated without retraining on fine-tuned and distilled variants. Atom-based signatures retain nearly unchanged semantic classification performance across modifications.

Model	Raw	PCA	Random	Global	Local	TopK
Mistral	1.000	.986	.984	.986	.974	.862
LLaMA	.999	.937	.940	.942	.907	.684
Gemma	.992	.879	.914	.898	.798	.587
Qwen	.993	.887	.891	.893	.852	.636
GLM	.998	.909	.916	.916	.866	.632
DeepSeek	.998	.958	.958	.965	.934	.769

Appendix

Table 24: Three-way model-source classification accuracy for base, fine-tuned, and distilled variants. Dense

256

-dimensional atom signatures retain substantial model-specific information, whereas aggressive TopK sparsification removes a larger fraction of that signal.

	Cos $\uparrow$		$S/T$ $\uparrow$		Nui. Energy $\downarrow$
Method	Qwen	Mistral	Qwen	Mistral	Qwen	Mistral
Local	.578	.540	1.60	1.87	2.27	2.21
Global	.562	.530	1.61	1.68	5.29	6.30
PCA	.574	.528	1.29	1.36	1160.1	1757.4
Random	.186	.172	1.32	1.54	134.4	160.0

Appendix

Table 25: External evaluation on 210 perturbation pairs pooled across three subsets. The learned atom frames achieve reconstruction cosines comparable to PCA, with higher semantic selectivity and lower nuisance response energies on both models.

Model	Frame	$E_{\mathrm{rec}}\downarrow$	$\mathrm{Cos}\uparrow$	$S/T\uparrow$	$D_{\mathrm{coord}}\downarrow$	$\mathrm{EffNum}$
Mistral	Global	0.701	0.703	1.27	1.49	21.5
	Local	0.689	0.714	1.42	1.58	21.0
LLaMA	Global	0.729	0.670	1.88	1.61	23.0
	Local	0.719	0.679	1.98	1.86	23.1
Gemma	Global	0.711	0.686	1.59	2.50	21.8
	Local	0.696	0.701	2.12	2.87	21.7

Appendix

Table 26: Full comparison between the shared global atom frame and local context-dependent diagonal scaling. Local improves semantic reconstruction and

S/T

across all seven model families while preserving approximately the same effective sparsity.

Model	Frame	$E_{\mathrm{rec}}\downarrow$	$\mathrm{Cos}\uparrow$	$S/T\uparrow$
Mistral	Global	0.701	0.703	1.27
	Local	0.689	0.714	1.42
	LowRank-QR	0.952	0.261	9.64
	LowRank-Soft	0.995	0.133	8.72
LLaMA	Global	0.728	0.670	1.88
	Local	0.719	0.679	1.98

Appendix

Table 27: Comparison with low-rank directional adaptation. The apparently large

S/T

values of LowRank-QR and LowRank-Soft coincide with severe reconstruction failure and therefore do not indicate improved semantic geometry.

	Mistral		Gemma
SC Type	Global	Local	Global	Local
Entity	0.68	0.84	0.65	1.14
Attribute	0.73	0.92	0.98	1.64
Relation	0.94	1.12	1.44	2.10
Negation	2.04	2.21	2.69	3.35
Quantity	1.26	1.41	1.60	2.21

Appendix

Table 28:

S/T

by semantic-change type for global and locally scaled atom frames. Local improves semantic selectivity broadly rather than specializing to a single SC category.

SP Style	Global	Local	Reduction
Formal	8.65	2.35	$3.7\times$
Casual	8.79	2.44	$3.6\times$
Question	6.44	1.75	$3.7\times$
Technical	11.65	2.74	$4.3\times$
Simplified	9.97	2.66	$3.7\times$
Verbose	11.65	2.93	$4.0\times$

Appendix

Table 29: Nuisance projection energy for Gemma-2-9B. The Local frame suppresses SP variation consistently across all eight paraphrase styles.

Model	Frame	$L_{\mathrm{val,sem}}$		$L_{\mathrm{val,nui}}$
		Epoch 1	Final	Epoch 1	Final
Mistral	Global	0.605	0.355	1.312	0.040
	Local	0.380	0.342	0.031	0.010
	LowRank-QR	0.715	0.616	0.716	0.332
	LowRank-Soft	0.767	0.616	0.014	0.257
Qwen	Global	0.599	0.355	1.910	0.043

Appendix

Table 30: Representative training dynamics for global, scaled, and directionally adapted atom frames. LowRank-Soft develops increasing nuisance energy during training, whereas LowRank-QR retains large nuisance error and weak semantic reconstruction.

Model	Layer
Mistral-7B-Instruct-v0.3	20
LLaMA-3.1-8B-Instruct	16
Gemma-2-9B	12
Qwen2.5-7B-Instruct	20
GLM-4-9B-Chat	20
DeepSeek-MoE-16B-Chat	20

Appendix

Table 31: Primary representation layer used for each model family.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

Invariant Features in Language Models: Geometric Characterization and Model Attribution
May 7, 2026Agnibh Dasgupta, Abdullah Tanvir, Xin ZhongNeural Representation Geometry Invariant Representation Learning
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA
Meaning Contracts: Local Semantic Geometry in Large Language Models
Oct 22, 2026Andrés Daniel Godoy OrtizRepresentation Geometry in Language Models LLM Interpretability
Imperial College London
Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models
Sep 29, 2026Abdullah All Tanvir, Xin ZhongInformation Geometry Neural Representation Geometry
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

May 7, 2026J/K move · Enter open · S save

Invariant Features in Language Models: Geometric Characterization and Model Attribution

Agnibh Dasgupta, Abdullah Tanvir, Xin Zhong

Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

Neural Representation Geometry Invariant Representation Learning Representation Geometry in Language Models

Open paper AskPDF arXiv

Language models exhibit strong robustness to paraphrasing, suggesting that semantic information may be encoded through stable internal representations, yet the structure and origin of such invariance remain unclear. We propose a local geometric framework in which semantically equivalent inputs occupy structured regions in latent space, with paraphrastic variation along nuisance directions and semantic identity preserved in invariant subspaces. Building on this view, we make three contributions: (1) a geometric characterization of invariant latent features, (2) a contrastive subspace discovery method that separates semantic-changing from semantic-preserving variation, and (3) an application of invariant representations to zero-shot model attribution. Across models and layers, empirical results support these contributions. Invariant structure emerges in specific depth regions, semantic displacement lies largely outside the nuisance subspace, and representation-level interventions indicate a causal role of invariant components in model outputs. Invariant representations also capture model-specific geometric patterns, enabling accurate attribution. These findings suggest that semantic invariance can be viewed as a local geometric property of latent representations, offering a principled perspective on how language models organize meaning.

Abstract

Representation Geometry in Language Models

Explore similar work

Invariant Features in Language Models: Geometric Characterization and Model Attribution

Meaning Contracts: Local Semantic Geometry in Large Language Models

Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models