Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models

Abdullah All Tanvir; Xin Zhong

cs.LGSep 29, 2026

Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models

Authors: Abdullah All Tanvir, Xin Zhong

Organizations: Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

Abstract

Semantic-preserving transformations can induce substantial motion in learned representations, while small changes may strongly affect model predictions, raising a basic question: what local metric best captures semantically consequential variation? We propose Fisher-induced invariant representation geometry (Fisher-IRG), which measures local representation directions through their predictive sensitivity. Around each representation, we construct semantic-preserving and semantic-changing neighborhoods, aggregate their local Fisher information, and recover invariant directions through a contrastive generalized eigenvalue problem. Controlled displacement analyses first show that comparable Euclidean motion can have substantially different predictive consequences, supporting the need for a predictive geometry. Across language and vision models, Fisher-IRG yields stronger semantic-versus-nuisance predictive selectivity and generally more reproducible subspaces than covariance-based geometry, while recovering systematically distinct local directions. Representation interventions further localize semantic effects to the Fisher-derived subspace, and held-out separation and retrieval show that the recovered geometry generalizes beyond the discovery neighborhoods. These results support Fisher-IRG as a principled framework for characterizing local invariant representation geometry.

Figures & tables

Figure 1: Fisher-induced invariant representation geometry. (A) Surface-level variation and semantic change induce different local motions in language and vision representations. (B) Fisher information weights local directions by predictive sensitivity, characterizing the local geometry around an anchor. (C) Contrastive Fisher analysis recovers an invariant subspace in which surface-level variations map consistently while semantically meaningful directions are preserved.

Figure 2: From representation manifolds to predictive local geometry. A model maps high-dimensional inputs onto a representation manifold, whose tangent space captures infinitesimal local motion around an anchor $z$ . A predictive metric $G(z)$ then reweights these tangent directions by their effect on model behavior, inducing anisotropic local geometry that differs from the Euclidean metric.

Figure 3: Example visual neighborhood construction. SP variants preserve the source semantics, whereas easy, medium, and hard SC variants introduce progressively larger semantic modifications. Red circles manually highlight the changes.

Model	Rate	Layers	$\rho_{\mathrm{KL}}$
Language
Mistral	24.1%	32/32	2.03–16.99 $\times$
Qwen	12.6%	27/28	2.84–7.57 $\times$
LLaMA	5.8%	22/32	2.03–4.27 $\times$
Gemma	17.1%	28/28	2.40–187.50 $\times$
Vision

Table 1: Predictive-consequence mismatch among distance-matched perturbations. Rate is the fraction of matched cases with

\rho_{\mathrm{KL}}\geq 2

; Layers count layers containing such cases.

Model	Mean $\Delta S\uparrow$
Language
LLaMA	0.218
Mistral	0.230
Gemma	0.170
Qwen	0.368
DeepSeek	0.248

Table 2: Predictive-selectivity advantage of Fisher-IRG over Cov-IRG. Mean

\Delta S

is averaged over five normalized depths.

Language
Model	Fisher $\downarrow$	Cov $\downarrow$
Mistral	2.459	2.535
LLaMA	2.553	2.644
Gemma	2.732	2.781
Qwen	2.672	2.811
DeepSeek	2.331	2.456

Table 3: Grassmann reproducibility of Fisher-IRG and Cov-IRG. Distances are averaged over three layers and

k\in\{8,16,32\}

Language
Model	$d(F,C)\uparrow$	Prin. Cos. $\downarrow$	$\Delta_{\mathrm{geo}}\uparrow$
Mistral	.998	.064	.559
LLaMA	.997	.070	.484
Gemma	.998	.069	.549
Qwen	.997	.056	.559
DeepSeek	.998	.071	.507

Table 4: Geometric separation between Fisher- and Cov-IRG.

d(F,C)

is the cross-method projection distance;

\Delta_{\mathrm{geo}}

is its excess over within-method variation.

Setting	$\overline{D}_{\mathrm{SP,inv}}$	$\overline{D}_{\mathrm{SC,nuis}}$	$\overline{D}_{\mathrm{SC,inv}}$
Language	$.003$	$2.040$	$3.790$
Vision	$.002$	$.187$	$.571$

Table 5: Causal localization under Fisher-IRG intervention. Mean divergences summarize intervention strength; counts and ranges summarize the two causal gaps.

Setting	Fisher Wins	Fisher $\Delta_{\mathrm{sep}}$ Range
Language	21/21	$.000434$ – $.021707$
Vision	9/9	$.026423$ – $.724569$

Table 6: Held-out semantic separation. Wins counts settings in which Fisher-IRG achieves the largest

\Delta_{\mathrm{sep}}

among Fisher-IRG, Cov-IRG, and the original representation.

Representation	Pairwise $\uparrow$	MRR $\uparrow$	Hit@1 $\uparrow$
Original	.949	.921	.873
Random	.916	.876	.803
PCA	.935	.896	.829
cPCA	.945	.905	.841
Cov-IRG	.914	.872	.791
Fisher-IRG	.959	.931	.885

Table 7: Held-out semantic retrieval averaged across seven language models.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Text
Anchor	The city council approved the proposal to build a new public library downtown.
SP-1	The proposal for constructing a new public library downtown was approved by the city council.
SP-2	City council members authorized the plan to construct a new downtown public library.
SP-3	The council approved plans for a new public library in the downtown area.
Easy SC	The city council rejected the proposal to build a new public library downtown.
Hard SC	The city council approved the proposal to renovate an existing private library outside the downtown area.

Appendix

Table 8: Example text neighborhood containing an anchor, representative semantic-preserving variants, and easy and hard semantic-changing variants. The SP variants alter wording and sentence structure while preserving the anchor’s meaning. The easy SC variant introduces an explicit change in the council’s decision, whereas the hard SC variant retains much of the anchor’s surface form while changing several related semantic attributes.

Figure 4: Additional examples of visual semantic neighborhoods. Each source image is paired with SP variants and SC variants at easy, medium, and hard difficulty levels. Red circles manually highlight the modified regions for readability.

Modality	Anchors	SP	SC	Total / anchor
Text	1,800	10	64	75
Vision	1,000	10	64	75

Appendix

Table 9: Composition of the controlled semantic neighborhoods used for language and vision experiments. Variant counts are reported per anchor.

Modality	Model	Scale / training characteristic
Text	Mistral-Instruct-v0.3	7B
Text	LLaMA-3-Instruct	8B
Text	Gemma-IT	7B
Text	Qwen2.5-Instruct	7B
Text	DeepSeek-MoE-Chat	16B, mixture-of-experts
Text	Phi-4-Mini	4B

Appendix

Table 10: Language and vision models used in the experiments. The suite spans different model families, scales, architectures, and representation-learning objectives.

Layer	Matched	Qualifying	Fraction	Median L2 Gap	Median KL Ratio
Mistral
1	109	50	45.87%	2.57%	16.99 $\times$
2	210	17	8.10%	0.08%	2.40 $\times$
3	210	11	5.24%	0.19%	2.36 $\times$
4	210	19	9.05%	1.00%	2.23 $\times$
5	210	30	14.29%	3.46%	3.14 $\times$

Appendix

Table 11: Complete layer-wise displacement–consequence separability results for language models. Matched counts semantic groups satisfying

\delta_{\mathrm{L2}}\leq 0.10

; Qualifying additionally requires

\rho_{\mathrm{KL}}\geq 2

; Fraction is Qualifying/Matched. Median L2 Gap and Median KL Ratio are computed only over qualifying groups.

Layer	Matched	Qualifying	Fraction	Median L2 Gap	Median KL Ratio
1	348	72	20.69%	3.14%	3.04 $\times$
2	410	96	23.41%	2.25%	3.20 $\times$
3	406	32	7.88%	1.77%	2.72 $\times$
4	409	16	3.91%	2.38%	2.72 $\times$
5	416	26	6.25%	1.65%	2.75 $\times$
6	404	20	4.95%	1.68%	2.98 $\times$

Appendix

Table 12: Complete layer-wise displacement–consequence separability results for ViT-B/16 using CLS-token representations. Column definitions are identical to Table 11 .

Language
Model	$0.00$	$0.25$	$0.50$	$0.75$	$1.00$
LLaMA-3-Instruct	0.140	0.270	0.370	0.190	0.120
Mistral-Instruct-v0.3	0.120	0.330	0.170	0.220	0.310
Gemma-IT	0.260	0.170	0.120	0.090	0.210
Qwen2.5-Instruct	0.280	0.340	0.470	0.520	0.230
DeepSeek-MoE-Chat	0.220	0.410	0.180	0.240	0.190

Appendix

Table 13: Full layer-wise predictive-selectivity advantage of Fisher-IRG over Cov-IRG. Each entry reports

\Delta S=S^{\mathrm{Fisher}}-S^{\mathrm{Cov}}

, averaged across semantic groups. Positive values indicate stronger semantic-versus-nuisance predictive selectivity under Fisher-IRG.

Model	$k$	Layer 3	Layer 8	Layer 10
Mistral-Instruct-v0.3	8	2.427/2.593	2.381/2.557	2.439/2.603
	16	2.404/2.612	2.639/2.763	2.480/2.538
	32	2.352/2.396	2.307/2.447	2.699/2.307
LLaMA-3-Instruct	8	2.422/2.615	2.685/2.715	2.606/2.767
	16	2.556/2.629	2.426/2.536	2.560/2.583
	32	2.621/2.594	2.694/2.728	2.411/2.626

Appendix

Table 14: Full Grassmann reproducibility comparison for language models. Each entry reports Fisher-IRG/Cov-IRG Grassmann distance, averaged across semantic groups and independent run pairs. Lower is better.

Model	$k$	Layer 3	Layer 8	Layer 10
ViT-B/16	8	3.238/3.430	3.334/3.389	3.309/3.371
	16	3.329/3.405	3.346/3.412	3.199/3.254
	32	3.235/3.334	3.340/3.424	3.217/3.313
DINOv2 ViT-B/14	8	3.388/3.582	3.152/3.208	3.415/3.523
	16	3.226/3.362	3.598/3.791	3.112/3.356
	32	3.298/3.402	3.352/3.388	3.256/3.480

Appendix

Table 15: Full Grassmann reproducibility comparison for vision models. Each entry reports Fisher-IRG/Cov-IRG Grassmann distance. All results use CLS-token representations. Lower is better.

Model	$k$	$C_{\mathrm{sub}}\uparrow$	$C_{\mathrm{vec}}\uparrow$	$\Delta C$
Mistral-Instruct-v0.3	8	.6154	.3636	.2518
	16	.6228	.3458	.2770
	32	.7431	.1489	.5942
LLaMA-3-Instruct	8	.6655	.3840	.2815
	16	.6589	.2736	.3853
	32	.7238	.2035	.5203

Appendix

Table 16: Subspace-level versus individual-vector reproducibility of Fisher-IRG for language models. Results are averaged across evaluated layers, semantic groups, and independent run pairs.

Model	$k$	SC	$C_{\mathrm{sub}}\uparrow$	$C_{\mathrm{vec}}\uparrow$	$\Delta C$
ViT-B/16	8	Easy	.6233	.2714	.3519
		Medium	.6064	.2374	.3690
		Hard	.5554	.1936	.3618
	16	Easy	.6504	.2694	.3810
		Medium	.6301	.2518	.3783
		Hard	.6384	.2603	.3781

Appendix

Table 17: Subspace-level versus individual-vector reproducibility of Fisher-IRG for vision models across semantic-change difficulty and subspace dimension. Results use CLS-token representations and are averaged across evaluated layers, Visual Genome groups, and independent run pairs.

Language
Model	$d(F,F^{\prime})\downarrow$	$d(C,C^{\prime})\downarrow$	$d(F,C)\uparrow$	Prin. Cos. $\downarrow$	$\Delta_{\mathrm{geo}}\uparrow$	Pos. Groups
Mistral-Instruct-v0.3	.410	.439	.998	.064	.559	100%
LLaMA-3-Instruct	.381	.513	.997	.070	.484	100%
Gemma-IT	.417	.449	.998	.069	.549	100%
Qwen2.5-Instruct	.413	.438	.997	.056	.559	100%
DeepSeek-MoE-Chat	.473	.491	.998	.071	.507	100%

Appendix

Table 18: Direct geometric comparison between Fisher-IRG and Cov-IRG.

d(F,F^{\prime})

and

d(C,C^{\prime})

measure within-method variation across independent discovery splits, while

d(F,C)

measures cross-method separation. Principal Cos. measures mean subspace overlap, and

\Delta_{\mathrm{geo}}

is the cross-method distance minus the larger within-method distance. Positive Groups reports the fraction of semantic groups for which the cross-method distance exceeds both within-method distances.

Model / Group	$D_{\mathrm{SP,nuis}}\downarrow$	$D_{\mathrm{SC,nuis}}\downarrow$	$D_{\mathrm{SP,inv}}\downarrow$	$D_{\mathrm{SC,inv}}\uparrow$	$\Delta_{\mathrm{causal}}\uparrow$	$\Delta_{\mathrm{local}}\uparrow$
Mistral-Instruct-v0.3 / 1	.041859	.069642	.000323	.084293	.083970	.014651
Mistral-Instruct-v0.3 / 500	.028741	1.214668	.000060	3.002461	3.002401	1.787793
Mistral-Instruct-v0.3 / 1000	.018359	1.649377	.000019	2.001373	2.001354	.351996
LLaMA-3-Instruct / 1	.033231	3.677786	.000048	4.021666	4.021618	.343880
LLaMA-3-Instruct / 500	.008527	1.752320	.000030	1.915718	1.915688	.163398
LLaMA-3-Instruct / 1000	.005825	1.000402	.000194	1.010367	1.010173	.009965

Appendix

Table 19: Group-level causal intervention results for language models.

D_{\mathrm{SP},\mathrm{nuis}}

and

D_{\mathrm{SC},\mathrm{nuis}}

replace the complementary component using SP and SC donors, while

D_{\mathrm{SP},\mathrm{inv}}

and

D_{\mathrm{SC},\mathrm{inv}}

replace the Fisher-IRG component. Positive

\Delta_{\mathrm{causal}}

and

\Delta_{\mathrm{local}}

indicate semantic selectivity and localization, respectively. Results use mean-pooled representations at layer 10 with

k=16

SC / Group	$D_{\mathrm{SP,nuis}}\downarrow$	$D_{\mathrm{SC,nuis}}\downarrow$	$D_{\mathrm{SP,inv}}\downarrow$	$D_{\mathrm{SC,inv}}\uparrow$	$\Delta_{\mathrm{causal}}\uparrow$	$\Delta_{\mathrm{local}}\uparrow$
Easy / 300	.386655	.195308	.003225	.459543	.456318	.264235
Easy / 550	.114132	.179609	.001291	.625532	.624241	.445923
Easy / 800	.231401	.243617	.003217	.684134	.680917	.440517
Medium / 300	.372400	.037511	.003132	.107466	.104334	.069955
Medium / 550	.114486	.212069	.001107	1.006177	1.005070	.794108
Medium / 800	.314213	.114786	.001499	.741347	.739848	.626561

Appendix

Table 20: Group-level causal intervention results for ViT-B/16 across semantic-change difficulty. Results use CLS-token representations at layer 10 with

k=16

. Positive causal and localization gaps indicate that the Fisher-IRG component responds more strongly to semantic-changing than semantic-preserving replacement and more strongly than the complementary component under the same SC intervention.

		Original			Fisher-IRG			Cov-IRG
Model	Layer	$d_{\mathrm{SP}}\downarrow$	$d_{\mathrm{SC}}\uparrow$	$\Delta_{\mathrm{sep}}\uparrow$	$d_{\mathrm{SP}}\downarrow$	$d_{\mathrm{SC}}\uparrow$	$\Delta_{\mathrm{sep}}\uparrow$	$d_{\mathrm{SP}}\downarrow$	$d_{\mathrm{SC}}\uparrow$	$\Delta_{\mathrm{sep}}\uparrow$
Mistral-Instruct-v0.3	3	.000019	.000073	.000053	.000129	.000563	.000434	.000004	.000019	.000015
	8	.000280	.001212	.000932	.003044	.015950	.012906	.000062	.000362	.000300
	10	.000421	.001851	.001430	.005120	.026828	.021707	.000094	.000568	.000474
Llama-3-Instruct	3	.000058	.000196	.000137	.000175	.000743	.000568	.000015	.000059	.000043
	8	.000368	.001709	.001341	.000788	.004143	.003356	.000081	.000497	.000416

Appendix

Table 21: Held-out semantic separation for language models.

d_{\mathrm{SP}}

and

d_{\mathrm{SC}}

denote cosine distances from the anchor to held-out semantic-preserving and semantic-changing variants, respectively, and

\Delta_{\mathrm{sep}}=d_{\mathrm{SC}}-d_{\mathrm{SP}}

. Subspaces are estimated from discovery variants and evaluated on disjoint held-out variants. Results use mean-pooled representations and

k=16

and are averaged across semantic groups.

		Original			Fisher-IRG			Cov-IRG
SC Difficulty	Layer	$d_{\mathrm{SP}}\downarrow$	$d_{\mathrm{SC}}\uparrow$	$\Delta_{\mathrm{sep}}\uparrow$	$d_{\mathrm{SP}}\downarrow$	$d_{\mathrm{SC}}\uparrow$	$\Delta_{\mathrm{sep}}\uparrow$	$d_{\mathrm{SP}}\downarrow$	$d_{\mathrm{SC}}\uparrow$	$\Delta_{\mathrm{sep}}\uparrow$
Easy	3	.004884	.003291	-.001592	.007024	.033447	.026423	.006320	.003236	-.003084
	8	.032660	.054270	.021609	.022415	.551965	.529550	.042919	.072215	.029296
	10	.121349	.229988	.108639	.040845	.764698	.723853	.070417	.213771	.143354
Medium	3	.004889	.004332	-.000556	.007745	.043607	.035862	.005922	.004327	-.001595
	8	.032724	.068977	.036253	.017568	.595776	.578208	.039122	.090228	.051106

Appendix

Table 22: Held-out semantic separation for ViT-B/16 across semantic-change difficulty and representative layers. Fisher-IRG achieves the largest separation margin in every reported configuration.

Model	Representation	Pairwise $\uparrow$	MRR $\uparrow$	Hit@1 $\uparrow$
Mistral-Instruct-v0.3	Original	.974	.961	.942
	Random	.965	.921	.850
	PCA	.961	.934	.889
	cPCA	.979	.959	.928
	Cov-IRG	.957	.902	.810
	Fisher-IRG	.985	.975	.957

Appendix

Table 23: Held-out semantic retrieval across language models. Results use mean-pooled representations at layer 10 with

k=16

and are averaged over 1,800 semantic groups. Bold indicates the best result for each model and metric.

Explore similar work

Invariant Features in Language Models: Geometric Characterization and Model Attribution
May 7, 2026Agnibh Dasgupta, Abdullah Tanvir, Xin ZhongLanguage Modeling Latent Space
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA
FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers
May 17, 2026Sihan Wang, Jiayi ZhaoLinear Activation Steering Steering
South China University of Technology
Scale Determines Whether Language Models Organize Representation Geometry for Prediction
May 16, 2026Weilun XuSubspace Representation Space
School of Computer and Communication Sciences ´Ecole Polytechnique F´ed´erale de Lausanne

May 7, 2026, cs.LGJ/K move · Enter open · S save

Invariant Features in Language Models: Geometric Characterization and Model Attribution

Agnibh Dasgupta, Abdullah Tanvir, Xin Zhong

Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

Language Modeling Latent Space

Open paper AskPDF arXiv

Language models exhibit strong robustness to paraphrasing, suggesting that semantic information may be encoded through stable internal representations, yet the structure and origin of such invariance remain unclear. We propose a local geometric framework in which semantically equivalent inputs occupy structured regions in latent space, with paraphrastic variation along nuisance directions and semantic identity preserved in invariant subspaces. Building on this view, we make three contributions: (1) a geometric characterization of invariant latent features, (2) a contrastive subspace discovery method that separates semantic-changing from semantic-preserving variation, and (3) an application of invariant representations to zero-shot model attribution. Across models and layers, empirical results support these contributions. Invariant structure emerges in specific depth regions, semantic displacement lies largely outside the nuisance subspace, and representation-level interventions indicate a causal role of invariant components in model outputs. Invariant representations also capture model-specific geometric patterns, enabling accurate attribution. These findings suggest that semantic invariance can be viewed as a local geometric property of latent representations, offering a principled perspective on how language models organize meaning.

Abstract

Explore similar work

Invariant Features in Language Models: Geometric Characterization and Model Attribution

FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers

Scale Determines Whether Language Models Organize Representation Geometry for Prediction