SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprinting

Hanxiu Zhang; Qiang Xu; Yue Zheng

cs.CRDec 3, 2025

SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprinting

Authors: Hanxiu Zhang, Qiang Xu, Yue Zheng

Organizations: The Chinese University of Hong Kong, Shenzhen

Abstract

Safeguarding the Intellectual Property (IP) of Large Language Models (LLMs) against model theft represents a critical challenge in contemporary AI research. Although fingerprinting is essential for establishing forensic evidence for model derivation, existing methods are susceptible to false claim attacks, weight manipulations, or high false positive rates, undermining their utility. To overcome these limitations, we propose SELF, \rev{a novel intrinsic weight-based fingerprinting scheme tailored for robust IP forensics via adversarial model lineage verification}. SELF eliminates dependency on input and inherently resists false claims, achieving robust \rev{provenance detection} through unique, scalable and transformation-invariant fingerprint extraction via singular value and eigenvalue decomposition of LLM attention weights. Experimental results demonstrate that SELF maintains high IP infringement detection accuracy while significantly outperforming state-of-the-art baselines in reducing false positives and showing strong robustness against various downstream modifications, including quantization, pruning, and fine-tuning attacks. Our code is available at https://github.com/HanxiuZhang/SELF_v4.

Figures & tables

Fig. 1: Model provenance verification pipeline using SELF. After training the target model, the owner can extract the model fingerprint for lineage tracking. Even if the derived model has been modified to evade detection, the owner can still verify provenance by extracting and comparing the models’ fingerprints. A high fingerprint similarity provides evidence of derivation from the foundational target model.

Fig. 2: Fingerprint extraction via matrix decomposition. For a given model, we extract its fingerprint using the first

N_{\mathcal{F}}

Transformer block layers. Specifically, we first compute the fingerprint of each individual layer

\mathcal{F}^{i}

and then aggregate the

N_{\mathcal{F}}

layer-wise fingerprints to form the overall model fingerprint

\mathcal{F}

. For each layer, we first compute the singular value invariant matrix

X_{\sigma}^{i}

and the eigenvalue invariant matrix

X_{\lambda}^{i}

from the attention weights

W_{Q}^{i}

and

W_{K}^{i}

. Then, we extract the normalized singular value vector

\sigma^{i}

and eigenvalue vector

\lambda^{i}

, which together form the fingerprint of layer

i

Qwen2.5-7B Unrelated		Qwen2.5-7B Related		Llama2-7B Unrelated		Llama2-7B Related
Model	Distance	Model	Distance	Model	Distance	Model	Distance
Mistral-7B-V0.3	44.9266	Fine-tuned Variants		Mistral-7B-V0.3	34.5286	Fine-tuned Variants
Llama2-7B	57.4234	Qwen2.5-7B-Instruct	0.6330	Qwen1.5-7B	44.3750	Llama-2-7B-Chat	0.8223
Baichuan2-7B	70.0009	Qwen2.5-Math-7B	13.2143	Baichuan2-7B	48.5008	CodeLlama-7B	12.6612
InternLM2.5-7B	55.7048	Qwen2.5-Coder-7B	20.9766	InternLM2.5-7B	65.0354	Llemma-7B	12.7050
GPT2-Large	66.6810	TableGPT2-7B	1.3130	GPT2-Large	49.9396	Pruned Variants

TABLE I: Fingerprint

L_{1}

Distance (Target Models: Qwen2.5-7B and Llama2-7B)

	Mis	Qwe	Bai	Int	Gpt	Cer	Cha	OPT	Pyt	MPT
Mistral-7B-V0.3		40.6418	50.7201	61.4726	64.8229	76.6427	47.9270	119.4732	101.0111	62.9584
Qwen1.5-7B	40.6418		48.9473	53.5701	56.1560	75.6061	66.0833	112.9491	84.2514	63.4418
Baichuan2-7B	50.7201	48.9473		85.4194	53.6894	66.7556	66.2255	97.0414	116.8085	52.9459
InternLM2.5-7B	61.4726	53.5701	85.4194		99.1982	119.8395	94.6055	158.5690	46.9688	85.8946
GPT2-Large	64.8229	56.1560	53.6894	99.1982		43.3696	72.7921	73.1991	118.0726	52.0082
Cerebras-GPT-1.3B	76.6427	75.6061	66.7556	119.8395	43.3696		73.0203	60.1043	141.7880	61.5820

TABLE II: Fingerprint

L_{1}

Distance between Unrelated Models

Base Model	Distilled Model	Distance
Qwen2.5-Math-1.5B	DeepSeek-R1-Distill-Qwen-1.5B	4.7757
Qwen2.5-Math-7B	DeepSeek-R1-Distill-Qwen-7B	3.3650
Llama-3.1-8B	DeepSeek-R1-Distill-Llama-8B	4.7858

TABLE III: Fingerprint

L_{1}

Distance of DeepSeek models

Panel of Figure 6 — Fig. 3: Fingerprint distance and PPL change of Llama2-7B under fine-tuning attacks. (a) and (b) show the results under untargeted evasion attacks. (c) and (d) show the results under targeted evasion attacks. Since SOTA LSTM achieves PPL less than 60 on PTB dataset [ 19 ] , we consider a PPL of 60 or higher as “unacceptable” for larger models like Transformers.

Fig. 4: Fingerprint distance and PPL change of Llama2-7B under SliceGPT pruning.

Logit Difference	$W_{Q}$ $L_{1}$ Distance	$W_{K}$ $L_{1}$ Distance	Fingerprint $L_{1}$ Distance
$8.21\times 10^{-5}$	2,682,229	2,402,315	0.0049

TABLE IV: Experimental Results of the Permutation Attack

Layer	Logit Difference	$W_{Q}$ $L_{1}$ Distance	$W_{K}$ $L_{1}$ Distance	Fingerprint $L_{1}$ Distance
0	$9.16\times 10^{-5}$	319,859	144,190	0.0065
1	$6.77\times 10^{-5}$	525,442	208,674	0.0032
2	$1.88\times 10^{-4}$	568,894	328,554	0.0107
3	$1.21\times 10^{-4}$	582,215	326,985	0.0107
4	$2.68\times 10^{-4}$	611,205	329,017	0.0071
5	$6.62\times 10^{-4}$	641,005	367,997	0.0194

TABLE V: Experimental Results of Linear-Mapping Attack on OPT-1.3B

	Method	Requirement	Fingerprint size
HuRef-PCS	Parameter’s vector direction	All parameters	$\text{parameter\_num}~(\sim 10^{9})$
HuRef-ICS	Invariant terms’ similarity	Partial parameters and input	$3\times\text{selected\_layer\_num}\times(\text{sample\_num})^{2}~(\sim 10^{9})$
REEF	Representation’s CKA	Partial parameters and input	$\text{sample\_num}\times d_{\text{model}}~(\sim 10^{4}-10^{6})$
SELF	Invariant matrix decomposition	Partial parameters	$2N_{\mathcal{F}}\times h~(\sim 10^{3})$

TABLE VI: Comparison of Mechanism and Fingerprint Size

Methods	Model Fine-tuning	Model Pruning	Permutation Attack	Linear-mapping Attack	False Claim Attack
HuRef-PCS	$\checkmark$				$\checkmark$
HuRef-ICS	$\checkmark$		$\checkmark$	$\checkmark$
REEF	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
SELF	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE VII: Comparison of Robustness Against Various Attacks

	Fine-tuning
	Llama-2-finance-7B (5M Tokens)	Vicuna-1.5-7B (370M Tokens)	Wizardmath-7B (1.8B Tokens)	Chinesellama-2-7B (13B Tokens)	Codellama-7B (500B Tokens)	Llemma-7B (700B Tokens)
HuRef-PCS	0.9979	0.9985	0.0250	0.0127	0.0105	0.0098
HuRef-ICS	0.9952	0.9949	0.9994	0.4996	0.2550	0.2257
REEF	0.9950	0.9985	0.9979	0.9974	0.9947	0.9962
SELF (Distance)	0.8504	0.8598	0.4559	1.2147	12.6612	12.7050

TABLE VIII: Comparison of fingerprint similarity on Llama2-7B fine-tuned models

	Structured Pruning
	Sheared-Llama- 1.3B-pruned	Sheared-Llama- 1.3B	Sheared-Llama- 1.3B-sharegpt	Sheared-Llama- 2.7B-pruned	Sheared-Llama- 2.7B	Sheared-Llama- 2.7B-sharegpt
HuRef-PCS	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
HuRef-ICS	0.4927	0.3512	0.3510	0.6055	0.4580	0.4548
REEF	0.9368	0.9676	0.9710	0.9278	0.9701	0.9991
SELF (Distance)	23.2506	16.8061	16.7971	11.0471	9.8122	11.0611

TABLE IX: Comparison of fingerprint similarity on Llama2-7B structured pruned models

Unstructured Pruning
	Sparse-Llama- 2-7B	Wanda-Llama- 2-7B	GBLM-Llama- 2-7B
HuRef-PCS	0.9560	0.9620	0.9616
HuRef-ICS	0.9468	0.9468	0.9478
REEF	0.9985	0.9986	0.9991
SELF (Distance)	2.7556	3.0326	3.1229

TABLE X: Comparison of fingerprint similarity score on Llama2-7B unstructured pruned models

	Shisa-gamma- 7B-v1	Wizard-math- 7B-1.1	Abel- 7B-002
HuRef-PCS	0.9992	0.9990	0.9989
HuRef-ICS	0.9992	0.9988	0.9988
REEF	0.9635	0.9526	0.9374
SELF (Distance)	2.1069	2.1487	2.1867

TABLE XI: Comparison of fingerprint similarity between EvoLLM-jp-7B and its parent models

Method	AUC	ACC	TPR@1%FPR	MD
HuRef-ICS	0.987	0.983	0.888	2.547
REEF	0.878	0.894	0.627	1.993
SELF	0.986	0.996	0.979	2.585

TABLE XII: Comparison of Evaluation on LeafBench

Panel of Figure 17 — Fig. 5: Distance Distribution with Different Metrics on LeafBench

Fig. 6: Accuracy on LeafBench with Different Thresholds

Invariant Matrix	First 8	Middle 8	Last 8
Singular Value	3.7414	0.7141	-5.3285
Eigenvalue	7.5367	-1.6868	-7.0671
Both	11.2780	-0.9727	-10.3403

TABLE XIII: Fingerprint Margin with Different Layer Selections

Invariant Matrix	$W_{Q},W_{K}$	$W_{V},W_{O}$	Both
Singular Value	3.7414	-3.5293	1.6019
Eigenvalue	7.5367	-6.5116	6.1064
Both	11.2780	-8.0503	7.7083

TABLE XIV: Fingerprint Margin with Different Weight Selections

Invariant Matrix	32	64	128	256
Singular Value	-0.1411	0.5282	2.5339	3.7414
Eigenvalue	-1.0599	0.9153	4.2879	7.5367
Both	-1.0805	2.3152	7.8773	11.2780

TABLE XV: Fingerprint Margin under Varying

h

Invariant Matrix	2	4	6	8
Singular Value	-1.6139	1.9465	3.4775	3.7414
Eigenvalue	-2.3803	1.1200	4.9059	7.5367
Both	-3.4706	3.0666	8.9341	11.2780

TABLE XVI: Fingerprint Margin under Varying

N_{\mathcal{F}}

Method	Dataset	Mistral-7B	MPT-7B	InternLM2.5	GPT2-Large	Cerebras-GPT	Pythia-6.9B	OPT-6.7B
REEF	TruthfulQA	0.1975	0.2111	0.1942	0.2320	0.2257	0.2307	0.2437
	ConfAIde	0.2340	0.2449	0.2308	0.2464	0.2471	0.2723	0.2419
	PKU-SafeRLHF	0.5099	0.5150	0.4307	0.4808	0.4390	0.4844	0.4764
	ToxiGen	0.5894	0.5528	0.6067	0.5366	0.6358	0.6151	0.4716
	SST2	0.6772	0.6221	0.6628	0.6034	0.5945	0.6783	0.5982
SELF (Distance)		34.5286	44.6246	65.0354	49.9396	67.3757	102.5610	108.3884

TABLE XVII: Comparison of Llama2-7B unrelated models fingerprint similarity with different input datasets

Fig. 7: HuRef calculation process in our false claim attack.

Panel of Figure 25 — Fig. 8: False-claim attack on HuRef: both the feature vector and the human-readable fingerprint of the unrelated model (Qwen1.5-7B) become highly similar to that of the target model (Llama2-7B).

Layer	Llama-2-7B-chat				CodeLlama-7B
Layer	$\boldsymbol{\\|\Delta W_{Q}\\|_{2}}$	$\boldsymbol{\\|\Delta W_{K}\\|_{2}}$	$\boldsymbol{\\|\Delta X_{\sigma}\\|_{2}}$	$\boldsymbol{\\|\Delta X_{\lambda}\\|_{2}}$	$\boldsymbol{\\|\Delta W_{Q}\\|_{2}}$	$\boldsymbol{\\|\Delta W_{K}\\|_{2}}$	$\boldsymbol{\\|\Delta X_{\sigma}\\|_{2}}$	$\boldsymbol{\\|\Delta X_{\lambda}\\|_{2}}$
0	0.6441	0.9481	14.9893	18.6290	10.2179	16.3403	308.7895	414.0640
1	0.5998	1.0492	19.8387	27.8605	12.1084	15.3005	405.8361	526.0145
2	0.4577	0.6067	16.3128	20.4678	7.0955	7.2867	408.6551	438.2098
3	0.4623	0.5108	13.0404	17.4805	6.0730	6.3525	343.4302	387.0950
4	0.4217	0.4619	14.5159	17.8721	6.5702	6.7162	370.9959	403.0934

TABLE XVIII:

L_{2}

norm differences of weight matrices (

W_{Q}

W_{K}

) and invariant matrices (

X_{\sigma}

X_{\lambda}

) between Llama-2-7B base and its fine-tuned variants (First 8 layers).

Layer	Llama-2-7B-chat					CodeLlama-7B
Layer	$\boldsymbol{\\|\Delta\tilde{\sigma}\\|_{1}}$	$\boldsymbol{\\|\Delta\tilde{\lambda}\\|_{1}}$	$\boldsymbol{\\|\Delta\sigma\\|_{1}}$	$\boldsymbol{\\|\Delta\lambda\\|_{1}}$	$\boldsymbol{\\|\Delta\mathcal{F}^{i}\\|_{1}}$	$\boldsymbol{\\|\Delta\tilde{\sigma}\\|_{1}}$	$\boldsymbol{\\|\Delta\tilde{\lambda}\\|_{1}}$	$\boldsymbol{\\|\Delta\sigma\\|_{1}}$	$\boldsymbol{\\|\Delta\lambda\\|_{1}}$	$\boldsymbol{\\|\Delta\mathcal{F}^{i}\\|_{1}}$
0	29.6924	18.0723	0.0200	0.0978	0.1178	507.6064	257.8611	1.9152	1.3635	3.2786
1	102.5995	55.8582	0.0254	0.1169	0.1423	436.0782	172.6528	1.0252	0.8056	1.8307
2	84.8931	41.9358	0.0168	0.0894	0.1062	2196.6628	1199.6815	0.4681	0.9231	1.3912
3	57.7201	30.8182	0.0159	0.0607	0.0765	2075.8862	1122.6079	0.7942	1.0659	1.8601
4	61.4231	25.2875	0.0133	0.0907	0.1040	2110.1177	992.6726	0.4244	1.0701	1.4945

TABLE XIX:

L_{1}

norm differences of raw spectra (

\tilde{\sigma}

\tilde{\lambda}

), normalized spectra (

\sigma

\lambda

), and layer-wise model fingerprints (

\mathcal{F}^{i}

) (First 8 layers).

Layer	Mean			Max			Min
Layer	Llama2	Chat	Code	Llama2	Chat	Code	Llama2	Chat	Code
0	35.1765	31.1541	35.1168	170.0865	113.2376	103.7457	2.6866	2.7253	4.3320
1	20.9765	21.7992	26.6926	89.5765	78.0695	228.4924	3.4169	3.2861	2.2162
2	21.6719	26.3377	22.4314	62.7738	183.8452	170.6535	2.3612	2.3520	2.1549
3	31.8259	20.7790	21.1383	930.7057	177.2809	82.8434	1.9210	1.9385	1.7110
4	25.4155	22.7667	30.4696	217.4204	98.1534	155.5914	1.9837	2.0088	1.7797

TABLE XX: Condition number statistics of the top-256 eigenvalues for the invariant matrix

X_{\lambda}

across Llama2-7B, Llama2-7B-chat, and CodeLlama-7B.

Language Model Fingerprinting

Language model fingerprinting identifies and verifies models or their lineage from distinctive outputs, internal representations, or embedded signals.

See all 81 papers

Jul 15, 2024

Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique

Growing concerns over the theft and misuse of Large Language Models (LLMs) underscore the need for effective fingerprinting to link a model to its original version and detect misuse. We define five essential properties for a successful fingerprint: Transparency, Efficiency, Persistence, Robustness, and Unforgeability. We present a novel fingerprinting framework that provides verifiable proof of ownership while preserving fingerprint integrity. Our approach makes two main contributions. First, a chain and hash technique that cryptographically binds fingerprint prompts to their responses, preventing collisions and enabling irrefutable ownership claims. Second, we address a realistic threat model in which instruction-tuned models' output distribution can be significantly altered through meta-prompts. By incorporating random padding and varied meta-prompt configurations during training, our method maintains robustness even under significant output style changes. Experiments show that our framework securely proves ownership, resists both benign transformations (e.g., fine-tuning) and adversarial fingerprint removal, and extends to fingerprinting LoRA adapters\footnote{We release our code at: https://github.com/microsoft/Chain-Hash.

Mark Russinovich, Yanan Cai, Ahmed Salem

Microsoft

Language Model Fingerprinting

Ask PDF

Apr 7, 2026

AttnDiff: Attention-based Differential Fingerprinting for Large Language Models

Protecting the intellectual property of open-weight large language models (LLMs) requires verifying whether a suspect model is derived from a victim model despite common laundering operations such as fine-tuning (including PPO/DPO), pruning/compression, and model merging. We propose AttnDiff, a data-efficient white-box framework that extracts fingerprints from models via intrinsic information-routing behavior. AttnDiff probes minimally edited prompt pairs that induce controlled semantic conflicts, captures differential attention patterns, summarizes them with compact spectral descriptors, and compares models using CKA. Across Llama-2/3 and Qwen2.5 (3B--14B) and additional open-source families, it yields high similarity for related derivatives while separating unrelated model families (e.g.,

>0.98

vs.\

<0.22

with

M=60

probes). With 5--60 multi-domain probes, it supports practical provenance verification and accountability.

Haobo Zhang, Zhenhua Xu, Junxian Li +3

Zhejiang University of Technology · Binjiang Institute of Zhejiang University · Zhejiang University +2

Language Model Fingerprinting Data Provenance in Language Models Fingerprint Identification

Ask PDF

Apr 18, 2026

SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models

The public accessibility of large vision-language models (LVLMs) raises serious concerns about unauthorized model reuse and intellectual property infringement. Existing ownership verification methods often rely on semantically abnormal queries or out-of-distribution responses as fingerprints, which can be easily detected and removed by adversaries. We expose this vulnerability through a Semantic Divergence Attack (SDA), which identifies and filters fingerprint queries by measuring semantic divergence between a suspect model and a reference model, showing that existing fingerprints are not semantic-preserving and are therefore easy to detect and bypass. To address these limitations, we propose SIF (Semantically In-Distribution Fingerprints), a non-intrusive ownership verification framework that requires no parameter modification. SIF introduces Semantic-Aligned Fingerprint Distillation (SAFD), which transfers text watermarking signals into the visual modality to produce semantically coherent yet fingerprinted responses. In addition, Robust-Fingerprint Optimization (RFO) enhances robustness by simulating worst-case representation perturbations, making the fingerprints resilient to model modifications such as fine-tuning and quantization. Extensive experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate that SIF achieves strong stealthiness and robustness, providing a practical solution for LVLM copyright protection. Code is available at https://github.com/UCF-ML-Research/SIF-VLM-Fingerprint

Yifei Zhao, Qian Lou, Mengxin Zheng

University of Central Florida

VLM Robustness Language Model Fingerprinting Large Vision-Language Models LLM Watermarking

Ask PDF

Abstract

Language Model Fingerprinting

Explore similar work

Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique

AttnDiff: Attention-based Differential Fingerprinting for Large Language Models

SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models