AttnDiff: Attention-based Differential Fingerprinting for Large Language Models

Haobo Zhang; Zhenhua Xu; Junxian Li; Shangfeng Sheng; Dezhang Kong; Meng Han

cs.CRApr 7, 2026

AttnDiff: Attention-based Differential Fingerprinting for Large Language Models

Authors: Haobo Zhang, Zhenhua Xu, Junxian Li, Shangfeng Sheng, Dezhang Kong, Meng Han

Organizations: Zhejiang University of Technology · Binjiang Institute of Zhejiang University · Zhejiang University · Shanghai Jiao Tong University ·

Abstract

Protecting the intellectual property of open-weight large language models (LLMs) requires verifying whether a suspect model is derived from a victim model despite common laundering operations such as fine-tuning (including PPO/DPO), pruning/compression, and model merging. We propose AttnDiff, a data-efficient white-box framework that extracts fingerprints from models via intrinsic information-routing behavior. AttnDiff probes minimally edited prompt pairs that induce controlled semantic conflicts, captures differential attention patterns, summarizes them with compact spectral descriptors, and compares models using CKA. Across Llama-2/3 and Qwen2.5 (3B--14B) and additional open-source families, it yields high similarity for related derivatives while separating unrelated model families (e.g., $>0.98$ vs.\ $<0.22$ with $M=60$ probes). With 5--60 multi-domain probes, it supports practical provenance verification and accountability.

Figures & tables

Figure 1: AttnDiff pipeline. Left: construct $M$ origin/corrupted prompt pairs $(p,\tilde{p})$ via a single-word pivot substitution. Right: extract causal attention maps over $L$ layers and $H$ heads, compute $\Delta A=\tilde{A}-A$ , and summarize $\Delta A$ by SVD with TopK $(\sigma)$ (largest $K$ singular values). Concatenating over heads/layers and stacking over pairs yields the fingerprint matrix $F$ . Parentheses denote dimensions (e.g., $N,H,L,M$ ).

	Llama-2- Finance-7B (5M tokens)	Vicuna- 1.5-7B (370M tokens)	WizardMath- 7B (1.8B tokens)	ChineseLLaMA- 2-7B (13B tokens)	CodeLLaMA- 7B (500B tokens)	Llemma- 7B (700B tokens)	Avg
PCS	0.9979	0.9985	0.9965	0.9390	0.5301	0.5052	0.8279
ICS	0.9952	0.9949	0.9985	0.7309	0.5112	0.5104	0.7902
Logits	0.9999	0.9999	0.9999	0.7033	0.7833	0.6367	0.8538
REEF	0.9950	0.9985	0.9979	0.9974	0.9947	0.9962	0.9966
ProFlingo	0.2400	0.5200	0.4200	0.2800	0.2000	0.1400	0.3000
LLMMap	0.8986	0.7294	0.7691	0.8720	0.9555	0.8998	0.8541

Table 1: SFT robustness results (similarity score) on Llama-2-7B-derived suspect models. We annotate each suspect model with its fine-tuning data scale (tokens). Cell shading (per column): =best, =second, =third.

	Weight Merging (Evollm-jp-7b)			Distribution Merging (Fusellm-7b)
	Shisa-gamma-7b-v1	Wizardmath-7b-1.1	Abel-7b-002	Llama-2-7b	Openllama-2-7b	Mpt-7b
PCS	0.9992	0.9990	0.9989	0.9997	0.0194	0.0000
ICS	0.9992	0.9988	0.9988	0.9986	0.2478	0.1014
Logits	0.9933	0.9999	0.9999	0.9999	0.0100	0.0000
REEF	0.9635	0.9526	0.9374	0.9996	0.6713	0.6200
ProFlingo	0.3000	0.4000	0.2800	0.5400	0.1400	0.1600

Table 2: Model merge robustness results (similarity score) on open-source merged suspects. Cell shading (per column): =best, =second, =third.

	Structured Pruning						Unstructured Pruning
	Sheared- llama-1.3b- pruned	Sheared- llama-1.3b	Sheared- llama-1.3b- sharegpt	Sheared- llama-2.7b- pruned	Sheared- llama-2.7b	Sheared- llama-2.7b- sharegpt	Sparse- llama-2-7b	Wanda- llama-2-7b	GBLM- llama-2-7b
PCS	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.9560	0.9620	0.9616
ICS	0.4927	0.3512	0.3510	0.6055	0.4580	0.4548	0.9468	0.9468	0.9478
Logits	0.9967	0.9999	0.9999	0.9967	0.9999	0.9999	0.9999	0.9999	0.9999
REEF	0.9368	0.9676	0.9710	0.9278	0.9701	0.9991	0.9985	0.9986	0.9991
ProFlingo	0.0400	0.0200	0.0800	0.0200	0.1000	0.0800	0.1600	0.1200	0.1800

Table 3: Robustness results (similarity score) on pruned suspect models, comparing structured and unstructured pruning strategies. Cell shading (per column): =best, =second, =third.

Model (vs Llama-2-7B)	CKA(origin)	CKA(diff)	Diff - Origin
Related Models
CodeLlama-7b	0.5134	0.9890	+0.4756
Llama-2-finance-7B	0.9958	0.9989	+0.0031
Sheared-LlaMA-1.3B-Pruned	0.9752	0.9879	+0.0127
Sheared-LlaMA-1.3B	0.9800	0.9938	+0.0138
Sheared-LlaMA-2.7B-Pruned	0.9814	0.9929	+0.0115

Table 4: Ablation of the differential mechanism. “Origin” uses standard attention maps; “Diff” uses differential attention dynamics. Diff restores similarity for heavily adapted models (e.g., Llemma, CodeLlama) while reducing spurious similarity for unrelated architectures.

Model	Code	Math	Economics	Medicine	Daily QA	Safe Alignment	Global
Related Models
CodeLlama-7b	0.9634	0.9912	0.9786	0.9915	0.9849	0.9063	0.9890
Llama-2-finance-7B	0.9994	0.9996	0.9532	0.9994	0.9991	0.9974	0.9989
WizardMath-7B	0.9972	0.9651	0.9862	0.9949	0.9922	0.9998	0.9985
Chinese-Llama-2-7B	0.9865	0.9987	0.9896	0.9971	0.9744	0.9957	0.9963
Llemma-7b	0.9580	0.9537	0.9884	0.9762	0.9952	0.9537	0.9856

Table 5: Domain-wise similarity analysis. Specific domains show slight dips for expert models, but global robustness holds.

Figure 2: Impact of sample size ( $M$ ) on CKA similarity. $M=60$ achieves optimal discrimination.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Transformation Type	Model Example	CKA Score	$1-$ CKA	$\varepsilon$	$2\varepsilon^{2}$ (bound)	Bound Validity
Fine-tuning	Llama-2-7B $\rightarrow$ WizardMath-7B	0.9985	0.0015	0.0777	0.0121	Valid
	Llama-2-7B $\rightarrow$ Llemma-7B	0.9856	0.0144	0.1771	0.0627	Valid
	Llama-2-7B $\rightarrow$ CodeLLaMA-7B	0.9890	0.0110	0.1625	0.0528	Valid
Model Merging	Llama-2-7B $\rightarrow$ Breadcrumbs-Llama-2-7B	0.9992	0.0008	0.0564	0.0064	Valid
	Llama-2-7B $\rightarrow$ Breadcrumbs+Ties-Llama-2-7B	0.9992	0.0008	0.0564	0.0064	Valid
	Llama-2-7B $\rightarrow$ Della-Llama-2-7B	0.9986	0.0014	0.0774	0.0120	Valid

Appendix

Table 6: Representative

\varepsilon

values under different model transformations, where

\varepsilon:=\|\bar{K}-\bar{K}^{\prime}\|_{F}/\|\bar{K}\|_{F}

is computed from centered probe-wise Gram matrices. Related models exhibit small

\varepsilon

while unrelated models yield

\varepsilon\approx 1

, which explains the large margin in CKA and reduces false positives in provenance decisions.

Figure 3: Visualization of single-sample layer–head fingerprint heatmaps ( $L\times H$ ) under fine-tuning, pruning, merging, and distillation.

Symbol	Meaning	Setting
$M$	# probe pairs	60
$L$	# transformer layers	all layers (model-dependent)
$H$	# attention heads per layer	all heads (model-dependent)
$K$	spectral rank (top singular values)	$3$ (Table 10 )
$N,\tilde{N}$	token lengths of $p,\tilde{p}$	variable; controlled but not forced equal
$N^{\ast}$	pooling resolution	$\min(N,\tilde{N})$

Appendix

Table 7: Default hyperparameters for AttnDiff.

Domain	Pivot Pair (Semantic Flip)	Example Prompt Template
Math	never $\leftrightarrow$ always	In Euclidean geometry, two distinct parallel lines {never/always} intersect, no matter how far they are extended.
Math	convergent $\leftrightarrow$ divergent	The infinite series formed by summing 1 over $n^{2}$ is mathematically {convergent/divergent} as $n\to\infty$ .
Code	True $\leftrightarrow$ False	In Python, the boolean expression (len([1,2,3]) > 2) evaluates to {True/False} when executed.
Code	return $\leftrightarrow$ yield	This function uses a {return/yield} statement inside a loop to send back values one at a time.
Economics	increases $\leftrightarrow$ decreases	According to microeconomic theory, when consumer demand rises, market price typically {increases/decreases} .
Economics	supply $\leftrightarrow$ demand	A persistent shortage in the market occurs when {supply/demand} consistently exceeds the other side.

Appendix

Table 8: Representative examples of domain-specific Pivot Rules and constructed prompt templates. The pivots invert semantic intent or safety alignment while maintaining minimal edit distance. Note that this table only displays a subset of the full rule set.

Target Length $\ell$	Sentence Type	Avg. CKA
Target Length $\ell$	Sentence Type	Related	Unrelated
10	Short (Phrase)	0.9587	0.3417
30	Medium (Sentence)	0.9937	0.1647
60	Long (Paragraph)	0.9941	0.1633

Appendix

Table 9: Ablation study on probe length

\ell

(in words). Medium-length probes (

\ell\approx 30

) provide a favorable trade-off between semantic context and attention stability.

Figure 4: Visualization of the 2D adaptive average pooling used to align origin/corrupted attention matrices to a shared resolution $N^{\ast}=\min(N,\tilde{N})$ before computing $\Delta A$ .

Rank $K$	Avg. CKA
Rank $K$	Related	Unrelated
1	0.9928	0.2708
2	0.9941	0.2413
3	0.9937	0.1647
5	0.9854	0.1544
10	0.9711	0.1476

Appendix

Table 10: Ablation study on spectral rank

K

. We select

K=3

as it provides the best trade-off in our setting.

Panel of Figure 5 — Figure 5: Layer-wise CKA similarity matrices for representative fine-tuned, pruned, and unrelated model pairs. Related pairs exhibit strong diagonal structure, while unrelated pairs show uniformly low similarity.

Figure 6: Layer-wise similarity trends for representative related, pruned, and unrelated model pairs under our CKA-based fingerprinting.

Figure 7: ProFlingo workflow. A short trigger prefix is optimized to induce a target response; a suspect model is then evaluated by its target response rate (TRR) over the query set.

Figure 8: Radar comparison of AttnDiff and baseline fingerprinting methods across deployment transformations. Each axis shows the average similarity score of each method on that transformation dimension (averaged over all suspects); larger radii indicate stronger robustness.

Figure 9: Discovery stage: layer-wise column Gini coefficient (mean $\pm$ SD) computed from the differential attention matrix $\Delta A$ under controlled semantic conflicts. We plot the metric against relative layer depth and show that Llama-2-7B derivatives follow highly similar trajectories while Qwen2.5-7B exhibits a consistently lower concentration regime.

Figure 10: Discovery stage: visualization of key differential-attention structure metrics (mean over probe instances) for Qwen2.5-7B and Llama-2-7B derivatives. Llama-2-7B derivatives exhibit highly consistent metric profiles, while Qwen2.5-7B differs in multiple concentration-related dimensions.

Model	$\\|\Delta A\\|_{F}$	Spectral ratio	Effective rank	Col. Gini	Row Gini	$\Delta H$	Locality
CodeLlama-7b-hf	$0.499\pm 0.558$	$0.851\pm 0.145$	$4.681\pm 3.233$	$0.690\pm 0.105$	$0.791\pm 0.154$	$0.003\pm 0.046$	$0.208\pm 0.192$
Llama-2-7B	$0.504\pm 0.531$	$0.834\pm 0.152$	$5.024\pm 3.408$	$0.697\pm 0.107$	$0.781\pm 0.159$	$0.004\pm 0.048$	$0.237\pm 0.197$
WizardMath-7B-V1.0	$0.517\pm 0.520$	$0.832\pm 0.150$	$5.088\pm 3.440$	$0.691\pm 0.105$	$0.780\pm 0.160$	$0.004\pm 0.049$	$0.236\pm 0.196$
llemma_7b	$0.512\pm 0.533$	$0.821\pm 0.162$	$5.054\pm 3.424$	$0.691\pm 0.111$	$0.771\pm 0.167$	$-0.001\pm 0.053$	$0.218\pm 0.201$
Qwen2.5-7B	$0.317\pm 0.313$	$0.764\pm 0.170$	$5.475\pm 3.297$	$0.652\pm 0.102$	$0.742\pm 0.176$	$0.008\pm 0.042$	$0.255\pm 0.241$

Appendix

Table A1: Structural statistics (mean

\pm

SD) of the differential attention matrix

\Delta A

under controlled semantic conflicts. Llama-2-7B derivatives exhibit highly similar statistics, while Qwen2.5-7B consistently differs in concentration-related metrics (e.g., column Gini), supporting the pilot-study motivation for AttnDiff.

Figure 11: Validation stage: PCA visualization of the fingerprint matrix on a broader set of models. Each point corresponds to a probe instance in fingerprint space and colors denote different models (listed in the legend). The resulting structure shows separated regions across diverse models, providing a qualitative validation of the discovery-stage observation.

	PPO		DPO
	llama-2-7b- ppo-v0.1- reward	llama-2-7b- ppo-v0.1- policy	tulu-2-dpo- 7b	llama-2-7b- dpo
PCS	1.0000	0.9935	0.9999	0.9972
ICS	0.9999	0.9874	0.9961	0.9984
Logits	0.9884	0.9891	0.6735	0.7144
REEF	0.6152	0.7458	0.9264	0.9496
ProFlingo	1.0000	1.0000	0.0400	0.2800

Appendix

Table A2: Preference optimization robustness results (similarity score) on Llama-2-7B-derived suspect models (PPO/DPO). Cell shading (per column): =best, =second, =third.

	Breadcrumbs		Breadcrumbs+Ties		Della		Task
	Llama-2-7B	Wizardmath-7b	Llama-2-7B	Wizardmath-7b	Llama-2-7B	Wizardmath-7b	Llama-2-7B	Wizardmath-7b
PCS	0.9990	0.9999	0.9968	0.9999	0.9864	0.9999	0.9921	0.9999
ICS	0.9988	0.9997	0.9988	0.9997	0.9968	0.9982	0.9996	0.9996
Logits	0.8830	0.9575	0.8830	0.9575	0.7841	0.9556	0.9192	0.9484
REEF	0.8131	0.8831	0.8131	0.8831	0.9261	0.9289	0.9442	0.8717
ProFlingo	0.5600	0.5200	0.5600	0.3800	0.4000	0.4400	0.7400	0.6200

Appendix

Table A3: Model merge robustness results (similarity score) under eight merging strategies. Cell shading (per column): =best, =second, =third.

	5%		10%		20%
	Llama-2-7B	Qwen2.5-7B	Llama-2-7B	Qwen2.5-7B	Llama-2-7B	Qwen2.5-7B
Random pruning
PCS	0.9878	0.9869	0.9750	0.9738	0.9497	0.9468
ICS	0.9208	0.9502	0.8415	0.8954	0.6878	0.7715
Logits	0.9923	0.9999	0.9875	0.9928	0.9839	0.9832
REEF	0.6368	0.2417	0.3689	0.2259	0.2629	0.2021

Appendix

Table A4: Pruning robustness results (similarity score) under Random,

L_{1}

, and Taylor pruning criteria at different sparsity levels. Cell shading (per column): =best, =second, =third.

	Qwen2.5						gemma-2-2b			Mistral-7B-v0.3
	Qwen2.5- Coder-1.5B	Qwen2.5- Math-1.5B	Qwen2.5- 1.5B-Instruct	Qwen2.5- 14B-Instruct	oxy-1- small	Qwen2.5-14B- Gutenberg-Instruct- Slerpeno	gemma-2-2b- neogenes-ita	gemma-2- baku-2b	gemma-2-2b- merged	KurmaAI/ AQUA-7B	openfoodfacts/ spellcheck- mistral-7b	grimjim/ Mistral-7B-Instruct- demi-merge- v0.3-7B
PCS	0.8385	0.6681	0.9999	0.9999	0.9961	0.9999	0.9996	0.9971	0.9996	0.9999	0.9998	0.9999
ICS	0.2661	0.4486	0.9954	0.9997	0.9925	0.9997	0.9954	0.9880	0.9957	0.9820	0.9996	0.9997
Logits	0.5275	0.2728	0.8832	0.6228	0.5571	0.6847	0.7615	0.8036	0.7355	0.6859	0.9076	0.9225
ProFlingo	0.8400	0.8000	0.7400	0.7800	0.6600	0.5400	0.6200	0.6800	0.5400	0.5800	0.6600	0.7600
LLMmap	0.8231	0.8593	0.6929	0.6737	0.7505	0.6221	0.7429	0.9655	0.6748	0.7658	0.8960	0.7940

Appendix

Table A5: Cross-scale and cross-family robustness results (similarity score) across Qwen2.5-derived suspects (1.5B/14B), gemma-2-derived suspects (2B), and Mistral-derived suspects (7B). Cell shading (per column): =best, =second, =third.

	Qwen2.5-7B	Qwen2.5-14B
Deployable distilled model	Qwen2.5-7B-Open-R1-Distill	DeepSeek-R1-Distill-Qwen-14B
PCS	0.9998	0.9867
ICS	0.9992	0.9738
Logits	0.6990	0.5056
REEF	0.9926	0.7512
ProFlingo	0.5600	0.4400

Appendix

Table A6: Distillation robustness results (similarity score) across teacher–student pairs spanning Qwen2.5 and Llama families. Each block reports similarity between a base model and its distilled variant. Cell shading (per column): =best, =second, =third.

	Mixtral-8x7B
	Instruct_Mixtral -8x7B-v0.1_Dolly15K	Nous-Hermes-2 -Mixtral-8x7B-DPO	openbuddy-mixtral -8x7b-v15.4
AttnDiff	0.9925	0.9814	0.9905

Appendix

Table A7: Similarity score between Mixtral-8x7B and its instruction-tuned MoE derivatives, illustrating the applicability of AttnDiff to MoE architectures.

	Qwen2.5-7B		Llama-2-7B
	Instruct-GPTQ-Int4	Instruct-GPTQ-Int8	Chat-GPTQ
Ours	0.9890	0.9913	0.9933

Appendix

Table A8: Quantization robustness results (AttnDiff similarity score) between each full-precision base model and its GPTQ-quantized derivatives.

Model	Clean	Paraphrase	Rewrite	Style-transfer
CodeLlama-7b-hf	0.9535	0.9484	0.9484	0.9500
llemma_7b	0.8886	0.7174	0.7152	0.7433
llama-2-7b-logit-watermark-distill- kgw-k1-gamma0.25-delta2	0.9233	0.8934	0.8904	0.8851
WizardMath-7B-V1.0	0.6998	0.6147	0.6210	0.6330

Appendix

Table A9: Cosine similarity of LLMMap fingerprints before and after meaning-preserving paraphrasing, rewriting, and style-transfer attacks.

Category	Type	Base (Victim)	Suspect(s) / Derivative(s)
Fine-tuning	Instruction	Llama-2-7B	Llama-2-finance-7b, Vicuna-1.5-7b, WizardMath-7b, Chinese-LLaMA-2-7b, CodeLLaMA-7b, Llemma-7b
Merge	Weight	Shisa-gamma-7b-v1	Evollm-jp-7b
		WizardMath-7b-1.1
		Abel-7b-002
	Dist./Behav.	Llama-2-7B	Fusellm-7b
		OpenLLaMA-2-7b

Appendix

Table A10: Models used in the main paper and appendix analyses.

ID	Repository URL
1	https://huggingface.co/renyiyu/llama-2-7b-ppo-lora-v0.1
2	https://huggingface.co/allenai/tulu-2-dpo-7b
3	https://huggingface.co/mncai/llama2-7b-dpo-v1
4	https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B
5	https://huggingface.co/Qwen/Qwen2.5-Math-1.5B
6	https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct

Appendix

Table A11: Repository URLs corresponding to the superscript IDs in Table A10 .

Language Model Fingerprinting

Language model fingerprinting identifies and verifies models or their lineage from distinctive outputs, internal representations, or embedded signals.

See all 77 papers

Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
Jul 15, 2024Mark Russinovich, Yanan Cai, Ahmed SalemLanguage Model Fingerprinting
Microsoft
Targeted Counterfactual Fingerprinting for Black-Box LLM Ownership Verification
Aug 8, 2026Yutong Wu, Xiaofan Bai, Shixin Li +10Language Model Fingerprinting
Huazhong University of Science and Technology · Alibaba Group · Microsoft Corporation
Fingerprinting Multimodal Large Language Models
Sep 17, 2026Chao Huang, Meng Tong, Kejiang ChenLLM Auditing Language Model Fingerprinting
University of Science and Technology of China Anhui Province Key Laboratory of Digital Security Hefei, Anhui, China

Jul 15, 2024J/K move · Enter open · S save

Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique

Mark Russinovich, Yanan Cai, Ahmed Salem

Microsoft

Language Model Fingerprinting

Open paper AskPDF arXiv

Growing concerns over the theft and misuse of Large Language Models (LLMs) underscore the need for effective fingerprinting to link a model to its original version and detect misuse. We define five essential properties for a successful fingerprint: Transparency, Efficiency, Persistence, Robustness, and Unforgeability. We present a novel fingerprinting framework that provides verifiable proof of ownership while preserving fingerprint integrity. Our approach makes two main contributions. First, a chain and hash technique that cryptographically binds fingerprint prompts to their responses, preventing collisions and enabling irrefutable ownership claims. Second, we address a realistic threat model in which instruction-tuned models' output distribution can be significantly altered through meta-prompts. By incorporating random padding and varied meta-prompt configurations during training, our method maintains robustness even under significant output style changes. Experiments show that our framework securely proves ownership, resists both benign transformations (e.g., fine-tuning) and adversarial fingerprint removal, and extends to fingerprinting LoRA adapters\footnote{We release our code at: https://github.com/microsoft/Chain-Hash.

Abstract

Language Model Fingerprinting

Explore similar work

Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique

Targeted Counterfactual Fingerprinting for Black-Box LLM Ownership Verification

Fingerprinting Multimodal Large Language Models