Auditor-Assisted Summary-Channel Verification for Hosted LLM Identity Substitution

Ziyi Zhang; Ziyao Wang; Guoheng Sun; Ang Li; Jian Li

cs.CROct 19, 2026

Auditor-Assisted Summary-Channel Verification for Hosted LLM Identity Substitution

Authors: Ziyi Zhang, Ziyao Wang, Guoheng Sun, Ang Li, Jian Li

Organizations: Stony Brook University · University of Maryland, College Park

Abstract

Hosted LLM services increasingly expose model identity as a product claim, but external users and auditors often cannot inspect which model configuration served a request. We study an auditor-assisted regime in which a trusted auditor enrolls claimed identities and prepares verification material before deployment, while audit-time verification uses only the claimed identity and released output, without runtime access to model internals, inference logs, routing decisions, or provider metadata. We propose SUMMARK, a summary-channel watermarking framework that uses user-visible reasoning summaries as a keyed audit surface. During setup, SUMMARK constructs identity-specific vocabulary carriers, trains lightweight summary adapters, and calibrates per-identity thresholds. At deployment time, a keyed statistical test assesses whether the released summary is consistent with the claimed identity and, when needed, attributes the most likely enrolled identity. Across model families, scales, and task domains, SUMMARK reliably detects cross-family and cross-scale substitutions at low false-positive rates while preserving task-facing answer quality. We further study FP16->INT8/INT4 precision-consistency diagnostics and robustness under bounded post-processing, rewriting stress tests, and text-only spoofing, revealing both strong performance and clear failure boundaries. Our code is publicly available at https://github.com/trinitr0toluene/Summark.

Figures & tables

Figure 1: Auditor-assisted summary-channel verification. The user requests a claimed model identity $m$ and receives a task-facing answer with a user-visible reasoning summary. Given the claim and released summary, a trusted auditor tests whether the summary is statistically consistent with $m$ using secret verification material prepared offline. When answer and summary generation use separate pipelines, the guarantee is summary-channel consistency. End-to-end answer provenance requires additional coupling or deployment-side attestation.

Figure 2: Workflow of SumMark . Offline, the trusted auditor uses a secret key to construct identity-specific vocabulary carriers, train summary watermark adapters, and calibrate thresholds. Online, the provider returns a task-facing answer and reasoning summary under a claimed identity. The auditor uses only the claim, released summary, and secret key to test summary-channel consistency and, when needed, attribute the most likely enrolled identity.

	Math			General			Code
Method	AvgT	MinT	AUC	AvgT	MinT	AUC	AvgT	MinT	AUC
BERT	0.7683	0.1232	0.9840	0.5439	0.0877	0.9401	0.6567	0.0000	0.9916
GPT-2	0.3333	0.0000	0.7969	0.2506	0.0000	0.8101	0.3886	0.0000	0.8174
T5	0.8676	0.6359	0.9787	0.5213	0.0000	0.9703	0.7494	0.1053	0.9975
SumMark	0.9994	0.9960	0.9996	0.9750	0.8740	0.9990	1.0000	1.0000	1.0000

Table 1: Summary of substitution detection across seven substitution pairs per dataset. We report average (AvgT) TPR@FPR=1%, minimum (MinT) TPR@FPR=1%, and average AUC. Full per-pair results are in Appendix E.1 .

Table 4

	Unigram			Gumbel Max			SynthID			SumMark
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
Qwen-7B $\rightarrow$ Qwen-3B	0.4730	0.7315	0.9523	0.0050	0.4975	0.4828	0.0050	0.4975	0.5293	1.0000	0.9950	1.0000
Qwen-7B $\rightarrow$ Qwen-1.5B	0.0690	0.5295	0.7288	0.0300	0.5100	0.4968	0.0200	0.5050	0.5506	0.9960	0.9930	0.9974
Qwen-3B $\rightarrow$ Qwen-1.5B	0.0000	0.4950	0.2753	0.0950	0.5425	0.6150	0.0950	0.5425	0.6174	1.0000	0.9950	0.9998
Qwen-3B $\rightarrow$ LLaMA-3B	0.0010	0.4955	0.6302	0.0400	0.5150	0.4865	0.0800	0.5350	0.4961	1.0000	0.9950	0.9997
LLaMA-3B $\rightarrow$ Qwen-3B	0.1650	0.5775	0.7466	0.0000	0.4950	0.5585	0.0000	0.4950	0.5240	1.0000	0.9950	1.0000

Table 4: Substitution detection on Math with watermarking baselines. We report TPR@FPR=1%, ACC@FPR=1%, and AUC. Setting A

\rightarrow

B denotes serving B while claiming A.

		Math			Code			General
Model	Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
LLaMA-3B	FP16 $\rightarrow$ INT8	0.9990	0.9960	0.9990	0.9160	0.9530	0.9694	0.9580	0.9740	0.9971
LLaMA-3B	FP16 $\rightarrow$ INT4	0.9990	0.9995	0.9990	0.9680	0.9790	0.9791	0.9930	0.9920	0.9998
Qwen-1.5B	FP16 $\rightarrow$ INT8	0.1132	0.5521	0.7575	0.0360	0.5130	0.7010	0.4454	0.7182	0.8534
Qwen-1.5B	FP16 $\rightarrow$ INT4	0.3417	0.6663	0.9209	0.4120	0.7010	0.9497	0.7858	0.8884	0.9454
Qwen-3B	FP16 $\rightarrow$ INT8	0.9860	0.9890	0.9973	0.9460	0.9680	0.9952	0.9910	0.9910	0.9964

Table 5: Within-identity precision consistency across Math, Code, and General. FP16

\rightarrow

INT8 denotes serving a quantized variant while claiming the FP16 configuration of the same identity. Precision variants share the same carrier set and adapter; the verifier is calibrated on FP16 summaries.

Stress test	Avg.	Worst
Original TPR@1%	0.9994	0.9960
Bounded edits TPR@1%	0.9992	0.9950
Bilingual rewrite TPR@1%	0.7900	0.5740
Distilled spoofing ASR@1%	0.1651	0.6290
Watermark-stealing ASR@1%	0.0417	0.2160

Table 6: Robustness summary on Math across seven substitution pairs. For post-processing, we report average/min TPR@FPR=1%; for spoofing, average/max ASR@FPR=1%. Full results are in Appendix E.2 .

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

	BERT			GPT-2			T5			SumMark
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
Qwen-7B $\to$ Qwen-3B	0.1232	0.5602	0.9221	0.0000	0.5000	0.4458	0.6359	0.8053	0.9348	1.0000	0.9950	1.0000
Qwen-7B $\to$ Qwen-1.5B	0.9412	0.9706	0.9955	0.2105	0.5965	0.6037	0.9804	0.9902	0.9939	0.9960	0.9930	0.9974
Qwen-3B $\to$ Qwen-1.5B	0.9720	0.9790	0.9959	0.2807	0.6316	0.8339	0.9804	0.9748	0.9976	1.0000	0.9950	0.9998
Qwen-3B $\to$ LLaMA-3B	0.9076	0.9468	0.9939	0.6842	0.8333	0.9120	0.7003	0.8235	0.9605	1.0000	0.9950	0.9997
LLaMA-3B $\to$ Qwen-3B	0.8431	0.9202	0.9939	0.3684	0.6754	0.9477	0.8627	0.9300	0.9745	1.0000	0.9950	1.0000

Appendix

Table 7: Substitution detection on Math (TPR@FPR=1%, ACC@FPR=1%, AUC). Setting

A\!\to\!B

: serve

B

, claim

A

. Bold is best.

	BERT			GPT-2			T5			SumMark
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
Qwen-7B $\to$ Qwen-3B	0.7895	0.8772	0.9621	0.0000	0.5000	0.6331	0.7193	0.8596	0.9898	0.9990	0.9945	0.9999
Qwen-7B $\to$ Qwen-1.5B	0.1228	0.5439	0.8689	0.0526	0.5263	0.5389	0.0000	0.5000	0.9514	0.8740	0.9320	0.9954
Qwen-3B $\to$ Qwen-1.5B	0.0877	0.5439	0.9267	0.1053	0.5526	0.7014	0.0000	0.5000	0.9338	1.0000	0.9950	1.0000
Qwen-3B $\to$ LLaMA-3B	0.5439	0.7719	0.9298	0.3509	0.6754	0.9464	0.5088	0.7544	1.0000	1.0000	0.9950	0.9999
LLaMA-3B $\to$ Qwen-3B	0.8596	0.9211	0.9917	0.3684	0.6842	0.9372	0.9474	0.9561	0.9800	1.0000	0.9950	1.0000

Appendix

Table 8: Substitution detection on General (TPR@FPR=1%, ACC@FPR=1%, AUC). Setting

A\!\to\!B

: serve

B

, claim

A

. Q7/Q3/Q1.5 = Qwen-7B/3B/1.5B, L3 = LLaMA-3B. Bold is best.

	BERT			GPT-2			T5			SumMark
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
Qwen-7B $\to$ Qwen-3B	0.6491	0.8246	0.9825	0.0000	0.5000	0.5720	0.8947	0.9474	0.9963	1.0000	0.9950	1.0000
Qwen-7B $\to$ Qwen-1.5B	0.8246	0.9123	0.9834	0.1400	0.5700	0.6690	1.0000	1.0000	1.0000	1.0000	0.9950	1.0000
Qwen-3B $\to$ Qwen-1.5B	0.0000	0.5000	0.9769	0.5090	0.7110	0.7960	0.1053	0.5526	0.9982	1.0000	0.9950	1.0000
Qwen-3B $\to$ LLaMA-3B	0.2281	0.6140	0.9991	0.7370	0.8250	0.9330	0.3684	0.6842	0.9942	1.0000	0.9950	1.0000
LLaMA-3B $\to$ Qwen-3B	0.9474	0.9737	0.9991	0.4040	0.6750	0.9160	0.8947	0.9474	0.9935	1.0000	0.9950	1.0000

Appendix

Table 9: Substitution detection on Code (TPR@FPR=1%, ACC@FPR=1%, AUC). Setting

A\!\to\!B

: serve

B

, claim

A

. Bold is best.

	BERT			GPT-2			T5			IDP			SumMark
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
$Q7\to Q3$	0.1232	0.5602	0.9221	0.0000	0.5000	0.4458	0.6359	0.8053	0.9348	0.0000	0.5000	0.5000	1.0000	0.9950	1.0000
$Q7\to Q1.5$	0.9412	0.9706	0.9955	0.2105	0.5965	0.6037	0.9804	0.9902	0.9939	0.0000	0.5000	0.5000	0.9960	0.9930	0.9974
$Q3\to Q1.5$	0.9720	0.9790	0.9959	0.2807	0.6316	0.8339	0.9804	0.9748	0.9976	1.0000	1.0000	1.0000	1.0000	0.9950	0.9998
$Q3\to L3$	0.9076	0.9468	0.9939	0.6842	0.8333	0.9120	0.7003	0.8235	0.9605	0.0000	0.5000	0.5000	1.0000	0.9950	0.9997
$L3\to Q3$	0.8431	0.9202	0.9939	0.3684	0.6754	0.9477	0.8627	0.9300	0.9745	0.0000	0.5000	0.5000	1.0000	0.9950	1.0000

Appendix

Table 10: Substitution detection on Math (TPR@FPR=1%, ACC@FPR=1%, AUC). Setting

A\!\to\!B

: serve

B

, claim

A

. Q7/Q3/Q1.5 = Qwen-7B/3B/1.5B, L3 = LLaMA-3B. Bold is best.

Method	ACC	Macro-F1
BERT	0.9386	0.9383
GPT-2	0.7121	0.6981
T5	0.9128	0.9122
IDP	0.7500	0.6670
SumMark	0.9958	0.9957

Appendix

Table 11: Attribution performance on Math . Higher is better.

	SynthID			Gumbel Max			SumMark
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
Q7 $\rightarrow$ Q3	0.0000	0.4950	0.5350	0.0500	0.5200	0.6041	1.0000	0.9950	1.0000
Q7 $\rightarrow$ Q1.5	0.0350	0.5125	0.5623	0.0300	0.5100	0.5463	0.9960	0.9930	0.9974
Q3 $\rightarrow$ Q1.5	0.0700	0.5300	0.5235	0.0200	0.5050	0.4112	1.0000	0.9950	0.9998
Q3 $\rightarrow$ L3	0.0250	0.5075	0.5216	0.0450	0.5175	0.4143	1.0000	0.9950	0.9997
L3 $\rightarrow$ Q3	0.0000	0.4950	0.4727	0.0000	0.4950	0.4770	1.0000	0.9950	1.0000

Appendix

Table 12: Identity-Substitution detection on Math. SynthID and Gumbel Max are applied directly to the task-facing answer, while SumMark applies the watermark only to the summary channel. We report TPR@FPR=1%, ACC@FPR=1%, and AUC.

	BERT		GPT-2		T5		SumMark
Setting	TPR@1%	TPR@5%	TPR@1%	TPR@5%	TPR@1%	TPR@5%	TPR@1%	TPR@5%
Qwen-7B $\to$ Qwen-3B	0.1232	0.5322	0.0000	0.0351	0.6359	0.8095	1.0000	1.0000
Qwen-7B $\to$ Qwen-1.5B	0.9412	0.9720	0.2105	0.2807	0.9804	0.9804	0.9960	0.9970
Qwen-3B $\to$ Qwen-1.5B	0.9720	0.9720	0.2807	0.2807	0.9804	0.9804	1.0000	1.0000
Qwen-3B $\to$ LLaMA-3B	0.9076	0.9692	0.6842	0.8070	0.7003	0.7199	1.0000	1.0000
LLaMA-3B $\to$ Qwen-3B	0.8431	0.9160	0.3684	0.7368	0.8627	0.8627	1.0000	1.0000

Appendix

Table 13: Identity-Substitution detection performance on Math dataset. We report TPR@FPR=1% and TPR@FPR=5%. Best is bold.

	GPT-2		T5		SumMark
#Identities	TPR@1%	AUC	TPR@1%	AUC	TPR@1%	AUC
4	0.6250	0.9735	0.8279	0.9774	0.9986	0.9987
6	0.6213	0.9807	0.8089	0.9819	0.7811	0.9397
8	0.5900	0.9792	0.7360	0.9766	0.7462	0.9105
10	0.7015	0.9826	0.7340	0.9765	0.7919	0.9339
12	0.7188	0.9833	0.7518	0.9776	0.8187	0.9443

Appendix

Table 14: Identity-count scaling on Math (nested identity sets, 1,000 training examples per identity). Best is bold.

	GPT-2		T5		SumMark
#Families	TPR@1%	AUC	TPR@1%	AUC	TPR@1%	AUC
1	0.5788	0.9646	0.5825	0.9382	0.7850	0.9527
2	0.6344	0.9769	0.6702	0.9626	0.8251	0.9603
3	0.6501	0.9795	0.6487	0.9670	0.8121	0.9596
4	0.6977	0.9836	0.7061	0.9720	0.7935	0.9543

Appendix

Table 15: Family-count scaling on Math . Each family contributes two identities; results average over all family subsets, so the 4-family point uses 8 identities. Best is bold.

Within family
		GPT-2		T5		SumMark
Setting	Red.	TPR@1%	AUC	TPR@1%	AUC	TPR@1%	AUC
Qwen-32B $\rightarrow$ Qwen-0.5B	64.0 $\times$	0.7800	0.9866	0.8850	0.9924	1.0000	0.9999
Qwen-32B $\rightarrow$ Qwen-1.5B	21.3 $\times$	0.8250	0.9891	0.9700	0.9986	1.0000	1.0000
Qwen-32B $\rightarrow$ Qwen-3B	10.7 $\times$	0.8600	0.9912	0.9350	0.9966	0.9900	0.9997
Qwen-14B $\rightarrow$ Qwen-0.5B	28.0 $\times$	0.3150	0.9374	0.3450	0.8949	0.6900	0.9278

Appendix

Table 16: Identity-Substitution detection with large parameter gaps on Math (1,000 training examples per identity). Red. is the claimed-to-served parameter ratio; macro averages are over the six pairs in each panel. Best is bold.

		SumMark
Held-out axis	Setting	TPR@1%	AUC
Family	Qwen-3B $\rightarrow$ LLaMA-3B	0.9990	0.9991
	Qwen-7B $\rightarrow$ LLaMA-3B	0.9990	1.0000
	Aggregate	0.9990	0.9995
Scale	Qwen-3B $\rightarrow$ Qwen-1.5B	1.0000	0.9986
	Qwen-7B $\rightarrow$ Qwen-1.5B	0.9920	0.9984

Appendix

Table 17: Claim rejection on Math for substitutes held out of the

L_{\mathrm{sep}}

negative-identity pool. The task is claim rejection, not open-set attribution.

	FP16			INT8			INT4
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
Q7 $\rightarrow$ Q3	1.0000	0.9950	1.0000	1.0000	0.9950	0.9985	1.0000	0.9950	0.9995
Q7 $\rightarrow$ Q1.5	0.9960	0.9930	0.9974	1.0000	0.9950	0.9995	0.9920	0.9910	0.9943
Q3 $\rightarrow$ Q1.5	1.0000	0.9950	0.9998	1.0000	0.9950	0.9985	0.9970	0.9935	0.9945
Q3 $\rightarrow$ L3	1.0000	0.9950	0.9997	1.0000	0.9950	0.9995	0.9970	0.9935	0.9960
L3 $\rightarrow$ Q3	1.0000	0.9950	1.0000	1.0000	0.9950	0.9995	0.9960	0.9930	0.9983

Appendix

Table 18: Substitution detection results on Math under different serving precisions. For each substitution pair, the provider serves the substituted model at FP16, INT8, or INT4 precision while claiming the target identity.

Table 20

	Original			Model Finetuning			Watermark Removal
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
Q7 $\rightarrow$ Q3	1.0000	0.9950	1.0000	0.9630	0.9765	0.9950	1.0000	0.9950	0.9995
Q7 $\rightarrow$ Q1.5	1.0000	0.9950	1.0000	0.9770	0.9835	0.9978	0.9970	0.9935	0.9973
Q3 $\rightarrow$ Q1.5	1.0000	0.9950	1.0000	0.9780	0.9840	0.9975	1.0000	0.9950	0.9994
Q3 $\rightarrow$ L3	1.0000	0.9950	1.0000	0.4480	0.7190	0.9553	1.0000	0.9950	0.9994
L3 $\rightarrow$ Q3	1.0000	0.9950	1.0000	0.7160	0.8530	0.9832	1.0000	0.9950	0.9990

Appendix

Table 21: Robustness on Code under downstream fine-tuning and lexical watermark removal. For fine-tuning, we compose a task-specific summarization LoRA with the watermark adapter at inference. For watermark removal, we apply a copy-paste attack that replaces contiguous spans in the summary.

	Distilled Adapter Spoofing			Watermark Stealing Spoofing
Setting	TPR@1%	ASR@1%	AUC	TPR@1%	ASR@1%	AUC
Q7 $\rightarrow$ Q3	0.9980	0.0020	0.9999	0.7840	0.2160	0.9930
Q7 $\rightarrow$ Q1.5	0.9830	0.0170	0.9939	0.9240	0.0760	0.9960
Q3 $\rightarrow$ Q1.5	0.9960	0.0040	0.9983	1.0000	0.0000	1.0000
Q3 $\rightarrow$ L3	0.9570	0.0430	0.9938	1.0000	0.0000	1.0000
L3 $\rightarrow$ Q3	0.3710	0.6290	0.9474	1.0000	0.0000	1.0000

Appendix

Table 22: Robustness under spoofing attacks on Math. We evaluate two text-only spoofing strategies: distilled adapter spoofing and watermark-stealing spoofing. For both attacks, lower ASR@1% and higher TPR@1%/AUC indicate stronger audit robustness.

	128 tokens			256 tokens			384 tokens			512 tokens
Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
Q7 $\rightarrow$ Q3	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Q7 $\rightarrow$ Q1.5	0.9250	0.9625	0.9944	1.0000	1.0000	1.0000	0.9750	0.9875	0.9969	0.9750	0.9875	0.9994
Q3 $\rightarrow$ Q1.5	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	0.7750	0.8875	0.9944	0.9750	0.9875	0.9981
Q3 $\rightarrow$ L3	0.7000	0.8500	0.9825	0.7000	0.8500	0.9788	0.6750	0.8375	0.9856	0.6500	0.8250	0.9756
L3 $\rightarrow$ Q3	0.9750	0.9875	0.9938	0.9750	0.9875	0.9925	1.0000	1.0000	1.0000	0.9750	0.9875	0.9994

Appendix

Table 23: Length sensitivity of substitution detection on Code. We truncate each released summary to the first 128, 256, 384, or 512 tokens, and report TPR@FPR=1%, ACC@FPR=1%, and AUC. Setting A

\rightarrow

B denotes serving B while claiming A.

	Original		Sentence Shuffle		Random Deletion		Truncate Words		Bilingual Rewrite
Setting	TPR@1%	AUC	TPR@1%	AUC	TPR@1%	AUC	TPR@1%	AUC	TPR@1%	AUC
Q7 $\rightarrow$ Q3	1.0000	1.0000	1.0000	0.9995	1.0000	0.9995	1.0000	0.9995	0.9760	0.9963
Q7 $\rightarrow$ Q1.5	0.9960	0.9974	0.9960	0.9969	0.9950	0.9969	0.9950	0.9981	0.9010	0.9931
Q3 $\rightarrow$ Q1.5	1.0000	0.9998	1.0000	0.9994	1.0000	0.9993	1.0000	0.9994	0.8220	0.9840
Q3 $\rightarrow$ L3	1.0000	0.9997	1.0000	0.9993	1.0000	0.9992	1.0000	0.9994	0.6570	0.9652
L3 $\rightarrow$ Q3	1.0000	1.0000	1.0000	0.9990	1.0000	0.9990	1.0000	0.9990	0.6390	0.9646

Appendix

Table 24: Robustness of watermark detection under summary post-processing and rewriting on Math. We report three bounded surface transformations (sentence shuffle, random deletion, and word truncation), and one stronger bilingual rewriting stress test.

	SumMark
Train $\rightarrow$ test	TPR	Observed FPR	Balanced ACC	AUC
Math $\rightarrow$ Math	0.9994	0.0100	0.9947	0.9996
Math $\rightarrow$ Code	0.9993	0.0061	0.9966	0.9991
Math $\rightarrow$ General	0.9989	0.0123	0.9933	0.9990

Appendix

Table 25: Cross-domain transfer from Math to Code and General . Adapters and detection thresholds are trained and calibrated on Math and kept fixed; observed FPR is measured on target-domain data. Metrics are macro-averaged over seven substitution pairs.

	SumMark
Poison rate	TPR@1%	ACC@1%	AUC
0%	1.0000	0.9950	1.0000
5%	0.9900	0.9925	0.9998
15%	0.7950	0.8925	0.9761
25%	0.6900	0.8400	0.9602
50%	0.0000	0.4950	0.8767

Appendix

Table 26: Carrier-label poisoning on Qwen-7B

\rightarrow

Qwen-3B ( Math , 1,000 training examples per rate). Prompts and teacher summaries are unchanged; only the

L_{\mathrm{wm}}

and

L_{\mathrm{sep}}

carrier targets are corrupted. At 100%, labels are fully swapped and the score ordering is inverted.

	TPR@1%		AUC
Method	Mean	Min	Mean	Min
T5 + SFT adapter	0.8937	0.6550	0.9899	0.9619
Keyed Unigram adapter	0.3328	0.0020	0.7890	0.1340
Keyed Greenlist adapter	0.0163	0.0000	0.5319	0.4823
SumMark	0.9980	0.9920	0.9981	0.9970

Appendix

Table 27: Adapter-matched baselines on Math (1,000 training examples per identity). Mean and minimum are taken over seven substitution pairs. T5 + SFT adapter uses unwatermarked summaries; keyed Unigram/Greenlist adapters use a separate key and adapter per identity. Best is bold.

Figure 3: Visualizing identity footprints: tokens in

G_{\text{Qwen-3B}}

are highlighted. The Qwen-3B adapter yields a denser, more uniform green-token pattern than no-watermark, while a substituted identity (Qwen-7B) does not reproduce this footprint when colored by

G_{\text{Qwen-3B}}

Model	Setting	w/o $L_{\mathrm{qs}}$			w/ $L_{\mathrm{qs}}$
Model	Setting	TPR@1%	ACC@1%	AUC	TPR@1%	ACC@1%	AUC
LLaMA-3B	FP16 $\rightarrow$ INT8	0.0150	0.5040	0.5034	0.9990	0.9960	0.9990
LLaMA-3B	FP16 $\rightarrow$ INT4	0.0000	0.5000	0.4256	0.9990	0.9995	0.9990
Qwen-1.5B	FP16 $\rightarrow$ INT8	0.0090	0.5000	0.4952	0.1132	0.5521	0.7575
Qwen-1.5B	FP16 $\rightarrow$ INT4	0.0181	0.5050	0.4875	0.3417	0.6663	0.9209
Qwen-3B	FP16 $\rightarrow$ INT8	0.0020	0.5000	0.5298	0.9860	0.9890	0.9973

Appendix

Table 28: Effect of the quantization-separation loss

L_{\mathrm{qs}}

on within-identity precision-consistency detection. We compare watermark adapters trained with and without

L_{\mathrm{qs}}

when the claimed configuration is FP16 and the served configuration is INT8 or INT4.

	5%			10%			15%			25%			50%			65%
Setting	TPR	ACC	AUC	TPR	ACC	AUC	TPR	ACC	AUC	TPR	ACC	AUC	TPR	ACC	AUC	TPR	ACC	AUC
Q7 $\to$ Q3	1.0000	0.9910	0.9995	1.0000	0.9910	0.9994	1.0000	0.9920	0.9994	1.0000	0.9930	0.9993	1.0000	0.9950	0.9990	1.0000	0.9970	0.9986
Q7 $\to$ Q1.5	1.0000	0.9910	0.9995	1.0000	0.9910	0.9994	0.9933	0.9910	0.9931	0.9920	0.9910	0.9955	0.9940	0.9920	0.9953	0.9969	0.9950	0.9957
Q3 $\to$ Q1.5	1.0000	0.9910	0.9994	1.0000	0.9910	0.9989	1.0000	0.9920	0.9990	1.0000	0.9930	0.9985	1.0000	0.9950	0.9989	1.0000	0.9970	0.9985
Q3 $\to$ L3	1.0000	0.9910	0.9994	1.0000	0.9910	0.9992	1.0000	0.9920	0.9993	1.0000	0.9930	0.9992	1.0000	0.9950	0.9988	1.0000	0.9970	0.9940
L3 $\to$ Q3	1.0000	0.9910	0.9990	1.0000	0.9910	0.9989	1.0000	0.9920	0.9988	1.0000	0.9930	0.9987	1.0000	0.9950	0.9990	1.0000	0.9970	0.9986

Appendix

Table 29: Sensitivity to Substitution Ratio on Math . We report TPR@FPR=1%, ACC@FPR=1%, and AUC. Q7/Q3/Q1.5 = Qwen-7B/3B/1.5B, L3 = LLaMA-3B. Best is bold.

	5%			10%			15%			25%			50%			65%
Setting	TPR	ACC	AUC	TPR	ACC	AUC	TPR	ACC	AUC	TPR	ACC	AUC	TPR	ACC	AUC	TPR	ACC	AUC
Q7 $\to$ Q3	1.0000	0.9910	0.9999	1.0000	0.9910	0.9999	1.0000	0.9920	0.9999	1.0000	0.9930	0.9999	1.0000	0.9950	1.0000	1.0000	0.9970	0.9999
Q7 $\to$ Q1.5	0.8200	0.9820	0.9958	0.8400	0.9750	0.9931	0.8800	0.9740	0.9951	0.9840	0.9890	0.9965	0.9860	0.9880	0.9971	0.9877	0.9890	0.9969
Q3 $\to$ Q1.5	1.0000	0.9910	1.0000	1.0000	0.9910	1.0000	1.0000	0.9920	1.0000	1.0000	0.9930	1.0000	0.9980	0.9940	1.0000	0.9985	0.9960	1.0000
Q3 $\to$ L3	1.0000	0.9910	1.0000	1.0000	0.9910	0.9999	1.0000	0.9920	0.9999	1.0000	0.9930	0.9999	1.0000	0.9950	0.9999	1.0000	0.9970	0.9999
L3 $\to$ Q3	1.0000	0.9910	1.0000	1.0000	0.9910	1.0000	1.0000	0.9920	1.0000	1.0000	0.9930	1.0000	1.0000	0.9950	1.0000	1.0000	0.9970	1.0000

Appendix

Table 30: Sensitivity to Substitution Ratio on General . We report TPR@FPR=1%, ACC@FPR=1%, and AUC. Q7/Q3/Q1.5 = Qwen-7B/3B/1.5B, L3 = LLaMA-3B. Best is bold.

Figure 4: Effect of training objective components

\lambda_{\mathrm{sep}}

and

\lambda_{\mathrm{wm}}

on substitution performance on Code dataset. Ratio is defined as

\lambda_{\mathrm{sep}}/\lambda_{\mathrm{wm}}

. We report TPR@FPR = 1% as the score.

Figure 5: Effect of green ratio

\rho

and overlap

\tau

on substitution performance on Math dataset.

Figure 6: Effect of training objective components

\lambda_{\mathrm{sep}}

and

\lambda_{\mathrm{wm}}

on substitution performance on Code dataset. Ratio is defined as

\lambda_{\mathrm{sep}}/\lambda_{\mathrm{wm}}

. We report AUC as the score.

Figure 7: Effect of training objective components

\lambda_{\mathrm{sep}}

and

\lambda_{\mathrm{wm}}

on substitution performance on Code dataset. Ratio is defined as

\lambda_{\mathrm{sep}}/\lambda_{\mathrm{wm}}

. We report TPR@FPR = 1% as the score.

Figure 8: Effect of training objective components

\lambda_{\mathrm{sep}}

and

\lambda_{\mathrm{wm}}

on substitution performance on General dataset. Ratio is defined as

\lambda_{\mathrm{sep}}/\lambda_{\mathrm{wm}}

. We report AUC the score.

Figure 9: Effect of training objective components on substitution performance on General dataset. We report the TPR@FPR = 1% as the score.

Figure 10: Effect of green ratio

\rho

on substitution performance on Math dataset.

Figure 11: Effect of overlap

\tau

on substitution performance on Math dataset.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

May 28, 2026

Implicit Identity Technologies for LLMs: Fingerprinting and Watermarking across Datasets, Models, and Generated Content

This paper presents a survey and taxonomy of LLM fingerprinting and watermarking for identity, ownership verification, provenance, and generated-content attribution. Large language models (LLMs) require substantial investments in data, computation, and expertise, and are increasingly deployed in high-stakes settings, making it critical to protect LLM-related assets and trace their origins. Existing work has rapidly expanded across dataset provenance, model ownership, and generated-content detection, but the field remains fragmented: fingerprinting and watermarking are often used inconsistently, and methods are typically studied within isolated asset-specific settings. To address this gap, we introduce implicit identity as a unifying abstraction for verifiable but not directly observable identity signals in LLM systems. We distinguish fingerprinting as non-intrusive identity derived from intrinsic characteristics, and watermarking as intrusive identity deliberately embedded into data, models, or generated content. We then propose a lifecycle-based taxonomy that organises techniques across datasets, models, and generated content, and further separates them by verification semantics: similarity-based attribution and keyed verification. Finally, we establish an evaluation framework centred on identifiability, robustness, and deployability, summarising representative metrics under realistic access and transformation regimes. By unifying terminology, lifecycle stages, and evaluation objectives, this survey provides a structured foundation for studying LLM identity technologies and for developing more reliable mechanisms for asset protection and provenance.

Bing Liu, Shunping Wang, Yufan Zhu +5

School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, China · State Grid Henan Marketing Service Center, Henan, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +3

Data Provenance in Language Models Data Provenance Language Model Fingerprinting LLM Watermarking

Ask PDF

Aug 30, 2026

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

Commercial LLM APIs advertise a specific foundation model, but the served backbone may be silently substituted, quantized, or wrapped, for example to save deployment costs. All existing audits decide backbone identity from the text-output channel, which is structurally fragile for agentic APIs because modern serving stacks (OpenAI, Anthropic, Gemini, Cloudflare Workers AI, LangGraph) discard text and expose only structured actions when the model calls a tool, and provider-injected system prompts can distort text distributions enough that text-channel tests falsely accuse honest providers of substituting the claimed model. We observe that recent agentic post-training internalizes tool-use directly into the weights, opening a new audit channel that the serving stack still exposes and that is largely invariant to deployment context. We introduce Agentic Provenance (AgentProv), the first action-based identity audit for agentic LLM APIs: AgentProv fingerprints a deployed model through its categorical tool-call distribution and decides identity via an MMD permutation test. AgentProv catches every substituted model (100% on 630 evaluated checkpoint pairs), while holding the false-positive rate under system-prompt injection at 7% (vs. 67% for MET and 53% for RUT). On third-party API endpoints, AgentProv's disagreements with MET are consistent with an independent token-count side-channel that detects provider-injected system prompts.

Xun Wang, Bihe Zhao, Michael Backes +2

CISPA Helmholtz Center for Information Security

LLM Auditing Language Model Fingerprinting

Ask PDF

Apr 20, 2026

Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs

Hosted-LLM providers have a silent-substitution incentive: advertise a stronger model while serving cheaper replies. Probe-after-return schemes such as SVIP leave a parallel-serve side-channel, since a dishonest provider can route the verifier's probe to the advertised model while serving ordinary users from a substitute. We propose a commit-open protocol that closes this gap. Before any opening request, the provider commits via a Merkle tree to a per-position sparse-autoencoder (SAE) feature-trace sketch of its served output at a published probe layer. A verifier opens random positions, scores them against a public named-circuit probe library calibrated with cross-backend noise, and decides with a fixed-threshold joint-consistency z-score rule. We instantiate the protocol on three backbones -- Qwen3-1.7B, Gemma-2-2B, and a 4.5x scale-up to Gemma-2-9B with a 131k-feature SAE. Of 17 attackers spanning same-family lifts, cross-family substitutes, and rank-<=128 adaptive LoRA, all are rejected at a shared, scale-stable threshold; the same attackers all evade a matched SVIP-style parallel-serve baseline. A white-box end-to-end attack that backpropagates through the frozen SAE encoder does not close the margin, and a feature-forgery attacker that never runs M_hon is bounded in closed form by an intrinsic-dimension argument. Commitment adds <=2.1% to forward-only wall-clock at batch 32.

Ziyang Liu

Independent Researcher

LLM Auditing

Ask PDF

Abstract

LLM Auditing

Explore similar work

Implicit Identity Technologies for LLMs: Fingerprinting and Watermarking across Datasets, Models, and Generated Content

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs