Rethinking LLM Watermark Detection in Black-Box Settings: A Non-Intrusive Third-Party Framework

Zhuoshang Wang; Yubing Ren; Yanan Cao; Fang Fang; Xiaoxue Li; Li Guo

cs.CRMar 16, 2026

Rethinking LLM Watermark Detection in Black-Box Settings: A Non-Intrusive Third-Party Framework

Authors: Zhuoshang Wang, Yubing Ren, Yanan Cao, Fang Fang, Xiaoxue Li, Li Guo

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · National Computer Network Emergency Response Technical Team, Coordination Center of China (CNCERT/CC), Beijing, China

Abstract

While watermarking serves as a critical mechanism for LLM provenance, existing secret-key schemes tightly couple detection with injection, requiring access to keys or provider-side scheme-specific detectors for verification. This dependency creates a fundamental barrier for real-world governance, as independent auditing becomes impossible without compromising model security or relying on the opaque claims of service providers. To resolve this dilemma, we introduce TTP-Detect, a pioneering black-box framework designed for non-intrusive, third-party watermark verification. By decoupling detection from injection, TTP-Detect reframes verification as a relative hypothesis testing problem. It employs a proxy model to amplify watermark-relevant signals and a suite of complementary relative measurements to assess the alignment of the query text with watermarked distributions. Extensive experiments across representative watermarking schemes, datasets and models demonstrate that TTP-Detect achieves superior detection performance and robustness against diverse attacks.

Figures & tables

Figure 1: TTP-Detect framework overview.

	C4 Dataset								OpenGen Dataset
Watermark	Llama-3.1-8B				OPT-6.7B				Llama-3.1-8B				OPT-6.7B
	TPR	TNR	F1	AUC	TPR	TNR	F1	AUC	TPR	TNR	F1	AUC	TPR	TNR	F1	AUC
UPV	0.985	0.980	0.983	0.991	0.990	0.990	0.990	0.998	0.995	0.960	0.978	0.994	0.995	0.980	0.988	0.996
Logits-based Schemes
$\text{KGW}_{\texttt{TTP}}$	0.980	0.980	0.980	0.998	1.000	0.990	0.995	0.999	0.980	0.950	0.966	0.993	0.980	1.000	0.990	0.999
$\text{Unigram}_{\texttt{TTP}}$	1.000	0.990	0.995	0.999	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000

Table 1: Watermark detection performance of various watermarking schemes under TTP-Detect framework.

Figure 2: Robustness across various attack types.

Panel of Figure 3 — Figure 3: Ablation study for (a) proxy model and (b) relative measurement modules.

Panel of Figure 4 — Figure 4: (a) Impact of the size of reference set; (b) hyperparameter analysis of $k$ in Local Consistency Test.

Phase/Module	Model for Test	Time (s)
Service Provider Side
Generation of Reference Set	Llama-3.1-8B	6.1374
TTP Side
Representation Extraction	Qwen2.5-3B	0.8348
Local Consistency Test	$-$	0.0012
Relative Mahalanobis Score	$-$	0.0165

Table 2: Efficiency Analysis.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm	Source	Parameter	Value
UPV	Liu et al. (2024a)	window_size	3
		layers	5
		use_sampling	"True"
		delta	2.0
		sampling_temp	0.7
		train_num_samples	10000

Appendix

Table 3: Parameter configuration for chosen watermarking algorithms.

Panel of Figure 5 — Figure 5: Instruction template used to construct the LoRA fine-tuning data for representation extraction models.

Parameter	Value
finetuning_type	"lora"
lora_rank	8
lora_target	"all"
template	"qwen"
learning_rate	1.0e-4
num_train_epochs	3.0

Appendix

Table 4: Parameter configuration for the fine-tuning of proxy model using LLaMAFactory.

Panel of Figure 6 — Figure 6: Prompt used for the Rephrase-GPT5.1 paraphrasing attack.

Figure 7: Impact of generated text length.

Figure 8: Hyperparameter analysis of

\lambda

in Adaptive Rank Test module.

Panel of Figure 9 — Figure 9: Module ablation results for various watermarking schemes.

Panel of Figure 10 — Figure 10: Reverse-prompting instruction used to reconstruct an unknown user prompt from the query text.

LLM Watermarking

Large language model watermarking embeds or detects signals in text and models to trace provenance, ownership, and unauthorized use.

See all 111 papers

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization
Jul 11, 2026Huy Phan, Kieu Dang, Ojaswi Dulal +4Data Provenance LLM Watermarking
Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA
Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks
Aug 13, 2026Xiaoyan Feng, Yanjun Zhang, He Zhang +2Data Provenance LLM Watermarking
Griffith University
Could LLM Watermark Detection be Public?
Oct 8, 2026Georgios Milis, Tom Sander, Tomáš Souček +2LLM Watermarking Robust Watermarking
FAIR, Meta Superintelligence Labs · University of Maryland, College Park

Jul 11, 2026J/K move · Enter open · S save

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

Huy Phan, Kieu Dang, Ojaswi Dulal, Aiham AL Shukairi+3

Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA

Data Provenance LLM Watermarking Robust Watermarking

Open paper AskPDF arXiv

As large language models (LLMs) become widely deployed, their outputs can be copied, transformed, and redistributed at scale without reliable evidence of origin, creating risks for trust, accountability, intellectual property (IP) protection, and high-stakes decision-making. LLM watermarking addresses this problem by embedding detectable signals into text during or after generation. However, existing methods vary in design assumptions, threat models, and evaluation criteria, while deployment choices such as watermark placement, detection authority, and key management affect reliability, security, and scalability. This paper systematizes LLM watermarking as provenance infrastructure for large-scale data ecosystems. We organize existing approaches along four deployment dimensions: insertion point, verification authority, operational state, and transformation threat model, and relate them to the big data requirements of Volume, Velocity, Variety, Veracity, and Value. We further introduce a Big Data Watermarking Readiness framework centered on four deployment workloads: online generation, streaming detection, transformation pipelines, and ecosystem governance. The framework connects these workloads to system-level requirements including throughput, false-positive control, robustness, cross-domain reliability, governance, and downstream utility. Our analysis highlights a gap between benchmark performance and deployment readiness: false positives accumulate at scale, repeated transformations weaken watermark signals, computational overhead can limit online deployment, and centralized verification can create governance bottlenecks. We conclude with an evaluation blueprint and research directions for scalable, trustworthy provenance in big data ecosystems.

Abstract

LLM Watermarking

Explore similar work

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks

Could LLM Watermark Detection be Public?