Beyond A Fixed Seal: Adaptive Stealing Watermark in Large Language Models

Shuhao Zhang; Yuli Chen; Jiale Han; Bo Cheng; Jiabao Ma

cs.CRApr 13, 2026

Beyond A Fixed Seal: Adaptive Stealing Watermark in Large Language Models

Authors: Shuhao Zhang, Yuli Chen, Jiale Han, Bo Cheng, Jiabao Ma

Organizations: State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications · Hong Kong University of Science and Technology

Abstract

Watermarking provides a critical safeguard for large language model (LLM) services by facilitating the detection of LLM-generated text. Correspondingly, stealing watermark algorithms (SWAs) derive watermark information from watermarked texts generated by victim LLMs to craft highly targeted adversarial attacks, which compromise the reliability of watermarks. Existing SWAs rely on fixed strategies, overlooking the non-uniform distribution of stolen watermark information and the dynamic nature of real-world LLM generation processes. To address these limitations, we propose Adaptive Stealing (AS), a novel SWA featuring enhanced design flexibility through Position-Based Seal Construction and Adaptive Selection modules. AS operates by defining multiple attack perspectives derived from distinct activation states of contextually ordered tokens. During attack execution, AS dynamically selects the optimal perspective based on watermark compatibility, generation priority, and dynamic generation relevance. Our experiments demonstrate that AS significantly increases steal efficiency against target watermarks under identical experimental conditions. These findings highlight the need for more robust LLM watermarks to withstand potential attacks. We release our code to the community for future research\footnote{https://github.com/DrankXs/AdaptiveStealingWatermark}.

Figures & tables

Figure 1: Part 1 illustrates the watermark generation process in LLMs, while Part 2 depicts the traditional Stealing Watermark Algorithm (SWA).

Figure 2: The overall process of Adaptive Stealing (AS). Red arrows indicate the actions of AS, while black arrows indicate standard generation processes without the intervention of watermarking.

		$LM_{vic}$ =OPT-2.7b						$LM_{vic}$ =Llama3-8b
		Dolly			Harm			Dolly			Harm
Watermark	Method	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%
-	Random	-	0.50	0.01	-	0.50	0.01	-	0.50	0.01	-	0.50	0.01
KGW	Dipper	0 1.225	0.75	0.05	0 1.207	0.75	0.06	0 1.585	0.80	0.17	0 1.650	0.80	0.27
	Dipper+WS	0 0.958	0.69	0.08	0 1.008	0.72	0.08	0 0.507	0.61	0.06	0 0.470	0.61	0.06
	Dipper+AS	-0.002	0.47	0.03	0 0.169	0.50	0.03	-0.922	0.37	0.02	-0.519	0.40	0.03

Table 1: Results of scrubbing attack for different watermarks and different scrubbing method. “Random” represents random classification during detection. Lower WCS, AUC, and TPR@1% values indicate better attack effectiveness.

		$LM_{vic}$ =OPT-2.7b						$LM_{vic}$ =Llama3-8b
		Dolly			Harm			Dolly			Harm
Watermark	Mode	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%
KGW	w/o Attack	8.880	1.00	1.00	8.933	1.00	1.00	7.991	1.00	1.00	8.072	1.00	1.00
	WS	1.228	0.74	0.06	1.404	0.77	0.11	1.333	0.74	0.15	1.431	0.75	0.19
	AS(Ours)	2.698	0.91	0.36	2.770	0.92	0.42	2.646	0.89	0.47	2.648	0.90	0.48
SynthID	w/o Attack	0.584	1.00	1.00	0.583	1.00	1.00	0.569	1.00	1.00	0.570	1.00	1.00

Table 2: Results of spoofing attacks on different watermarks. w/o Attack represents the detectability performance of victim. Higher WCS, AUC, and TPR@1% values indicate better attack effectiveness.

	$n^{o}$								AVE	AS(Ours)
	0	1	2	3	4	5	6	7	AVE	AS(Ours)
$\|ctx\|=1$ & Left	0.675	0.997	0.749	0.929	0.599	0.843	0.540	0.584	0.821	0.907
$\|ctx\|=2$ & Left	0.590	0.709	0.994	0.945	0.649	0.635	0.855	0.692	0.834	0.864
$\|ctx\|=3$ & Left	0.560	0.559	0.680	0.694	0.993	0.923	0.903	0.737	0.825	0.886
$\|ctx\|=4$ & Left	0.527	0.501	0.522	0.535	0.621	0.559	0.559	0.511	0.532	0.613
$\|ctx\|=3$ & Min	0.692	0.940	0.915	0.932	0.888	0.860	0.845	0.753	0.850	0.941

Table 3: Spoofing attack performance (AUC) across different watermark configurations using various seals.

LM_{vic}

=Llama3-8b, and dataset is Dolly.

n^{o}

denotes the specific forged seal used for attack, “AVE” represents equally weighted ensemble of seals with

n^{o}

from 0 to 7. Left, Min and Max respectively indicates LeftHash-scheme, MinHash-scheme and MaxHash-scheme.

Figure 3: Trade-off between text quality (PPL) and watermark detectability (WCS) under spoofing attacks on KGW with varying

\delta_{att}

Figure 4: Results of spoofing attacks on KGW with varying

|D_{w}|

. The minimum

|D_{w}|

is 100.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	$LM_{vic}$ =OPT-2.7b		$LM_{vic}$ =Llama3-8b
	Dolly	Harm	Dolly	Harm
w/o WC	2.454	2.710	2.424	2.607
w/o GP	0.358	0.172	0.438	0.381
w/o DGR	2.666	2.752	2.615	2.725
AS	2.698	2.770	2.646	2.648

Appendix

Table 4: Ablation study of Adaptive Selection components (WC, GP, DGR) for spoofing attacks. Victim watermark is KGW, evaluation metric is WCS.

	$LM_{vic}$ =OPT-2.7b		$LM_{vic}$ =Llama3-8b
	Dolly	Harm	Dolly	Harm
top32	0 0.020	0 0.097	0 0.018	-0.069
top64	-0.020	0 0.020	0 0.011	0 0.032
top128	0 0.020	0 0.060	0 0.024	0 0.016
top256	0 0.021	0 0.048	0 0.003	0 0.044
top512	0 0.031	0 0.101	0 0.017	0 0.012

Appendix

Table 5: The confrontation results of DGR and non-DGR settings.

		$LM_{vic}$ =OPT-2.7b						$LM_{vic}$ =Llama3-8b
		Dolly			Harm			Dolly			Harm
$LM_{att}$	SWA	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%	WCS	AUC	TPR@1%
OPT-1.3b	WS	4.140	0.99	0.80	4.290	0.99	0.89	2.925	0.94	0.55	2.974	0.95	0.59
OPT-1.3b	AS(Ours)	5.060	1.00	0.92	5.395	1.00	0.98	4.156	0.99	0.83	4.174	0.99	0.88
OPT-2.7b	WS	4.054	0.98	0.74	4.144	0.98	0.79	2.876	0.94	0.53	2.900	0.94	0.56
OPT-2.7b	AS(Ours)	5.058	0.99	0.93	5.263	1.00	0.93	4.062	0.98	0.82	4.296	0.99	0.90

Appendix

Table 6: Results of spoofing attacks on KGW with different

LM_{att}

Figure 5: Results of spoofing attacks on KGW with different lengths of $ctx$ .

	$\|ctx\|$ =1	$\|ctx\|$ =2	$\|ctx\|$ =3	$\|ctx\|$ =4
KGW	0 8.65	0 8.68	0 8.63	0 8.63
SynthID	0 9.11	0 9.06	0 9.09	0 9.10
Unbiased	10.04	10.11	12.92	12.19
WS	10.65	11.07	10.85	11.91
AS(Ours)	11.69	11.15	16.92	16.34

Appendix

Table 7: Execution time (s) of watermarks and attack algorithms under varying

|ctx|

Full	Partial	Empty	WCS	AUC	WCS	AUC	WCS	AUC	WCS	AUC	WCS	AUC	WCS	AUC
			KGW				SynthID				Unbiased
			Dolly		Harm		Dolly		Harm		Dolly		Harm
✓	✗	✗	1.321	0.74	1.378	0.75	0.503	0.61	0.503	0.61	- 0.419	0.70	0.412	0.70
✗	✗	✓	0.373	0.54	0.385	0.56	0.500	0.49	0.499	0.45	-0.022	0.49	0.064	0.54
✓	✓	✓	0.994	0.70	1.056	0.70	0.502	0.58	0.502	0.56	- 0.237	0.63	0.291	0.65
✗	✓	✗	0.915	0.68	1.062	0.71	0.501	0.55	0.501	0.55	- 0.081	0.55	0.164	0.59

Appendix

Table 8: Spoofing attack result on different watermarks with WS as stealing watermark algorithm. The left three columns represent different settings of WS. Higher metric values indicate better attack performance.

Figure 6: Results of scrubbing attacks on KGW with different $|ctx|$ . The attack method is Dipper, $LM_{vic}$ =OPT-2.7b, and dataset is c4.

		KGW		SynthID		Unbiased
SWA	$LM_{vic}$	Dolly	Harm	Dolly	Harm	Dolly	Harm
CWS	OPT-2.7B	0.450	0.514	0.537	0.513	0.501	0.520
CWS	Llama3-8b	0.403	0.452	0.493	0.526	0.504	0.496
WRA	OPT-2.7b	0.580	0.574	0.509	0.532	0.500	0.525
WRA	Llama3-8b	0.544	0.572	0.531	0.526	0.522	0.484

Appendix

Table 9: Spoofing attack results of CWS and WRA. The metric is AUC.

	$LM_{vic}$ =OPT-2.7b		$LM_{vic}$ =Llama3-8b
	Dolly	Harm	Dolly	Harm
KGW	0.977	0.902	0.970	0.899
SynthID	0.982	0.897	0.938	0.855
Unbiased	0.976	0.905	0.939	0.860

Appendix

Table 10: Scrubbing attack results of MIP.

LM_{att}

=Dipper and metric is AUC. Lower AUC values indicate better watermark removal performance.

Adversarial Attacks on LLMs

Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.

See all 355 papers

A Unified Attack Framework against Statistical Generative Watermarks for Large Language Models
Oct 20, 2026Romina Omidi, Nima Naderloui, Yuan Hong +1LLM Watermarking Robustness of Machine-Generated Text Detection
Illinois Institute of Technology · University of Connecticut
Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs
May 8, 2026Jonathan Hong Jin Ng, Anh Tu Ngo, Anupam ChattopadhyayWatermark Removal LLM Watermarking
College of Computing and Data Science Nanyang Technological University Singapore
OpenStamp: A Watermark for Open-Source Language Models
Aug 28, 2026Miroojin Bakshi, Saksham Rastogi, Danish PruthiText Watermarking LLM Watermarking
Indian Institute of Science · Carnegie Mellon University

Oct 20, 2026J/K move · Enter open · S save

A Unified Attack Framework against Statistical Generative Watermarks for Large Language Models

Romina Omidi, Nima Naderloui, Yuan Hong, Binghui Wang

Illinois Institute of Technology · University of Connecticut

LLM Watermarking Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

Watermarking has emerged as a key tool for identifying text generated by large language models (LLMs), with current schemes embedding statistical signals into the generation process to enable later detection. In this work, we identify a shared weakness across a broad class of popular LLM watermarking methods, including KGW (and its variants), GumbelSoft, DipMark, ITS, and SynthID. We show that these schemes exhibit a fundamental link between their detection scores and the token probability shifts they introduce during generation. This link, which we call score-ratio alignment, allows an attacker with only black-box query access to the model--and no knowledge of secret keys or model weights--to produce text that evades detection while remaining fluent and natural. We prove that the resulting attack is optimal across all covered watermark families, and show that the weakness is intrinsic: any watermark designed to maximize statistical detection power necessarily induces the very structure that attackers can exploit. We evaluate our attack on seven popular generative LLM watermarking schemes, three LLMs, and tasks, which reduce detection rates to at most 0.05 across all settings while preserving text quality. Our findings show that current watermarking methods can be reliably defeated under realistic attack conditions, highlighting the need for fundamentally different watermarking designs that decouple detection scores from token probability shifts.

Abstract

Adversarial Attacks on LLMs

Explore similar work

A Unified Attack Framework against Statistical Generative Watermarks for Large Language Models

Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs

OpenStamp: A Watermark for Open-Source Language Models