A Unified Attack Framework against Statistical Generative Watermarks for Large Language Models

Romina Omidi; Nima Naderloui; Yuan Hong; Binghui Wang

cs.CROct 20, 2026

A Unified Attack Framework against Statistical Generative Watermarks for Large Language Models

Authors: Romina Omidi, Nima Naderloui, Yuan Hong, Binghui Wang

Organizations: Illinois Institute of Technology · University of Connecticut

Abstract

Watermarking has emerged as a key tool for identifying text generated by large language models (LLMs), with current schemes embedding statistical signals into the generation process to enable later detection. In this work, we identify a shared weakness across a broad class of popular LLM watermarking methods, including KGW (and its variants), GumbelSoft, DipMark, ITS, and SynthID. We show that these schemes exhibit a fundamental link between their detection scores and the token probability shifts they introduce during generation. This link, which we call score-ratio alignment, allows an attacker with only black-box query access to the model--and no knowledge of secret keys or model weights--to produce text that evades detection while remaining fluent and natural. We prove that the resulting attack is optimal across all covered watermark families, and show that the weakness is intrinsic: any watermark designed to maximize statistical detection power necessarily induces the very structure that attackers can exploit. We evaluate our attack on seven popular generative LLM watermarking schemes, three LLMs, and tasks, which reduce detection rates to at most 0.05 across all settings while preserving text quality. Our findings show that current watermarking methods can be reliably defeated under realistic attack conditions, highlighting the need for fundamentally different watermarking designs that decouple detection scores from token probability shifts.

Figures & tables

Scheme	Type	PRF output $\mathcal{E}$	Decoder $\Gamma$	Token score $\phi(\xi_{t},x_{t})$	Detection statistic $\Phi$
KGW [ 20 ]	D	$\{0,1\}^{\|\mathcal{V}\|}$	$\operatorname{softmax}(\mathbf{l}_{t}+\delta\xi_{t};\tau)$	$\xi_{t}[x_{t}]\in\{0,1\}$	$Z$ -score ( 1 )
Token-Specific KGW [ 17 ]	D	$\{0,1\}^{\|\mathcal{V}\|}$ (token-dep.)	$\operatorname{softmax}((\mathbf{l}_{t}{+}\delta_{t}{\odot}\xi_{t});\tau)$	$\delta_{t}[x_{t}]\,\xi_{t}[x_{t}]$	Normalized sum ( 2 )
Adaptive KGW [ 24 ]	D	$\{0,1\}^{\|\mathcal{V}\|}$ (adaptive $\gamma_{t}$ )	$\operatorname{softmax}((\mathbf{l}_{t}{+}\delta_{t}\xi_{t});\tau)$	$\delta_{t}\,\xi_{t}[x_{t}]$	Normalized sum ( 2 )
GumbelSoft [ 10 ]	DF	$\mathbb{R}^{\|\mathcal{V}\|}$ (Gumbel)	$\operatorname{softmax}((\mathbf{l}_{t}{+}\xi_{t});\tau)$	$\xi_{t}[x_{t}]$	Scaled mean ( 3 )
DipMark [ 41 ]	DF	Permutation $\iota_{t}$	Rank-reweighted ( 4 )	$\mathbf{1}[x_{t}\in\text{green region}]$	$Z$ -score ( 5 )
ITS [ 22 ]	DF	$(\mu_{t},\pi_{t})$	Inverse CDF ( 6 )	$\|\mu_{t}-\eta(\pi_{t}(x_{t}))\|$	Negative mean ( 8 )

Table 1: Summary of the seven watermarking schemes covered by our framework. “Distortional” (D) schemes modify the sampling distribution; “distortion-free” (DF) schemes preserve it in expectation.

\gamma

denotes the green-list fraction,

\delta

the logit boost,

\xi_{t}

the pseudorandom vector, and

\gamma_{\mathrm{EM}}

the Euler–Mascheroni constant.

\widetilde{t}^{(1)},\ldots,\widetilde{t}^{(N)}\sim\widetilde{M}(\cdot\mid h)

Algorithm 1 Black-Box Attack on LLM Watermarking

	C4				FakeNews				Eli-5
Scheme	TPR $\downarrow$	$p$ -value $\uparrow$	$Z$ -score $\downarrow$	PPL $\downarrow$	TPR $\downarrow$	$p$ -value $\uparrow$	$Z$ -score $\downarrow$	PPL $\downarrow$	TPR $\downarrow$	$p$ -value $\uparrow$	$Z$ -score $\downarrow$	PPL $\downarrow$
KGW	0.01	0.98	-4.61	14.72	0.01	0.98	-4.23	15.71	0.02	0.98	-4.39	15.16
Token-Specific KGW	0.01	0.97	-4.18	5.91	0.01	0.97	-3.47	6.35	0.01	0.96	-2.13	6.48
Adaptive KGW	0.03	0.24	0.73	3.86	0.02	0.34	0.29	4.54	0.03	0.31	0.46	4.13
GumbelSoft	0.02	0.49	-0.06	4.51	0.01	0.63	-0.36	4.92	0.01	0.61	-0.38	4.94
DipMark	0.01	0.74	-0.63	5.04	0.02	0.66	-0.41	5.27	0.01	0.64	-0.24	5.61

Table 2: Attack results on three datasets using OPT-1.3B (base model) as the surrogate model. TPR is reported at a 1% false positive rate. Larger

p

-values and lower TPRs indicate stronger evasion. A

p

-value above

0.05

indicates unsuccessful watermark detection. PPL denotes the perplexity of adversarially generated text; lower values correspond to more fluent outputs.

	C4				FakeNews				Eli-5
Scheme	TPR $\downarrow$	$p$ -value $\uparrow$	Z-score $\downarrow$	PPL $\downarrow$	TPR $\downarrow$	$p$ -value $\uparrow$	Z-score $\downarrow$	PPL $\downarrow$	TPR $\downarrow$	$p$ -value $\uparrow$	Z-score $\downarrow$	PPL $\downarrow$
KGW	0.00	1.00	-4.94	13.18	0.00	0.99	-4.67	13.99	0.03	0.99	-4.67	13.86
Token-Specific KGW	0.00	1.00	-5.37	6.23	0.00	0.99	-3.96	6.23	0.00	0.98	-2.27	6.56
Adaptive KGW	0.01	0.07	1.47	3.19	0.00	0.058	1.57	4.42	0.01	0.28	0.58	4.06
GumbelSoft	0.00	0.46	0.10	4.39	0.00	0.65	-0.40	4.98	0.00	0.66	-0.42	4.89
DipMark	0.00	0.68	-0.48	4.95	0.00	0.59	-0.25	5.15	0.00	0.61	-0.29	5.27

Table 3: Attack results on three datasets using LLaMA-7B as the surrogate model.

	C4				FakeNews				Eli-5
Scheme	TPR $\downarrow$	$p$ -value $\uparrow$	Z-score $\downarrow$	PPL $\downarrow$	TPR $\downarrow$	$p$ -value $\uparrow$	Z-score $\downarrow$	PPL $\downarrow$	TPR $\downarrow$	$p$ -value $\uparrow$	Z-score $\downarrow$	PPL $\downarrow$
KGW	0.00	0.99	-4.34	15.94	0.00	0.99	-3.98	16.43	0.00	0.99	-4.12	16.37
Token-Specific KGW	0.00	0.99	-3.89	5.63	0.00	0.99	-3.14	6.47	0.00	0.97	-1.98	6.56
Adaptive KGW	0.02	0.28	0.57	4.23	0.00	0.56	-0.17	4.65	0.02	0.28	0.58	4.06
GumbelSoft	0.01	0.43	0.17	4.62	0.00	0.69	-0.50	4.86	0.00	0.66	-0.42	4.89
DipMark	0.00	0.80	-0.86	5.12	0.00	0.62	-0.31	5.36	0.00	0.57	-0.18	5.93

Table 4: Attack results on three datasets using LLaMA-13B as the surrogate model.

Panel of Figure 1 — Figure 1: Perplexity (left axis) and watermark Z-score (right axis) as a function of #queries $N$ .

Panel of Figure 2 — Figure 2: Effect of generated text length $T$ (tokens) on attack effectiveness and text quality.

Panel of Figure 3 — Figure 3: Effect of $K$ in Top-K candidate set on attack effectiveness and text quality.

LLM Watermarking

Large language model watermarking embeds or detects signals in text and models to trace provenance, ownership, and unauthorized use.

See all 111 papers

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks
Aug 13, 2026Xiaoyan Feng, Yanjun Zhang, He Zhang +2Data Provenance LLM Watermarking
Griffith University
TripPattern: A Pattern-based Text Watermarking Method for Large Language Models
Sep 11, 2026Sangjun Moon, Dasom Choi, Jingun Kwon +3Text Watermarking LLM Watermarking
Chungnam National University · Nara Institute of Science and Technology (NAIST) · Institute of Science Tokyo
LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization
Jul 11, 2026Huy Phan, Kieu Dang, Ojaswi Dulal +4Data Provenance LLM Watermarking
Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA

Aug 13, 2026J/K move · Enter open · S save

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks

Xiaoyan Feng, Yanjun Zhang, He Zhang, Leo Yu Zhang+1

Griffith University

Data Provenance LLM Watermarking Robust Watermarking Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

Watermarking LLM-generated text is an important task for tracing its provenance. Existing LLM watermarks preserve provenance under editing, but this same robustness allows an adversary to alter critical content while retaining attribution, a vulnerability known as piggyback spoofing. We introduce an innovative watermark that jointly provides provenance and tamper evidence. It co-embeds a robust signal and a fragile signal into each generated token. The signals share the same mechanism but use independent keys and different seeding windows over normalized text, making one resilient to edits and the other sensitive to reader-visible changes. Multiple rounds of unbiased tournament reweighting preserve the expected generation distribution, while a periodic round-allocation pattern controls the trade-off between the two signals. At detection, their scores form a two-dimensional space supporting three decisions: Intact, Tampered, and No-Watermark. Across two large language models and two prompt datasets, our method demonstrates the highest tamper-detection rate among the evaluated methods while maintaining competitive attribution robustness and perplexity. Ablation studies show that reliable three-state detection requires a well-defined notion of intactness, co-embedding of the two signals, and complementary sensitivity to edits.

Abstract

LLM Watermarking

Explore similar work

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks

TripPattern: A Pattern-based Text Watermarking Method for Large Language Models

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization