Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate

Haocheng Ye; Aoting Hu; Xinwei Zhang; Xunzhu Tang; Shuchao Pang; Jason Xue

cs.CROct 4, 2026

Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate

Authors: Haocheng Ye, Aoting Hu, Xinwei Zhang, Xunzhu Tang, Shuchao Pang, Jason Xue

Organizations: Nanjing University of Science and Technology · Anhui University of Technology · The Hong Kong Polytechnic University · University of Luxembourg ·

Abstract

Text watermarking helps identify AI-generated content, but its effect on factual reliability remains underexplored. In this paper, we study watermarking hallucination: factual errors induced or amplified by watermarking even when the required evidence is present in the context and the unwatermarked model can answer correctly. Using a controlled retrieval-augmented generation setting, we compare unwatermarked and watermarked generations under the same context, query, and decoding configuration, and quantify their factual accuracy decrease. Across six representative watermarking methods, including KGW, SWEET, DiPmark, GumbelSoft, Gumbel-Max, and SynthID watermarking, we consistently observe watermark-induced hallucination. Watermarked outputs can remain fluent while introducing factual errors. We attribute this failure mode to two mechanisms: (1) token perturbations in the current-step arising from method-specific reweighting or keyed sampling, and (2) prefix-induced attention drift, which accumulates through autoregressive decoding and weakens later attention to the factual context. Motivated by this analysis, we propose two plug-in interventions at the token and attention levels that can be integrated into existing watermarking methods to improve factuality. At a matched TPR of 0.90 at 1% FPR, combining the two interventions reduces factual errors by approximately 90% relative to watermark-only decoding while preserving fluency and comparable decoding efficiency. Overall, this work highlights factuality as a first-class criterion in watermark evaluation, alongside detectability and robustness, and calls for careful factuality validation before deploying watermarks in fact-critical applications.

Figures & tables

Panel of Figure 1 — Figure 1: Factual accuracy decrease under watermarking. (a) Net factual accuracy decrease, $\Delta_{\mathrm{fact}}$ , as a function of watermark detection true-positive rate (TPR) at a fixed false-positive rate (FPR) of 1%, across six watermarking methods. Error bars indicate paired bootstrap confidence intervals. (b) Breakdown of $\Delta_{\mathrm{fact}}$ by fact type for KGW.

Figure 2: Decomposition of watermarking hallucination. Clean decoding preserves the factual margin by following the fact-bearing context. Watermarked decoding can first induce prefix drift, e.g., changing “meeting” to the fluent alternative “session”, which weakens factual reliance and lowers the score of the factual token. The current-step score perturbation then further compresses the margin, allowing a fluent distractor such as May 21 to overtake the evidence-supported token May 12 .

Panel of Figure 3 — Figure 3: Attention-level interference under watermarking. (a) Factual attention coefficient $\alpha_{t}$ decreases as watermark detectability increases, measured by TPR@1%FPR, across six representative methods. (b) At a fixed detectability level, $\mathrm{TPR@1\%FPR}=0.9$ , factual attention is lower when the same target query is answered later, indicating that attention degradation accumulates over the watermarked prefix.

Method / setting	LLaMA3.1-8B			Mistral-7B			Qwen2.5-14B
Method / setting	TPR=.50	TPR=.90	TPR=.98	TPR=.50	TPR=.90	TPR=.98	TPR=.50	TPR=.90	TPR=.98
KGW [ 5 ]	2.1%	12.9%	22.3%	2.0%	14.4%	23.1%	2.3%	13.7%	22.7%
+FPTI	1.5%	2.4%	8.7%	1.5%	2.4%	8.9%	1.8%	2.6%	8.9%
+FPTI+FPAI	0.6%	0.9%	6.3%	0.8%	1.2%	6.3%	0.8%	1.1%	6.4%
SWEET [ 17 ]	2.3%	15.0%	23.1%	2.5%	15.6%	23.9%	2.9%	15.7%	24.8%
+FPTI	1.1%	1.9%	8.6%	1.3%	2.0%	8.6%	1.1%	2.1%	9.5%

Table 1: Effectiveness of fact-preserving interventions. We report the net factual accuracy decrease

\Delta_{\mathrm{fact}}

(%) across six watermarking methods, three model families, and three matched TPR@1%FPR levels. Lower values indicate less watermark-induced factual degradation. FPTI limits factual-token perturbations, and FPTI+FPAI further reduces prefix-induced errors.

Setting	TPR@1%FPR = 0.50				TPR@1%FPR = 0.90
Setting	$\Delta_{\mathrm{fact}}$ $\downarrow$	Improv. $\uparrow$	PPL $\downarrow$	Speed $\uparrow$	$\Delta_{\mathrm{fact}}$ $\downarrow$	Improv. $\uparrow$	PPL $\downarrow$	Speed $\uparrow$
No watermark	0.0%	–	1.45	1.52	0.0%	–	1.45	1.52
KGW [ 5 ] (baseline)	2.1%	0.0%	3.01	1.49	12.9%	0.0%	12.98	1.48
KGW [ 5 ] + DoLa [ 27 ]	1.8%	14.3%	1.98	1.41	9.6%	25.6%	2.84	1.40
KGW [ 5 ] + DeCoRe [ 28 ]	1.6%	23.8%	1.86	1.14	3.2%	75.2%	6.74	1.10
KGW [ 5 ] + FPTI (Ours)	1.5%	28.6%	2.47	1.48	2.4%	81.4%	6.20	1.47

Table 2: Comparing proposed interventions with hallucination mitigation baselines. We compare our interventions with the hallucination mitigation baselines DoLa [ 27 ] and DeCoRe [ 28 ] under KGW watermarking [ 5 ] .

\Delta_{\mathrm{fact}}

measures the net factual accuracy decrease, and Improv. denotes its relative reduction over KGW. PPL measures fluency, and Speed is measured in generated tokens per second.

Figure 4: Comparing trade-offs under matched detectability. Lower-left (lower PPL and lower

\Delta_{\mathrm{fact}}

) is better. Marker size denotes decoding speed, with larger circles indicating higher speed.

Figure 5: Position-dependent residual error under KGW. Blue: residual $\Delta_{\mathrm{fact}}$ after FPTI; orange: reduction in $\Delta_{\mathrm{fact}}$ by FPTI vs. standard watermarking.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Parameter	Value
Temperature	0.7	Top- $p$	0.9
Top- $k$	40	Repetition penalty	1.1
No-repeat n-gram size	3	Maximum input context	300 tokens
Maximum new tokens	300	Sampling	Enabled

Appendix

Table 3: Decoding configuration. The same decoding parameters are used for unwatermarked, watermarked, and intervention-enhanced generations unless an experiment explicitly varies watermark strength or detectability.

Setting	Output	Factual?
No watermark	Fasting glucose range: 145–162 mg/dL; postprandial peak: 210 mg/dL; nocturnal low: 78 mg/dL.	Yes
Watermark	Fasting glucose ranged 145–160 mg/dL; peak value 200–210 mg/dL; nocturnal low around 73 mg/dL.	No
FPTI+FPAI	Fasting sugar level: 145–162 mg/dL; peak value: 210 mg/dL; nocturnal low: 78 mg/dL.	Yes

Appendix

Table 4: Successful recovery under FPTI+FPAI. The standard watermark produces plausible but incorrect numerical values, while FPTI+FPAI restores the context-supported answer.

Setting	Output	Factual?
No watermark	Fasting glucose range: 145–162 mg/dL; postprandial peak: 210 mg/dL; nocturnal low: 78 mg/dL.	Yes
Watermark	Fasting glucrose range for this particular patient span from approximately between 145–155 to 148 milligrm/dltrange.	No
FPTI+FPAI	Fasting sugar readings’ range: 145–162 mg/dL. Specific value was not listed.	No

Appendix

Table 5: Incomplete recovery under FPTI+FPAI. Severe prefix drift leaves residual factual errors after both interventions.

Method	Correct $\rightarrow$ Incorrect	Incorrect $\rightarrow$ Correct	$\Delta_{\mathrm{fact}}$ [95% CI]
Unbiased Watermark ( $\delta$ -reweighting)	7.1%	3.5%	3.6% [3.1%, 3.9%]
TextSeal	7.7%	3.8%	3.9% [1.8%, 5.8%]
MorphMark	16.4%	4.3%	12.1% [11.5%, 13.7%]

Appendix

Table 6: Factual transitions for additional watermark families. Correct-to-incorrect and incorrect-to-correct transitions use the full set of 1,000 paired generations as their denominator. The net factual accuracy decrease is their difference.

Question: What case sets the key precedent for liability standards in the Ninth Circuit?
Context (RAG)	No Watermark	With Watermark (SynthID)
… Legal context Precedent: Doe v. Smith Citation: 2023 WL 183472 (9th Cir.) Summary: Establishes supervisory liability standards in the Ninth Circuit. …	The controlling precedent is Doe v. Smith , 2023 WL 183472 (9th Cir.), which establishes the framework for supervisory liability. Factually correct PPL = 4.17	The controlling authority is Doe v. Smithson , 2023 WL 183472 (9th Cir.). Entity-level factual error PPL = 3.31 ( lower )

Appendix

Table 7: Fluent does not imply factual. The watermarked output by SynthID [ 7 ] has lower perplexity but introduces an entity-level factual error, while the non-watermarked output remains correct. In our evaluation under the standard SynthID-style setting, such entity-level watermarking hallucinations occur in 1.7% of cases.

Figure 6: Token-level perplexity versus factual correctness. Many incorrect factual tokens appear in low-perplexity regions, showing why fluency-oriented metrics can miss watermark-induced factual errors.

Figure 7: Entropy of factual tokens before and after watermarking. We report the mean token-level entropy for factual tokens across four fact types. The dashed line marks the SWEET entropy threshold (

0.695

). Many factual tokens lie above this threshold, indicating that entropy-based protection does not reliably cover all fact-bearing tokens.

Text Watermarking

Text watermarking embeds detectable signals in generated text to indicate provenance, identify sources, or carry messages.

See all 44 papers

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks
Aug 13, 2026Xiaoyan Feng, Yanjun Zhang, He Zhang +2Data Provenance LLM Watermarking
Griffith University
AI Watermark Evidence Fails Forensic Readiness: An Empirical Evaluation
Jul 17, 2026Saifur Rahman Tamim, Amir Labib KhanWatermark Removal Text Watermarking
Department of Computer Science and Engineering Northern University Bangladesh
Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs
May 28, 2026Zhihao Wu, Gracia Gong, Qinglin Zhu +2Watermark Removal LLM Watermarking
Department of Informatics, King’s College London, UK · Department of Mathematics, Imperial College London, UK · Department of Statistics, University of Warwick, UK.

Aug 13, 2026J/K move · Enter open · S save

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks

Xiaoyan Feng, Yanjun Zhang, He Zhang, Leo Yu Zhang+1

Griffith University

Data Provenance LLM Watermarking Robust Watermarking Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

Watermarking LLM-generated text is an important task for tracing its provenance. Existing LLM watermarks preserve provenance under editing, but this same robustness allows an adversary to alter critical content while retaining attribution, a vulnerability known as piggyback spoofing. We introduce an innovative watermark that jointly provides provenance and tamper evidence. It co-embeds a robust signal and a fragile signal into each generated token. The signals share the same mechanism but use independent keys and different seeding windows over normalized text, making one resilient to edits and the other sensitive to reader-visible changes. Multiple rounds of unbiased tournament reweighting preserve the expected generation distribution, while a periodic round-allocation pattern controls the trade-off between the two signals. At detection, their scores form a two-dimensional space supporting three decisions: Intact, Tampered, and No-Watermark. Across two large language models and two prompt datasets, our method demonstrates the highest tamper-detection rate among the evaluated methods while maintaining competitive attribution robustness and perplexity. Ablation studies show that reliable three-state detection requires a well-defined notion of intactness, co-embedding of the two signals, and complementary sensitivity to edits.

Abstract

Text Watermarking

Explore similar work

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks

AI Watermark Evidence Fails Forensic Readiness: An Empirical Evaluation

Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs