Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate
Authors: Haocheng Ye, Aoting Hu, Xinwei Zhang, Xunzhu Tang, Shuchao Pang, Jason Xue
Organizations: Nanjing University of Science and Technology · Anhui University of Technology · The Hong Kong Polytechnic University · University of Luxembourg · CSIRO
Text watermarking helps identify AI-generated content, but its effect on factual reliability remains underexplored. In this paper, we study watermarking hallucination: factual errors induced or amplified by watermarking even when the required evidence is present in the context and the unwatermarked model can answer correctly. Using a controlled retrieval-augmented generation setting, we compare unwatermarked and watermarked generations under the same context, query, and decoding configuration, and quantify their factual accuracy decrease. Across six representative watermarking methods, including KGW, SWEET, DiPmark, GumbelSoft, Gumbel-Max, and SynthID watermarking, we consistently observe watermark-induced hallucination. Watermarked outputs can remain fluent while introducing factual errors. We attribute this failure mode to two mechanisms: (1) token perturbations in the current-step arising from method-specific reweighting or keyed sampling, and (2) prefix-induced attention drift, which accumulates through autoregressive decoding and weakens later attention to the factual context. Motivated by this analysis, we propose two plug-in interventions at the token and attention levels that can be integrated into existing watermarking methods to improve factuality. At a matched TPR of 0.90 at 1% FPR, combining the two interventions reduces factual errors by approximately 90% relative to watermark-only decoding while preserving fluency and comparable decoding efficiency. Overall, this work highlights factuality as a first-class criterion in watermark evaluation, alongside detectability and robustness, and calls for careful factuality validation before deploying watermarks in fact-critical applications.
Figures & tables
Figure 1: Factual accuracy decrease under watermarking. (a) Net factual accuracy decrease, Δfact , as a function of watermark detection true-positive rate (TPR) at a fixed false-positive rate (FPR) of 1%, across six watermarking methods. Error bars indicate paired bootstrap confidence intervals. (b) Breakdown of Δfact by fact type for KGW.
Figure 2: Decomposition of watermarking hallucination. Clean decoding preserves the factual margin by following the fact-bearing context. Watermarked decoding can first induce prefix drift, e.g., changing “meeting” to the fluent alternative “session”, which weakens factual reliance and lowers the score of the factual token. The current-step score perturbation then further compresses the margin, allowing a fluent distractor such as May 21 to overtake the evidence-supported token May 12 .
Figure 3: Attention-level interference under watermarking. (a) Factual attention coefficient αt decreases as watermark detectability increases, measured by TPR@1%FPR, across six representative methods. (b) At a fixed detectability level, TPR@1%FPR=0.9 , factual attention is lower when the same target query is answered later, indicating that attention degradation accumulates over the watermarked prefix.
Method / setting
LLaMA3.1-8B
Mistral-7B
Qwen2.5-14B
TPR=.50
TPR=.90
TPR=.98
TPR=.50
TPR=.90
TPR=.98
TPR=.50
TPR=.90
TPR=.98
KGW [ 5 ]
2.1%
12.9%
22.3%
2.0%
14.4%
23.1%
2.3%
13.7%
22.7%
+FPTI
1.5%
2.4%
8.7%
1.5%
2.4%
8.9%
1.8%
2.6%
8.9%
+FPTI+FPAI
0.6%
0.9%
6.3%
0.8%
1.2%
6.3%
0.8%
1.1%
6.4%
SWEET [ 17 ]
2.3%
15.0%
23.1%
2.5%
15.6%
23.9%
2.9%
15.7%
24.8%
+FPTI
1.1%
1.9%
8.6%
1.3%
2.0%
8.6%
1.1%
2.1%
9.5%
Table 1: Effectiveness of fact-preserving interventions. We report the net factual accuracy decrease Δfact (%) across six watermarking methods, three model families, and three matched TPR@1%FPR levels. Lower values indicate less watermark-induced factual degradation. FPTI limits factual-token perturbations, and FPTI+FPAI further reduces prefix-induced errors.
Setting
TPR@1%FPR = 0.50
TPR@1%FPR = 0.90
Δfact↓
Improv. ↑
PPL ↓
Speed ↑
Δfact↓
Improv. ↑
PPL ↓
Speed ↑
No watermark
0.0%
–
1.45
1.52
0.0%
–
1.45
1.52
KGW [ 5 ] (baseline)
2.1%
0.0%
3.01
1.49
12.9%
0.0%
12.98
1.48
KGW [ 5 ] + DoLa [ 27 ]
1.8%
14.3%
1.98
1.41
9.6%
25.6%
2.84
1.40
KGW [ 5 ] + DeCoRe [ 28 ]
1.6%
23.8%
1.86
1.14
3.2%
75.2%
6.74
1.10
KGW [ 5 ] + FPTI (Ours)
1.5%
28.6%
2.47
1.48
2.4%
81.4%
6.20
1.47
Table 2: Comparing proposed interventions with hallucination mitigation baselines. We compare our interventions with the hallucination mitigation baselines DoLa [ 27 ] and DeCoRe [ 28 ] under KGW watermarking [ 5 ] . Δfact measures the net factual accuracy decrease, and Improv. denotes its relative reduction over KGW. PPL measures fluency, and Speed is measured in generated tokens per second.
Figure 4: Comparing trade-offs under matched detectability. Lower-left (lower PPL and lower Δfact ) is better. Marker size denotes decoding speed, with larger circles indicating higher speed.
Figure 5: Position-dependent residual error under KGW. Blue: residual Δfact after FPTI; orange: reduction in Δfact by FPTI vs. standard watermarking.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Parameter
Value
Temperature
0.7
Top- p
0.9
Top- k
40
Repetition penalty
1.1
No-repeat n-gram size
3
Maximum input context
300 tokens
Maximum new tokens
300
Sampling
Enabled
Appendix
Table 3: Decoding configuration. The same decoding parameters are used for unwatermarked, watermarked, and intervention-enhanced generations unless an experiment explicitly varies watermark strength or detectability.
Table 4: Successful recovery under FPTI+FPAI. The standard watermark produces plausible but incorrect numerical values, while FPTI+FPAI restores the context-supported answer.
Fasting glucrose range for this particular patient span from approximately between 145–155 to 148 milligrm/dltrange.
No
FPTI+FPAI
Fasting sugar readings’ range: 145–162 mg/dL. Specific value was not listed.
No
Appendix
Table 5: Incomplete recovery under FPTI+FPAI. Severe prefix drift leaves residual factual errors after both interventions.
Method
Correct → Incorrect
Incorrect → Correct
Δfact [95% CI]
Unbiased Watermark ( δ -reweighting)
7.1%
3.5%
3.6% [3.1%, 3.9%]
TextSeal
7.7%
3.8%
3.9% [1.8%, 5.8%]
MorphMark
16.4%
4.3%
12.1% [11.5%, 13.7%]
Appendix
Table 6: Factual transitions for additional watermark families. Correct-to-incorrect and incorrect-to-correct transitions use the full set of 1,000 paired generations as their denominator. The net factual accuracy decrease is their difference.
Question: What case sets the key precedent for liability standards in the Ninth Circuit?
Context (RAG)
No Watermark
With Watermark (SynthID)
… Legal context Precedent: Doe v. Smith Citation: 2023 WL 183472 (9th Cir.) Summary: Establishes supervisory liability standards in the Ninth Circuit. …
The controlling precedent is Doe v. Smith , 2023 WL 183472 (9th Cir.), which establishes the framework for supervisory liability. Factually correct PPL = 4.17
The controlling authority is Doe v. Smithson , 2023 WL 183472 (9th Cir.). Entity-level factual error PPL = 3.31 ( lower )
Appendix
Table 7: Fluent does not imply factual. The watermarked output by SynthID [ 7 ] has lower perplexity but introduces an entity-level factual error, while the non-watermarked output remains correct. In our evaluation under the standard SynthID-style setting, such entity-level watermarking hallucinations occur in 1.7% of cases.
Figure 6: Token-level perplexity versus factual correctness. Many incorrect factual tokens appear in low-perplexity regions, showing why fluency-oriented metrics can miss watermark-induced factual errors.
Figure 7: Entropy of factual tokens before and after watermarking. We report the mean token-level entropy for factual tokens across four fact types. The dashed line marks the SWEET entropy threshold ( 0.695 ). Many factual tokens lie above this threshold, indicating that entropy-based protection does not reliably cover all fact-bearing tokens.
Watermarking LLM-generated text is an important task for tracing its provenance. Existing LLM watermarks preserve provenance under editing, but this same robustness allows an adversary to alter critical content while retaining attribution, a vulnerability known as piggyback spoofing. We introduce an innovative watermark that jointly provides provenance and tamper evidence. It co-embeds a robust signal and a fragile signal into each generated token. The signals share the same mechanism but use independent keys and different seeding windows over normalized text, making one resilient to edits and the other sensitive to reader-visible changes. Multiple rounds of unbiased tournament reweighting preserve the expected generation distribution, while a periodic round-allocation pattern controls the trade-off between the two signals. At detection, their scores form a two-dimensional space supporting three decisions: Intact, Tampered, and No-Watermark. Across two large language models and two prompt datasets, our method demonstrates the highest tamper-detection rate among the evaluated methods while maintaining competitive attribution robustness and perplexity. Ablation studies show that reliable three-state detection requires a well-defined notion of intactness, co-embedding of the two signals, and complementary sensitivity to edits.
Governments are increasingly mandating that LLM-generated content carry watermarks. The EU AI Act calls for markings that are "sufficiently reliable and robust." California's SB 942 requires disclosure that is "permanent or extraordinarily difficult to remove." Both mandates rest on an untested assumption: that watermark detection yields evidence reliable enough for courts. This paper tests that assumption directly. We evaluate three representative LLM watermarking methods -- KGW, Unigram, and the MarkLLM implementation of SynthID-Text -- against the Daubert admissibility criteria and the NIST SP 800-86 digital forensic process. To structure this evaluation, we propose a Forensic Readiness Score (FRS) framework with 12 criteria, three mandatory gates, and a 60-point scoring system. We focus on meaning-preserving paraphrase as the attack vector, since it is both legally realistic and difficult to dismiss as evidence tampering. The results raise serious evidentiary concerns. Out of 846 valid paraphrase runs across 15 diverse prompts per method, every single initially-detected KGW and Unigram text lost its watermark after paraphrasing -- 100% conditional removal. SynthID fared only slightly better at 98.3%. Even before any attack, false-negative rates were already high: 70% for KGW, 83% for Unigram, 80% for SynthID. The SynthID configuration also flagged 5.4% of paraphrased human-written controls as AI-generated and showed an 18.6% paradox rate, with 80% of its own pristine watermarked output landing in the uncertainty deadband. None of the three methods satisfy more than two of five Daubert factors. We also find that the FRS point-based scoring system, despite working as designed, cannot fully capture forensic uselessness -- a limitation worth noting for future framework design. These configurations, as tested, do not meet the evidentiary bar that courts require.
Saifur Rahman Tamim, Amir Labib Khan
Department of Computer Science and Engineering Northern University Bangladesh
Watermarking embeds statistical signatures in AI-generated text for detection and attribution. We reveal a fundamental vulnerability: when users access multiple models (today's reality), watermarks trivially fail. Watermarks perturb output distributions away from the original, and in competitive markets, these perturbations are typically independent across providers. We theoretically prove that averaging output probability distributions recovers the unwatermarked distribution with up to a second-order error term. Empirically, simply averaging 3-5 models cancels out these perturbations. We introduce WASH (Watermark Attenuation via Statistical Hybridisation), which solves practical challenges in ensemble generation: vocabulary misalignment and tokenisation differences across heterogeneous models. Experiments across six watermarking schemes and three LLMs show that averaging across 3 models suppresses detection z-scores from 5-300 to below 2 (below the detection threshold of 4) and reduces TPR at 5% FPR to below 50%, while improving quality by 27.5% and running 6 times faster than the best baseline on the long sequence generation. Our results suggest that robust AI-text detection via watermarking requires either accepting this fundamental vulnerability or unprecedented coordination among model providers.
Zhihao Wu, Gracia Gong, Qinglin Zhu +2
Department of Informatics, King’s College London, UK · Department of Mathematics, Imperial College London, UK · Department of Statistics, University of Warwick, UK.