cs.CROct 8, 2026
SaveCould LLM Watermark Detection be Public?
Organizations: FAIR, Meta Superintelligence Labs · University of Maryland, College Park
Abstract
Watermarking large language models is popular for tracing chatbot and agentic outputs, yet detectors remain unreleased since exposing them could let attackers do targeted edits with the detector's feedback. However, watermarks are already vulnerable to uninformed tampering attacks. We thus first quantify whether a public detector would be an additional liability in a deployment setting at varying levels of access, from token-level scores to a binary verdict. Second, we introduce a split-key public-private watermarking method that exposes one key through a public detector while keeping the other for full verification and forensics. An informed attacker can only move the public signal, creating an imbalance between public and private scores. We introduce a statistical test for this imbalance, and combine it with the full key verdict in a two-stage mechanism. Third, we evaluate the split-key method on a wide range of removal and forgery attacks, comparing the uninformed to detector-informed settings. Public detection improves removal only at small edit budgets, since plain rephrasing already strips the watermark at a lower quality cost, but it does enable forgery, which the private pipeline can identify. Overall, releasing half of the watermark enables transparency and interoperability, and tampering with the released half stays detectable. This bounds the provider's liability and questions the need to keep detectors fully private.
Figures & tables
Figure 1: Overview of our approach. Alice, an LLM provider, splits the watermark signal across two keys, releasing a detector that uses only the public half while keeping private. Bob may try to use the public detector for informed attacks that remove (or forge) the watermark more effectively than uninformed attacks. Alice, who sees both keys, flags it as tampered from the imbalance between the two scores. Therefore, Bob has a better chance of evasion by using uninformed attacks, an already existing risk for Alice.
Figure 2: Removal and forgery attacks, under levels of detector access ranging from none to token-level scores.
Figure 3: Overview of the two-stage test. (Left) The public detector only tests based on the public -value . (Right) The private detector first tests to get , then re-evaluates it with the tampering test on . If we test for forgery ( ), if for removal ( ). Genuine watermarked texts have , while tampered ones are flagged as forged or removed.
| Removal | Forgery | |||||||||||
| Rephrase (Add.) | Word edits (10%) | Rephrase (Add.) | Word edits (10%) | |||||||||
| Detector: | no-box | black-box | white-box | no-box | black-box | white-box | no-box | black-box | white-box | no-box | black-box | white-box |
| Gumbel-max | 70.4% | 81.4% | 99.9% | 1.5% | 1.6% | 72.2% | 0.0% | 0.6% | 20.8% | 0.2% | 0.3% | 56.3% |
| Gumbel-max | 82.0% | 92.9% | 99.5% | 6.7% | 8.4% | 97.8% | 0.1% | 0.3% | 6.2% | 0.1% | 0.1% | 48.3% |
| Maryland | 90.1% | 96.1% | 100.0% | 8.6% | 10.7% | 15.9% | 0.4% | 1.1% | 78.6% | 0.1% | 0.5% | 1.3% |
| Maryland | 95.2% | 98.9% | 100.0% | 34.4% | 40.4% | 98.7% | 0.1% | 0.9% | 38.8% | 0.1% | 0.3% | 1.9% |
Table 1: Removal and forgery ASR at three access levels, on baselines with keys, where every key is published, for Gumbel-max and Maryland ( ) watermarks. Attacks are additive rephrasing and word edits, while distortion is BERT-F1 ( ), pooled over the four watermarks. A white-box detector is the most exploitable tier, and the only setting that directly enables forgery. Using a black-box detector as a stopping rule gives removal a few points of ASR at consistently lower distortion.
Figure 4: Attack success rate under the two-stage test , for removal (left) and forgery (right). Each bar represents the the attacker sees from the public detector, split into the part the test on the fused watermark score rejects, the part the tampering test then catches, and the that survives both. Informed attacks fool the public detector, but are overall not more effective than uninformed paraphrasing for removal, and are often flagged for forgery.
Figure 5: (Left) (dashed) and (solid) against median distortion, for removal by rephrasing (top) and word edits (bottom), at the no-box , white-box and black-box tiers. The swept parameter is temperature or edit fraction (for black-box , the maximum it will spend). The informed attacks with word edits are the most harmful ones for the public detector, but get caught by the two-stage tests (lower ). Overall, informed attacks are stronger only at low distortion, while no-box rephrasing is more effective at higher distortion. (Right) Median of the public, private and fused scores against median distortion, for the white-box word edit sweep, removal (top) and forgery (bottom). The private score drops slower than the public one for removal and stays fixed for forgery, so the gap the tampering test examines grows with attack strength.
| Removal | Forgery | |||||||||||||
| Rephrase | Word edits | Rephrase | Word edits | |||||||||||
| Metric | No-box | Adap. ( ) | Adap. ( ) | Adap. ( ) | No-box | Adap. ( ) | Adap. ( ) | Adap. ( ) | Adap. ( ) | Adap. ( ) | Adap. ( ) | Adap. ( ) | Adap. ( ) | Adap. ( ) |
| ASR pub | 92.3% | 99.5% | 99.9% | 100.0% | 62.5% | 99.9% | 99.8% | 99.8% | 18.1% | 19.6% | 20.8% | 99.8% | 99.9% | 99.9% |
| ASR fus | 82.0% | 88.5% | 95.1% | 99.4% | 26.9% | 8.8% | 30.8% | 99.3% | 4.8% | 7.5% | 5.5% | 45.8% | 69.8% | 85.1% |
| ADR | 0.4% | 0.2% | 0.2% | 0.7% | 0.0% | 0.0% | 0.3% | 99.4% | 0.0% | 1.3% | 5.5% | 10.9% | 36.0% | 66.2% |
| ASR net | 80.7% | 88.1% | 94.9% | 98.7% | 25.6% | 8.8% | 30.7% | 0.6% | 3.2% | 5.9% | 4.0% | 40.8% | 44.7% | 28.8% |
Table 2: Adaptive attacks against the no-box baseline and their unaware white-box counterparts, at the same operating points (rephrasing , word edits ). Bold marks the highest per family. Adaptive attacks increase the up to a point, with the private pipeline rejecting a smaller portion of them. However, they become detectable again at the large overshoot factor of . Despite the limited but real liability of the white-box detector in the adaptive attacks scenario, the with black-box adaptive attacks still matches no-box baselines (App. D.5 ).
Figure 6: The split-key construction applies to a Maryland (top) and a SynthID watermark (bottom), where the fused watermark test and the tampering test both reject white-box attacks on the public detector. Rephrasing is done at temperature of 1 in each case, and we also include cross-tokenizer rephrasing. White-box rephrasing remains less successful than its no-box counterpart on both schemes, while white-box word edits are more successful at the smaller edit budgets.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: (Top) Null calibration of the tampering tests on two sets of 10k untampered texts, both generated by Qwen2.5-7B. One is unwatermarked and the other is watermarked but untampered, covering both tampering nulls. (Bottom) Null calibration of the watermark tests for both detectors. We use two sets of 10k unwatermarked texts, one generated from Qwen2.5-7B, and the other being the human-written answers in ELI5.
| Watermark test | Tampering test | Either stage | ||
Table 3: Flag rates of the two-stage pipeline on unwatermarked generations, in percent. The second stage is the removal tail whenever the first stage returns not watermarked , and the forgery tail otherwise. Each stage holds its nominal level , but unwatermarked text can fail either of them and so is flagged at roughly .
| Detection | Tampering | ||||
| Permutations | – | ||||
| Resolvable | analytic | ||||
| Test duration (ms) | 43.9 | 23.3 | 276.9 | 1105.1 | 5518.5 |
Table 4: Average CPU cost of watermark detection and the tampering test, on the same 1k texts. Both tests share one fused -key scoring pass (the ms detection cost). The tampering test adds a permutation over replicates of the already-computed score matrix, so the tampering columns report the permutation alone. Larger lowers the smallest resolvable -value but scales the cost linearly. We use by default.
| Attack | Binomial | Permutation |
| Untampered (FPR) | 0.3 | 0.1 |
| Rephrase removal | 86.9 | 82.4 |
| Rephrase removal (additive) | 40.1 | 25.1 |
| Word-edit removal | 0.6 | 1.2 |
| Rephrase forgery | 12.1 | 25.2 |
| Word-edit forgery | 3.2 | 9.5 |
Table 5: Attack detection rate (%) at for the two exact tests, symmetric , white-box attacks, documents per row. Bold marks the better test per attack. The binomial is more powerful on rephrasing, the permutation on the concentrated attacks.
Figure 8: The scores (standardized using their mean and variance) of each channel and the areas where each test fails to reject its null hypothesis (e.g. means that the public -value is high enough for the text to be labeled as unwatermarked). The attacked data points are stratified samples at different strength levels from our main experiments on Qwen2.5-7B (additive rephrasing with a temperature sweep, and word edits with a replacement rate sweep). The public detector can capture genuinely watermarked text, but is far less sensitive to tampered text than the fused detector and the tampering test (diagonal band, signifying large score deviations). White-box attacks tend to move the scores outside of the tampering test band, into either the removal or the forgery region, whereas the no-box removal attacks move the scores only within the balanced score band. The white-box removal attacks also have similar private scores to their no-box counterparts, but are denser towards lower public score areas.
Figure 9: Watermark removal examples with white-box attacks, either rephrasing or word edits. While the public score may be lowered enough to indicate unwatermarked text, the fused watermark test along with the tampering test can discover that the text is actually either watermarked or tampered, indicating key correlation with the provider’s LLM.
Figure 10: Watermark forgery examples with white-box attacks, either rephrasing or word edits. While the public score may be increased enough to forge a watermark signal, the full detector is harder to deceive, while the tampering test may explicitly indicate forgery.
Figure 11: Additive rephrasing with the same model and dataset at , but with a strength sweep . The strength is similarly effective to temperature at manipulating the public score, and the fused score alongside it. However, extreme strengths also cause large channel deviations for forgery, which are detected by the tampering test.
Figure 12: Removal watermark score per channel over quality degradation with the rephrasing attack (top) and word edits (bottom). The settings are the same as Fig. 5 , namely temperature and edit fractions . The vertical axes are the same across all panels, but the horizontal axes show the different distortion measurements. Rephrasing causes stronger distortion on pure word overlap compared to word edits (which cause their nominal edit rate), but weaker distortion on fluency as quantified with the perplexity ratio, and similar distortions to the LM metrics (BERT and MiniLM) that capture both.
Figure 13: Forgery watermark score over quality degradation with rephrasing (top) and word edits (bottom).
| Median | Attacker | Defender | Distortion | |||||
| Attack | Public | Private | Fused | ASR pub | ASR fus | ADR | ASR net | BERT-F1 |
| Watermarked | 6.8 | 7.0 | 12.9 | 9.0% | 1.7% | 0.0% | 1.7% | 0.000 |
| Rephrase (matched, no-box, add, T=0.5) | 1.5 | 1.7 | 2.5 | 79.1% | 58.8% | 0.2% | 58.7% | 0.039 |
| Rephrase (matched, no-box, add, T=0.8) | 1.2 | 1.3 | 1.8 | 89.2% | 71.3% | 0.3% | 71.1% | 0.045 |
| Rephrase (matched, no-box, add, T=1) | 0.9 | 1.0 | 1.5 | 92.3% | 82.0% | 0.4% | 81.7% | 0.052 |
| Rephrase (matched, no-box, add, T=1.2) | 0.8 | 0.8 | 1.1 | 97.5% | 93.4% | 0.1% | 93.3% | 0.060 |
Table 6: Per-attack removal results behind Fig. 4 , over 1k attacked texts. Success rates are lower is better for the defender, and and BERT-F1 are higher is better. Light blue marks the representative operating points reported in the main text (Fig. 4 ).
| Median | Attacker | Defender | Distortion | |||||
| Attack | Public | Private | Fused | ASR pub | ASR fus | ADR | ASR net | BERT-F1 |
| Unwatermarked | 0.3 | 0.3 | 0.3 | 0.1% | 0.0% | 0.0% | 0.0% | 0.000 |
| Rephrase (matched, white-box, add, T=0.5) | 0.9 | 0.3 | 0.7 | 3.2% | 1.0% | 0.0% | 1.0% | 0.088 |
| Rephrase (matched, white-box, add, T=0.8) | 1.5 | 0.3 | 1.1 | 9.5% | 3.8% | 0.0% | 3.8% | 0.092 |
| Rephrase (matched, white-box, add, T=1) | 2.0 | 0.3 | 1.3 | 20.8% | 7.5% | 10.7% | 6.7% | 0.096 |
| Rephrase (matched, white-box, add, T=1.2) | 2.6 | 0.3 | 1.6 | 38.2% | 12.6% | 15.9% | 10.6% | 0.102 |
Table 7: Per-attack forgery results behind Fig. 4 , over 1k attacked texts. Columns follow Tab. 6 , and light blue again marks the representative operating points from the main text (Fig. 4 ).
| Attack | Generator | ADR | Distortion | ||
| Rephrase, no-box, T | Qwen2.5-7B | 82.0% | 0.4% | 81.7% | 0.052 |
| Gemma-3-4B | 93.7% | 0.2% | 93.5% | 0.107 | |
| SmolLM2-1.7B | 91.7% | 0.0% | 91.7% | 0.097 | |
| Rephrase, white-box, T | Qwen2.5-7B | 100% | 36.6% | 63.4% | 0.066 |
| Gemma-3-4B | 98.8% | 12.2% | 86.7% | 0.115 | |
| SmolLM2-1.7B | 100% | 50.3% | 49.7% | 0.116 |
Table 8: Other generators compared to Qwen2.5-7B, at on 1k texts. Gemma-3-4B generations have approximately the same watermark signal as Qwen per 100 tokens, so its rows are directly comparable. SmolLM2-1.7B is not, and carries a stronger watermark. The two-stage defense behaves the same way on all three.
| On the first tokens | ||||||||
| Median | TPR | |||||||
| Generator | Answer tokens | |||||||
| Qwen2.5-7B | 2.0 | 2.0 | 3.3 | 24.6% | 24.5% | 57.7% | ||
| Gemma-3-4B | 1.4 | 1.5 | 2.2 | 10.3% | 12.3% | 30.2% | ||
| Gemma-3-4B | 0.9 | 0.9 | 1.3 | 1.6% | 2.8% | 6.4% | ||
| SmolLM2-1.7B | 4.6 | 4.5 | 8.3 | 75.7% | 75.4% | 96.3% | ||
Table 9: Watermark strength before any attack, at over 1k texts. Both the median and the TPR are measured on the first tokens of each generation, so that generators of different verbosity are compared at equal length. Texts shorter than tokens are excluded from those columns. is the sampling temperature used to generate, and “Answer tokens” the length of the full generation (mean std) in the generator’s answers.
| Attacker | Proxy | Token overlap | ADR | Distortion | ||
| Uninformed | Matched (Qwen2.5-3B) | 100% | 82.0% | 0.4% | 81.7% | 0.052 |
| Cross (SmolLM3-3B) | 99.3% | 98.5% | 0.0% | 98.5% | 0.137 | |
| Cross (Gemma-3-4B) | 90.3% | 99.1% | 0.0% | 99.1% | 0.079 | |
| Informed | Matched (Qwen2.5-3B) | 100% | 100% | 36.6% | 63.4% | 0.066 |
| Cross (SmolLM3-3B) | 99.3% | 99.9% | 65.8% | 34.2% | 0.143 | |
| Cross (Gemma-3-4B) | 90.3% | 100% | 4.9% | 95.1% | 0.081 |
Table 10: Matched against cross-tokenizer rephrasing on the Qwen2.5-7B generator, removal at T on 1k texts. “Token overlap” is the fraction of the generator’s token spans that the proxy reproduces, measured on the generator’s own generations. Both the attack steering and the token score-based tampering detection are applicable only at this overlap.
| Edit budget | 5% | 10% | 20% | 40% | |||||||||
| Metric | No-box | No-box | No-box | No-box | |||||||||
| ASR pub | 19.2% | 16.9% | 17.7% | 35.7% | 31.5% | 31.7% | 72.6% | 64.5% | 62.5% | 99.1% | 93.4% | 94.9% | 91.8% |
| ASR fus | 1.8% | 1.8% | 3.3% | 2.3% | 3.0% | 6.7% | 4.6% | 16.3% | 26.9% | 10.4% | 54.5% | 69.3% | 77.0% |
| ADR | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| ASR net | 1.7% | 1.7% | 3.3% | 2.3% | 3.0% | 6.5% | 4.3% | 15.6% | 25.6% | 10.3% | 53.7% | 68.4% | 75.3% |
| Distortion | 0.008 | 0.008 | 0.010 | 0.015 | 0.016 | 0.019 | 0.022 | 0.029 | 0.032 | 0.022 | 0.042 | 0.052 | 0.055 |
Table 11: Black-box removal via word edits, at over 1k texts, grouped by edit budget. Within a group, is the overshoot factor: after the verdict flips at edits the attacker continues to , capped by the budget, so stops on the crossing and no-box spends the budget without using the verdict. Bold marks the highest . is everywhere: no stopping rule leaves the channel imbalance the tampering test looks for, so the fused score alone is what limits the attacker.
Figure 14: Same sweep as the left panel of Fig. 5 , viewed per text: empirical CDF of the smallest flipping distortion for removal, no-box vs. white-box, rephrasing (top) and word edits (bottom). White-box rephrasing barely improves over no-box at equal distortion, while white-box word edits flip the public detector at low distortion but rarely survive the two-stage test.
Figure 15: Watermark stealing. Using the public detector increases the public ASR but not , which stays no higher than the no-box attack, since supervision reaches only the public channel. The generation parameters are identical to previous experiments, with the exception of using as a context window, since our experiments with achieved at most 3% ASR.
| Median | Removal | Forgery | |||||||||||
| Language | Script | Public | Private | Fused | ASR pub | ASR fus | ADR | ASR net | ASR pub | ASR fus | ADR | ASR net | |
| Higher-resource | |||||||||||||
| Hungarian | Latin | 72 | 17.5 | 18.0 | 34.2 | 98.6% | 19.4% | 42.9% | 11.1% | 86.1% | 52.8% | 47.4% | 27.8% |
| Chinese | Han | 72 | 10.2 | 10.7 | 20.5 | 100% | 98.6% | 94.4% | 5.6% | 100% | 97.2% | 90.0% | 9.7% |
| German | Latin | 72 | 9.9 | 10.7 | 19.2 | 100% | 90.3% | 84.6% | 13.9% | 94.4% | 77.8% | 69.6% | 23.6% |
| Arabic | Arabic | 72 | 9.4 | 10.2 | 18.3 | 100% | 87.5% | 77.8% | 19.4% | 91.7% | 80.6% | 77.6% | 18.1% |
Table 12: CaLMQA by language at . The median columns are measured on unattacked watermarked text, and the attack columns report white-box word edits at , the strongest arm of Sec. 2.3 . is what the attacker sees, the percentage of tampering detections on texts that evade the fused detector, and what evades both stages. We see that both the split watermark construction and the tampering test remain effective in low-resource languages.
Figure 16: Idealised i.i.d. token edits at : the wide public context decays faster, so the gap grows with editing. Not enough, though: net success stays worse than at equal contexts.
| Scored tokens | ||||||
| 296 | 65.3 | 78.8 | 75.2 | 89.5 | 93.8 | 96.2 |
| 592 | 31.7 | 55.4 | 51.5 | 74.6 | 83.4 | 93.2 |
| 1.5k | 2.1 | 16.4 | 10.3 | 37.5 | 50.9 | 84.2 |
| 3.0k | 0.0 | 0.8 | 0.2 | 7.0 | 14.8 | 56.5 |
| 7.4k | 0.0 | 0.0 | 0.0 | 0.0 | 0.1 | 8.5 |
Table 13: (%, lower is better for the provider) against the number of tokens scored, with both tests run over all of it, at . Bold marks the better configuration per column. The tampering test does flag some of the asymmetric runs, with rising to about , but it recovers less than the fused test gives up.