AutoMark: Enabling Autoresearch to Discover Better LLM Watermarks

Thibaud Gloaguen; Robin Staab; Martin Vechev

cs.CRSep 29, 2026

AutoMark: Enabling Autoresearch to Discover Better LLM Watermarks

Authors: Thibaud Gloaguen, Robin Staab, Martin Vechev

Organizations: ETH Zurich

Abstract

With LLM watermarking being deployed commercially and now required by regulations, improving its reliability and effectiveness has become crucial. Yet, recent progress in the field of LLM watermarking has increasingly been driven by improving details of existing methods, an effort fundamentally limited by the pace of human researchers. In this work, we enable for the first time the autonomous discovery of new distortion-free state-of-the-art watermarking schemes. To enable this, we (i) establish strict criteria to ensure that watermarks are reliable (e.g., they do not have an unexpectedly high false positive rate), (ii) propose rigorous statistical tests to automatically evaluate whether a watermarking scheme satisfies our criteria, and (iii) design an evaluation suite to rank watermarks along three key dimensions: detectability, quality, and robustness. By running our framework with 3 frontier models (GPT-6 Astra, Opus 5, Gemini-3.8 Flash), we discover over 50 different watermarking schemes, including several that outperform prior works along all key dimensions. We complement this by a manual study of the discovered schemes, distilling the key ideas into smaller components, and individually studying the impact of each component across dimensions (detectability, quality, robustness) to better understand how the proposed schemes operate. Importantly, we find that the agents, on top of improving existing ideas, also discover fundamentally new ideas (e.g., aligning watermark scores with random per-request direction). Overall, our work establishes the first steps of fully autonomous watermarking research, enabling the discovery of more reliable and effective watermarks. Our code is available at https://github.com/eth-sri/automark, and a blogpost to visualize our results at https://www.sri.inf.ethz.ch/blog/automark.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of our work. In an isolated sandbox, an agent iterates on a watermarking scheme. Submissions are first verified and, upon passing, evaluated and added to a leaderboard. We then manually decompose the discovered schemes into 5 components and ablate them for insights. Lastly, we evaluate two selected schemes on a held-out test set and find that they outperform prior methods.

Figure 2: Submission history for each agent along two evaluated axes: robustness to paraphrasing and quality. Each dot corresponds to a submission and the solid lines are the best value so far. The baselines are in italic, and a parenthesis means the baseline fails our verification.

Panel of Figure 3 — Figure 3: Atomic components extracted from the agents’ watermarking schemes. Colors indicate the family ( Fresh race , Private geometry , Skeleton ) where components of the same family are interchangeable, and white components are compatible with every family. Skeleton schemes reuse the fresh-race scores and detectors (green edge). Components marked ∗ come from prior work.

Figure 4: Robustness-quality for TextSeal dual key routing and Gaussian copula . Robustness is the average of all 4 attacks’ TPR@1%FPR.

		Robustness TPR@1 $\uparrow$				Quality deviation (%) $\downarrow$
Scheme	Clean TPR@1 $\uparrow$	Deletion	Substitution	Back-trans.	Paraphrase	$\Delta$ PPL	$\Delta$ SB-2	$\Delta$ SB-3
Llama-3.1-8B
AAR	93 / 89	7 / 16	8 / 22	66 / 17	57 / 42	3.1 / 3.0	8.3 / 6.5	20 / 15
SynthID	95 / 90	12 / 25	10 / 31	84 / 24	71 / 53	3.0 / 1.0	3.5 / 3.3	9.3 / 8.0
TextSeal (high)	95 / 91	15 / 31	12 / 38	78 / 28	66 / 56	2.9 / 4.4	3.8 / 3.6	9.3 / 8.7
ConcordMark (low)	94 / 89	30 / 54	42 / 62	86 / 37	75 / 65	0.8 / 1.8	2.2 / 2.7	5.1 / 5.7

Table 1: Evaluation of selected agent schemes. Each cell reports English / Chinese results. We bold all results that outperform the best baseline, and highlight the best result in each column.

Figure 5: ResidualRace standalone or with Gaussian copula

\rho=0.95

to match its quality with the Gumbel race .

Figure 6: Comparison of detector variants without forward pass (left) and with forward pass (right).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13: Impact of the entropy Prelude .

Figure 14: Quality-robustness trade-offs for the different watermark scores.

Figure 15: Impact of common-pair filtering compared to the unfiltered Gaussian copula at the same correlation

\rho

. Parentheses give the unfiltered reference percentages for

\rho=0.8

\rho=0.9

\rho=1

, respectively.

Figure 16: Impact of correlated Channel variants compared to

L

independent Channels , keeping the channel count and Bonferroni correction fixed. Parentheses give the independent-channel reference percentages in each panel; bars show differences in percentage points.

Figure 19: Impact of logits transformations compared to the Gumbel race baseline.

Figure 20: Comparison of scalar detectors without a model forward pass (left) and with a forward pass (right). Quality is omitted because changing the detector does not change generation.

Figure 21: Impact of the Wheel compared to QuantileRace with the Quantile detector.

Scheme	Clean TPR@1 $\uparrow$	Deletion	Substitution	Back-trans.	Paraphrase	$\Delta$ PPL	$\Delta$ SB-2	$\Delta$ SB-3
		Robustness TPR@1 $\uparrow$				Quality deviation (%) $\downarrow$
ConcordMark-v0	100 / 100	86 / 83	99 / 96	99 / 99	91 / 90	5.7	8.9	23
CathedralMark	100 / 100	100 / 85	100 / 97	100 / 98	99 / 91	9.6	9.2	23
OverlayMark	100 / 100	100 / 84	100 / 97	100 / 99	100 / 92	7.1	9.4	24
MeridianMark	99 / 98	88 / 88	62 / 74	94 / 93	77 / 78	65	23	60

Appendix

Table 2: Evaluation of invalid agent schemes. Each TPR cell reports the result from the prior harness the scheme was designed in / from our current harness ( App. D ), whereas quality metrics only come from our current harness. Verified reports whether the scheme passes our current verification.

Figure 24: Smallest p-value multiplier passing our soundness tests.

Figure 25: Paraphrasing robustness of the submissions of each agent under the three p-value corrections, where each dot corresponds to a submission and the solid lines are the best value so far. The baselines are in italic, and a parenthesis means the baseline fails our soundness given a key test without correction.

Agent	Interface	Version	Reasoning	Submissions
GPT-6 Astra (xhigh)	Codex CLI	0.153.3	xhigh	$14$
GPT-6 Astra (low)	Codex CLI	0.153.3	low	$11$
Opus 5	Claude Code	2.1.231	high	$16$
Gemini-3.8 Flash (high)	Antigravity CLI	1.1.25	high	$11$

Appendix

Table 3: Research agents from Figure 2 .

Scheme	Reference	Ours (1)	Ours (8)
SynthID	$152.3$	$243.5$	$628.2$
AAR	$194.5$	$303.9$	$1{,}413.3$
TextSeal	$186.9$	$298.6$	$1{,}338.7$

Appendix

Table 4: End-to-end generation throughput (tokens/s). Reference implementations use Transformers; our library uses vLLM with one or up to eight prompts per request.

Figure 26: TPR as a function of the significance level

\alpha

, with 1k samples per model and language generated from WildChat prompts. Left is clean detection and right after paraphrasing.

Baselines
			Robustness TPR@1 $\uparrow$				Quality deviation (%) $\downarrow$
#	Scheme	Clean TPR@1 $\uparrow$	Deletion	Substitution	Back-trans.	Paraphrase	$\Delta$ PPL	$\Delta$ SB-2	$\Delta$ SB-3
	TextSeal	100 / 100 / 100	14 / 34 / 34	28 / 54 / 54	89 / 97 / 97	38 / 58 / 58	4.3	2.3	7.3
	AAR	100 / 100 / (100)	4 / 19 / (19)	8 / 30 / (30)	76 / 92 / (92)	19 / 41 / (41)	3.9	3.9	12
	SynthID	100 / 100 / (100)	9 / 30 / (30)	20 / 44 / (44)	88 / 95 / (95)	35 / 56 / (56)	2.5	2.0	7.1
GPT-6 Astra (xhigh)

Appendix

Table 5: Full results of every submission of Figure 2 , in submission order. Each TPR cell reports the guaranteed / empirical / no-correction settings of App. C (quality does not depend on the setting). Parentheses mark values without correction for schemes that then fail our soundness given a key test. Within each group, bold marks the best (unrounded) value of each column and setting.

Fixed-length
		Robustness TPR@1 $\uparrow$				Quality deviation (%) $\downarrow$
Variant	Clean TPR@1 $\uparrow$	Deletion	Substitution	Back-trans.	Paraphrase	$\Delta$ PPL	$\Delta$ SB-2	$\Delta$ SB-3
Core (fixed $k=2$ , Gumbel race , Gamma )	100 / 100	56 / 54	67 / 76	97 / 98	64 / 68	0.1 / 1.1	4.5 / 4.6	16 / 16
$k=0$	100 / 99	92 / 88	70 / 61	93 / 90	67 / 60	30 / 32	12 / 13	29 / 31
$k=1$	99 / 99	57 / 51	51 / 55	89 / 89	50 / 49	2.7 / 2.5	5.5 / 4.1	14 / 11
$k=3$	100 / 100	25 / 25	42 / 60	96 / 97	52 / 54	0.9 / 0.4	3.1 / 2.1	11 / 8.1

Appendix

Table 6: Watermark unit and context seeding ( Sections A.2 and A.3 ). Each cell reports identity grouping / Lexical group .

Prelude
		Robustness TPR@1 $\uparrow$				Quality deviation (%) $\downarrow$
Variant	Clean TPR@1 $\uparrow$	Deletion	Substitution	Back-trans.	Paraphrase	$\Delta$ PPL	$\Delta$ SB-2	$\Delta$ SB-3
Core (fixed $k=2$ , Gumbel race , Gamma )	100 / 100	56 / 54	67 / 76	97 / 98	64 / 68	0.1 / 1.1	4.5 / 4.6	16 / 16
On the core	100 / 100	54 / 52	67 / 74	96 / 97	62 / 63	3.2 / 2.6	2.4 / 2.8	11 / 11
On Ladder $K=4$ + floor	100 / 100	74 / 70	78 / 84	99 / 98	74 / 73	3.2 / 1.0	6.2 / 5.3	18 / 15
Gaussian copula

Appendix

Table 7: Watermark score ( Section A.4 ). Each cell reports identity grouping / Lexical group .

Races with private randomness
		Robustness TPR@1 $\uparrow$				Quality deviation (%) $\downarrow$
Variant	Clean TPR@1 $\uparrow$	Deletion	Substitution	Back-trans.	Paraphrase	$\Delta$ PPL	$\Delta$ SB-2	$\Delta$ SB-3
Core (fixed $k=2$ , Gumbel race , Gamma )	100 / 100	56 / 54	67 / 76	97 / 98	64 / 68	0.1 / 1.1	4.5 / 4.6	16 / 16
BandRace	98 / 98	16 / 16	20 / 29	73 / 75	29 / 29	2.0 / 0.7	0.3 / 0.1	1.7 / 2.5
PoolRace ( $K=16$ )	100 / 100	33 / 27	44 / 50	93 / 92	46 / 52	3.2 / 0.2	0.8 / 1.0	5.3 / 5.6
MomentBridge ( $K=8$ )	88 / 88	3 / 4	6 / 7	37 / 36	8 / 11	1.9 / 1.2	0.5 / 0.8	0.2 / 0.7

Appendix

Table 8: Logits transformation ( Section A.5 ). Each cell reports identity grouping / Lexical group .

Core texts, no forward pass
		Robustness TPR@1 $\uparrow$				Quality deviation (%) $\downarrow$
Variant	Clean TPR@1 $\uparrow$	Deletion	Substitution	Back-trans.	Paraphrase	$\Delta$ PPL	$\Delta$ SB-2	$\Delta$ SB-3
Gamma	100 / 100	56 / 54	67 / 76	97 / 98	64 / 68	0.1 / 1.1	4.5 / 4.6	16 / 16
Gaussian	100 / 100	40 / 40	51 / 64	95 / 96	54 / 57	0.1 / 1.1	4.5 / 4.6	16 / 16
Clustering	100 / 100	57 / 55	67 / 77	97 / 98	64 / 68	0.1 / 1.1	4.5 / 4.6	16 / 16
Excess	100 / 100	57 / 55	68 / 77	97 / 98	64 / 68	0.1 / 1.1	4.5 / 4.6	16 / 16

Appendix

Table 9: Detection ( Section A.6 ), on fixed generated texts: only the detector changes, so the quality is that of the generation. Each cell reports identity grouping / Lexical group .

Text Watermarking

Text watermarking embeds detectable signals in generated text to indicate provenance, identify sources, or carry messages.

See all 43 papers

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization
Jul 11, 2026Huy Phan, Kieu Dang, Ojaswi Dulal +4Data Provenance LLM Watermarking
Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA
LLM Output Detectability and Task Performance Can be Jointly Optimized
May 2, 2026Koshiro Saito, Ryuto Koike, Masahiro Kaneko +1AI-Generated Text Detection LLM Watermarking
Institute of Science Tokyo · MBZUAI · National Institute of Advanced Industrial Science and Technology +1
Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks
Aug 13, 2026Xiaoyan Feng, Yanjun Zhang, He Zhang +2Data Provenance LLM Watermarking
Griffith University

Jul 11, 2026J/K move · Enter open · S save

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

Huy Phan, Kieu Dang, Ojaswi Dulal, Aiham AL Shukairi+3

Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA

Data Provenance LLM Watermarking Robust Watermarking

Open paper AskPDF arXiv

As large language models (LLMs) become widely deployed, their outputs can be copied, transformed, and redistributed at scale without reliable evidence of origin, creating risks for trust, accountability, intellectual property (IP) protection, and high-stakes decision-making. LLM watermarking addresses this problem by embedding detectable signals into text during or after generation. However, existing methods vary in design assumptions, threat models, and evaluation criteria, while deployment choices such as watermark placement, detection authority, and key management affect reliability, security, and scalability. This paper systematizes LLM watermarking as provenance infrastructure for large-scale data ecosystems. We organize existing approaches along four deployment dimensions: insertion point, verification authority, operational state, and transformation threat model, and relate them to the big data requirements of Volume, Velocity, Variety, Veracity, and Value. We further introduce a Big Data Watermarking Readiness framework centered on four deployment workloads: online generation, streaming detection, transformation pipelines, and ecosystem governance. The framework connects these workloads to system-level requirements including throughput, false-positive control, robustness, cross-domain reliability, governance, and downstream utility. Our analysis highlights a gap between benchmark performance and deployment readiness: false positives accumulate at scale, repeated transformations weaken watermark signals, computational overhead can limit online deployment, and centralized verification can create governance bottlenecks. We conclude with an evaluation blueprint and research directions for scalable, trustworthy provenance in big data ecosystems.

Abstract

Text Watermarking

Explore similar work

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

LLM Output Detectability and Task Performance Can be Jointly Optimized

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks