The Feasibility of Topic-Based Watermarking on Academic Peer Reviews

Alexander Nemecek; Yuzhou Jiang; Erman Ayday

cs.CRMay 27, 2025

The Feasibility of Topic-Based Watermarking on Academic Peer Reviews

Authors: Alexander Nemecek, Yuzhou Jiang, Erman Ayday

Organizations: Case Western Reserve University

Abstract

Large language models (LLMs) are increasingly integrated into academic workflows, with many conferences and journals permitting their use for tasks such as language refinement and literature summarization. However, their use in peer review remains prohibited due to concerns around confidentiality breaches, hallucinated content, and inconsistent evaluations. As LLM-generated text becomes more indistinguishable from human writing, there is a growing need for reliable attribution mechanisms to preserve the integrity of the review process. In this work, we evaluate topic-based watermarking (TBW), a semantic-aware technique designed to embed detectable signals into LLM-generated text. We conduct a systematic assessment across multiple LLM configurations, including base, few-shot, and fine-tuned variants, using authentic peer review data from academic conferences. Our results show that TBW maintains review quality relative to non-watermarked outputs, while demonstrating robust detection performance under paraphrasing. These findings highlight the viability of TBW as a minimally intrusive and practical solution for LLM attribution in peer review settings.

Figures & tables

Figure 1: Perplexity distributions across model and watermark configurations. Lower values indicate better fluency. Values above 20 are truncated for clarity.

Model	Scheme	Samples Retained
Base	NW	508
	TBW	991
	KGW	840
	SynthID	538
Few-shot	All schemes	1000
Fine-tuned	All schemes	1000

Table 1: Number of retained generations with perplexity

\leq 20

across model configurations. TBW retains 991/1000 base model samples, significantly outperforming KGW (840/1000) and SynthID (538/1000), demonstrating superior outlier suppression.

		Detection Accurcy
Model	Attacks	TBW	KGW	SynthID
Base	No Attack	0.946	0.971	0.909
	PEGASUS	0.847	0.477	0.135
	DIPPER	0.876	0.754	0.173
Few-shot	No Attack	0.622	0.975	0.959
	PEGASUS	0.580	0.580	0.359

Table 2: Detection accuracy of TBW, KGW, and SynthID across model configurations and paraphrasing attack types. Scores reflect the proportion of correctly identified watermarked samples per condition.

BERT
	Base		Few-shot		Fine-tuned
Metric	NW	TBW	NW	TBW	NW	TBW
Accuracy	0.290	0.321	0.403	0.437	0.400	0.416
Precision	0.353	0.346	0.373	0.366	0.367	0.366
Recall	0.328	0.342	0.379	0.369	0.370	0.367
F1	0.278	0.317	0.360	0.358	0.364	0.366

Table 3: Overall classification performance on original LLM-generated reviews. Metrics are averaged over Accept, Borderline, and Reject classes.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Conference: Year	Number of Reviews
ICLR: 2018	935
ICLR: 2019	1419
ICLR: 2020	2213
ICLR: 2021	2594
ICLR: 2022	2617
ICLR: 2023	3793

Appendix

Table 4: Review counts per conference used in training and evaluation. The total number of unique reviews is 19,163.

Parameter	Value
Adapter type	LoRA
LoRA $r$ / $\alpha$	16/32
LoRA dropout	0.1
Training epochs	3
Batch size (per device)	2
Max sequence length	2048 tokens

Appendix

Table 5: Fine-tuning Hyperparameters

Figure 2: BERTScore F1 distributions across model configurations with and without TBW (

\tau=0.7

). Higher values indicate greater semantic similarity to the ground truth.

Model	Fluency	Coherence	Usefulness
Base	2.80 $\pm$ 1.16	2.47 $\pm$ 0.97	2.07 $\pm$ 1.05
Few-shot	3.70 $\pm$ 0.65	3.47 $\pm$ 0.73	2.87 $\pm$ 1.07
Fine-tuned	3.87 $\pm$ 0.78	3.80 $\pm$ 0.76	3.60 $\pm$ 0.97

Appendix

Table 6: Human evaluation results for TBW-generated reviews across model configurations. Scores are on a 5-point Likert scale (mean

\pm

std. dev.,

n=6

evaluators, 5 samples per configuration).

Figure 3: BERTScore F1 distributions across model configurations with KGW and SynthID. Higher values indicate greater semantic similarity to the human-written reference.

Figure 4: Perplexity distributions across model configurations with and without TBW (

\tau=0.3

). Lower values indicate better fluency. Values above 20 are truncated for clarity.

Model	Scheme	Samples Retained
Base	NW	508
Base	TBW	684
Few-shot	NW	1000
Few-shot	TBW	1000
Fine-tuned	NW	1000
Fine-tuned	TBW	1000

Appendix

Table 7: Number of generations with perplexity

\leq 20

, comparing unwatermarked (NW) and TBW outputs (

\tau=0.3

Figure 5: BERTScore F1 distributions across model configurations with and without TBW (

\tau=0.3

). Higher values indicate greater semantic similarity to the human-written reference.

Figure 6: ROC curves for TBW detection under no attack, PEGASUS, and DIPPER paraphrasing, across all model configurations. The curves demonstrate TBW’s robustness across attack severity and adaptation settings.

Model	Attack	AUC	F1	TPR@1%	TPR@10%
Base	None	0.968	0.955	0.908	0.956
	PEGASUS	0.936	0.893	0.746	0.861
	DIPPER	0.922	0.857	0.669	0.826
Few-shot	None	0.729	0.768	0.626	0.669
	PEGASUS	0.722	0.758	0.609	0.655
	DIPPER	0.765	0.754	0.565	0.659

Appendix

Table 8: Detection performance across model configurations and attack settings. Metrics include ROC-AUC, best F1 score, and true positive rate (TPR) at fixed false positive rates (FPRs) of 1% and 10%.

Parameter	Value
Model types	bert-base-uncased ,
	roberta-large
Number of classes	3 ( reject , borderline , accept )
Max sequence length	512 tokens
Training epochs	5
Batch size (per device)	16

Appendix

Table 9: Classifier Training Hyperparameters

Figure 7: Confusion matrix for the BERT classifier on 1,000 human-written peer reviews.

Figure 8: Confusion matrix for the RoBERTa classifier on 1,000 human-written peer reviews.

Classifier	Model	Accuracy	Precision	Recall	F1
BERT	Base	0.289	0.322	0.322	0.288
	Few-shot	0.387	0.334	0.342	0.333
	Fine-tuned	0.414	0.372	0.366	0.360
RoBERTa	Base	0.438	0.338	0.340	0.332
	Few-shot	0.360	0.339	0.344	0.335
	Fine-tuned	0.398	0.375	0.368	0.361

Appendix

Table 10: Classification performance for topic-based watermarking (TBW) at a lower similarity threshold of

\tau=0.3

. Results are shown across all model configurations (base, few-shot, fine-tuned) and for both BERT and RoBERTa classifiers.

Panel of Figure 9 — Figure 9: Comparison of confusion matrices across different model configurations and watermarking settings. Top row shows BERT results, middle row shows RoBERTa results, with columns representing no watermarking (NW), topic-based watermarking at $\tau=0.7$ , and topic-based watermarking at $\tau=0.3$ .

Panel of Figure 10 — Figure 10: Classification accuracy on paraphrased peer reviews across three watermarking settings: (a) no watermark (NW), (b) topic-based watermarking (TBW) with $\tau=0.7$ , and (c) TBW with $\tau=0.3$ . Results are shown across all model configurations (base, few-shot, fine-tuned) for both BERT and RoBERTa classifiers under PEGASUS and DIPPER paraphrasing attacks.

Classifier	Model	Watermark	PEGASUS	DIPPER
			Shifts	Shifts
BERT	Base	No watermark	58	54
		TBW ( $\tau=0.7$ )	37	23
		TBW ( $\tau=0.3$ )	51	45
	Few-shot	No watermark	24	14
		TBW ( $\tau=0.7$ )	24	24

Appendix

Table 11: Number of review classification shifts under paraphrasing attacks. Each entry reflects the count (out of 100 paraphrased samples) where the predicted class label differs from the original. Results are grouped by classifier, model variant, and watermarking scheme (NW, TBW (

\tau=0.7

), TBW (

\tau=0.3

)), and evaluated separately under PEGASUS and DIPPER paraphrasing models.

Text Watermarking

Text watermarking embeds detectable signals in generated text to indicate provenance, identify sources, or carry messages.

See all 44 papers

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization
Jul 11, 2026Huy Phan, Kieu Dang, Ojaswi Dulal +4Data Provenance LLM Watermarking
Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA
Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks
Aug 13, 2026Xiaoyan Feng, Yanjun Zhang, He Zhang +2Data Provenance LLM Watermarking
Griffith University
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality
Jun 18, 2026Zeyuan Chen, Ziqing Yang, Yihan Ma +2LLM Evaluation LLM Prompting
CISPA Helmholtz Center for Information Security

Jul 11, 2026J/K move · Enter open · S save

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

Huy Phan, Kieu Dang, Ojaswi Dulal, Aiham AL Shukairi+3

Truman State University, Kirksville, MO, USA · State University of New York at Albany, Albany, NY, USA

Data Provenance LLM Watermarking Robust Watermarking

Open paper AskPDF arXiv

As large language models (LLMs) become widely deployed, their outputs can be copied, transformed, and redistributed at scale without reliable evidence of origin, creating risks for trust, accountability, intellectual property (IP) protection, and high-stakes decision-making. LLM watermarking addresses this problem by embedding detectable signals into text during or after generation. However, existing methods vary in design assumptions, threat models, and evaluation criteria, while deployment choices such as watermark placement, detection authority, and key management affect reliability, security, and scalability. This paper systematizes LLM watermarking as provenance infrastructure for large-scale data ecosystems. We organize existing approaches along four deployment dimensions: insertion point, verification authority, operational state, and transformation threat model, and relate them to the big data requirements of Volume, Velocity, Variety, Veracity, and Value. We further introduce a Big Data Watermarking Readiness framework centered on four deployment workloads: online generation, streaming detection, transformation pipelines, and ecosystem governance. The framework connects these workloads to system-level requirements including throughput, false-positive control, robustness, cross-domain reliability, governance, and downstream utility. Our analysis highlights a gap between benchmark performance and deployment readiness: false positives accumulate at scale, repeated transformations weaken watermark signals, computational overhead can limit online deployment, and centralized verification can create governance bottlenecks. We conclude with an evaluation blueprint and research directions for scalable, trustworthy provenance in big data ecosystems.

Abstract

Text Watermarking

Explore similar work

LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks

PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality