OpenStamp: A Watermark for Open-Source Language Models

Miroojin Bakshi; Saksham Rastogi; Danish Pruthi

cs.CLAug 28, 2026

OpenStamp: A Watermark for Open-Source Language Models

Authors: Miroojin Bakshi, Saksham Rastogi, Danish Pruthi

Organizations: Indian Institute of Science · Carnegie Mellon University

Abstract

With the growing prevalence of large language model (LLM) generated content, watermarking is considered a promising approach for attributing text to LLMs and distinguishing it from human-written content. A prominent class of techniques embeds subtle but detectable signals in generated text by modifying token sampling probabilities. However, such methods are unsuitable for open-source models, where users have white-box access and can easily disable watermarking during inference. In this work, we introduce OpenStamp, a watermarking technique that encodes the watermarking logic directly into the model weights by modifying only the final projection, or unembedding, layer. Through experiments across two models, we show that OpenStamp achieves superior detection performance, with minimal degradation in model capabilities compared to prior methods. The implanted watermark is explicitly designed, and empirically confirmed, to be more robust to paraphrasing attacks and harder to scrub off through post-hoc fine-tuning than prior open-source watermarks. To enable developers to watermark their models, we release our code alongside watermarked versions of 4 popular open-source models.

Figures & tables

Figure 1: Overview of OpenStamp , our watermarking method. We add an offset matrix $\Delta W$ to the unembedding layer’s weights $U$ to produce watermark logits that bias token sampling, favoring tokens with higher watermark logit values. The watermark can be detected using a log-likelihood ratio (LLR) based score which measures how much more likely the observed tokens are under the watermarked model compared to the unwatermarked model.

Figure 2: Trade-off between detectability and text quality for various watermarking methods on Llama-2-7B . OpenStamp is on the Pareto frontier and outperforms all open-source compatible methods (shown in green) by a substantial margin. Vanilla KGW performs slightly worse than KGW + LLR, highlighting the effectiveness of the LLR detection method.

Method	Model	ArXiv		BookSum		Wikipedia
Method	Model	TPR	PPL	TPR	PPL	TPR	PPL
KGW †	Llama-2-7B	0.994 $\pm$ 0.005	32.3 $\pm$ 1.8	0.998 $\pm$ 0.002	24.1 $\pm$ 1.4	0.914 $\pm$ 0.036	13.8 $\pm$ 0.6
KGW †	Mistral-7B	0.998 $\pm$ 0.002	30.2 $\pm$ 2.4	0.998 $\pm$ 0.002	24.6 $\pm$ 0.7	0.911 $\pm$ 0.092	13.2 $\pm$ 0.6
KGW + LLR †	Llama-2-7B	0.994 $\pm$ 0.010	32.0 $\pm$ 1.1	1.000 $\pm$ 0.001	23.2 $\pm$ 0.4	0.996 $\pm$ 0.004	12.7 $\pm$ 0.4
KGW + LLR †	Mistral-7B	1.000 $\pm$ 0.000	29.2 $\pm$ 1.2	1.000 $\pm$ 0.001	23.6 $\pm$ 0.5	0.992 $\pm$ 0.009	13.0 $\pm$ 1.0
KGW Distilled	Llama-2-7B	0.886 $\pm$ 0.076	43.48 $\pm$ 0.7	0.927 $\pm$ 0.035	28.6 $\pm$ 2.4	0.813 $\pm$ 0.141	16.4 $\pm$ 1.7

Table 1: Comparing different watermarking approaches across multiple datasets. We report [email protected]%FPR (labeled TPR) to measure detectability and perplexity (PPL) to characterize text quality. Bold values indicate the best [email protected]%FPR at the lowest PPL for each dataset and model. KGW † and KGW + LLR † are decoding-based watermark methods included for reference. For both models, OpenStamp achieves near-perfect detection with PPL competitive with Unremovable and lower than GaussMark and KGW Distilled, demonstrating a superior tradeoff between text quality and detectability across the three datasets.

	Llama-2-7B		Mistral-7B
Method	AUROC	TPR@1%FPR	AUROC	TPR@1%FPR
SIR	0.973 $\pm$ 0.007	0.753 $\pm$ 0.035	0.971 $\pm$ 0.007	0.763 $\pm$ 0.043
KGW Distilled	0.950 $\pm$ 0.004	0.602 $\pm$ 0.021	0.954 $\pm$ 0.013	0.596 $\pm$ 0.051
GaussMark	0.946 $\pm$ 0.002	0.550 $\pm$ 0.035	0.925 $\pm$ 0.016	0.409 $\pm$ 0.061
Unremovable	0.966 $\pm$ 0.009	0.724 $\pm$ 0.098	0.966 $\pm$ 0.008	0.715 $\pm$ 0.010
OpenStamp w/o sem-align	0.971 $\pm$ 0.005	0.762 $\pm$ 0.024	0.960 $\pm$ 0.012	0.732 $\pm$ 0.038

Table 2: Detectability of paraphrased text. On both models, OpenStamp achieves the highest TPR@1%FPR on paraphrased text, outperforming SIR and the open-source baselines. OpenStamp w/o sem-align degrades performance, confirming that the projection matrix contributes to robustness against paraphrasing attacks.

	RealNewsLike		AlpacaEval
Method	AUROC	TPR@1%FPR	AUROC	TPR@1%FPR
KGW Distilled	0.996 $\pm$ 0.002	0.952 $\pm$ 0.006	0.634 $\pm$ 0.206	0.090 $\pm$ 0.135
GaussMark	0.980 $\pm$ 0.003	0.775 $\pm$ 0.014	0.885 $\pm$ 0.027	0.301 $\pm$ 0.092
Unremovable	0.996 $\pm$ 0.001	0.965 $\pm$ 0.010	0.984 $\pm$ 0.005	0.707 $\pm$ 0.101
OpenStamp	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.977 $\pm$ 0.005	0.699 $\pm$ 0.038

Table 3: Llama-2-7B detectability under instruction fine-tuning. TPR@1%FPR remains high on RealNewsLike but drops on AlpacaEval. Bold values mark the best in each column.

Method	Model	Unquantized	NF4	INT8
OpenStamp	Llama-2-7B	0.998	1.000	1.000
OpenStamp	Mistral-7B	1.000	1.000	1.000
KGW Distilled	Llama-2-7B	0.986	0.976	0.988
KGW Distilled	Mistral-7B	0.990	1.000	0.994
GaussMark	Llama-2-7B	0.846	0.854	0.842
GaussMark	Mistral-7B	0.746	0.736	0.704

Table 4: Detectability under quantization. TPR@1%FPR remains high for OpenStamp and the baselines under both NF4 and INT8.

Condition	PPL
Watermarked baseline	15.1
After 2,500 steps (80M tokens)	78,529.4
After 25,000 steps (800M tokens)	351.7

Table 5: Perplexity after retraining a reinitialized unembedding layer. Only the unembedding layer is reinitialized and retrained on FineWeb; all other weights remain frozen.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	RealNewsLike		AlpacaEval
Method	AUROC	TPR@1%FPR	AUROC	TPR@1%FPR
KGW Distilled	0.997 $\pm$ 0.002	0.936 $\pm$ 0.049	0.597 $\pm$ 0.178	0.066 $\pm$ 0.087
GaussMark	0.963 $\pm$ 0.007	0.595 $\pm$ 0.064	0.853 $\pm$ 0.044	0.223 $\pm$ 0.128
Unremovable	0.996 $\pm$ 0.001	0.939 $\pm$ 0.015	0.979 $\pm$ 0.003	0.589 $\pm$ 0.133
OpenStamp	1.000 $\pm$ 0.000	0.998 $\pm$ 0.002	0.948 $\pm$ 0.020	0.454 $\pm$ 0.009

Appendix

Table 6: Mistral-7B detectability under instruction fine-tuning. TPR@1%FPR remains high on RealNewsLike but drops on AlpacaEval, matching Table 3 . Bold values mark the best in each column.

Model	TPR@1%FPR	[email protected]%FPR	WM PPL	Baseline PPL
Olmo-3-1025-7B	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	19.329 $\pm$ 0.377	16.801
Qwen2.5-7B	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	15.869 $\pm$ 0.355	13.097
SmolLM2-1.7B	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	29.929 $\pm$ 2.310	20.275
phi-4	1.000 $\pm$ 0.000	0.999 $\pm$ 0.002	7.128 $\pm$ 0.158	6.157

Appendix

Table 7: Generalizability across LLMs. OpenStamp achieves perfect TPR@1%FPR across all four models, demonstrating its robustness and generalizability. The difference between WM PPL and Baseline PPL indicates the impact of watermarking on the model’s perplexity.

Method	Model	OpenWebMath		FineWeb-2 (Ja)		StarCoderData
Method	Model	TPR	PPL	TPR	PPL	TPR	PPL
KGW †	Llama-2-7B	0.748 $\pm$ 0.269	16.3 $\pm$ 1.0	0.678 $\pm$ 0.470	18.6 $\pm$ 0.7	0.491 $\pm$ 0.261	6.5 $\pm$ 0.3
KGW †	Mistral-7B	0.596 $\pm$ 0.496	15.1 $\pm$ 0.2	0.980 $\pm$ 0.015	27.3 $\pm$ 1.0	0.414 $\pm$ 0.180	7.1 $\pm$ 0.7
KGW + LLR †	Llama-2-7B	0.998 $\pm$ 0.002	15.4 $\pm$ 0.5	0.998 $\pm$ 0.001	17.7 $\pm$ 1.3	0.915 $\pm$ 0.006	6.6 $\pm$ 0.3
KGW + LLR †	Mistral-7B	0.988 $\pm$ 0.005	14.4 $\pm$ 0.7	1.000 $\pm$ 0.000	27.4 $\pm$ 0.7	0.868 $\pm$ 0.075	6.4 $\pm$ 0.2
KGW Distilled	Llama-2-7B	0.247 $\pm$ 0.329	20.5 $\pm$ 1.1	0.247 $\pm$ 0.222	25.5 $\pm$ 0.0	0.133 $\pm$ 0.100	8.3 $\pm$ 0.3

Appendix

Table 8: Detection under significant distribution shift. We report [email protected]%FPR (labeled TPR) and perplexity (PPL) on OpenWebMath ( Paster et al., 2024 ) , the Japanese subset of FineWeb-2 ( Penedo et al., 2025 ) , and the Python subset of StarCoderData ( Li et al., 2023a ) . Bold values indicate the highest TPR among open-weight watermarking methods for each dataset and model.

Figure 8: Effectiveness of Projection Matrix

P

. The distribution of cosine similarities between original and paraphrased text pairs is shown for both raw hidden states and projected hidden states. The projection significantly increases the similarity, indicating that

P

successfully aligns hidden states with semantic similarity.

Embedding Model	AUROC	TPR@1%FPR
OpenStamp w/o sem-align	0.971 $\pm$ 0.005	0.762 $\pm$ 0.024
Qwen3-Embedding-8B	0.990 $\pm$ 0.003	0.907 $\pm$ 0.003
multilingual-e5-large	0.988 $\pm$ 0.003	0.877 $\pm$ 0.018
bge-m3	0.989 $\pm$ 0.005	0.861 $\pm$ 0.055

Appendix

Table 9: Effect of the embedding model used to train the projection matrix

P

. Paraphrasing robustness on Llama-2-7B . The OpenStamp w/o sem-align row removes the projection matrix and trains the selector on raw hidden states.

Figure 9: Quality of Paraphrased Completions. The distribution of cosine similarities between original and paraphrased completions shows a high mean similarity of 0.92, indicating that the paraphrasing attack successfully preserves the semantic content of the original completions.

Method	Multinomial		Greedy
Method	TPR@1%FPR	seq-rep-3	TPR@1%FPR	seq-rep-3
RL Watermarking	0.25	0.03	0.99	0.58
OpenStamp	1.0	0.03	1.0	0.03

Appendix

Table 10: Detectability (TPR@1%FPR) and text repetition (mean seq-rep-3) for RL watermarking and OpenStamp under multinomial and greedy decoding. While RL watermarking achieves high detectability with greedy decoding, it causes severe repetition, whereas OpenStamp maintains both strong detectability and low repetition across settings.

Figure 12: Variability in Watermark Logits. As

L

increases, the mean Jaccard similarity between different top-biased token sets decreases, indicating greater variability in watermark logits across different contexts.

Condition	ARI	NMI
Different Dataset	0.44	0.65
Different Seed	0.46	0.67
Different Dataset & Seed	0.37	0.63

Appendix

Table 11: Agreement between cluster assignments from different selector matrices. The ARI and NMI scores indicate moderate agreement, showing that reconstructing the original selector matrix is challenging even with knowledge of

L

Method	TPR@1%FPR	[email protected]%FPR	[email protected]%FPR
Discrete LLR	0.96	0.82	0.79
Binomial Count	0.86	0.51	0.50
LLR	1.00	1.00	1.00

Appendix

Table 12: Comparing LLR to alternative detectors . The discretized-signal variants perform worse than the full LLR detector, indicating that collapsing the mixed green list structure into a single discrete choice loses useful information.

Method	Model	Native	LLR
GaussMark	Llama-2-7B	0.737 $\pm$ 0.089	0.922 $\pm$ 0.017
GaussMark	Mistral-7B	0.349 $\pm$ 0.121	0.566 $\pm$ 0.155
KGW Distilled	Llama-2-7B	0.965 $\pm$ 0.023	1.000 $\pm$ 0.000
KGW Distilled	Mistral-7B	0.996 $\pm$ 0.007	1.000 $\pm$ 0.000
Unremovable	Llama-2-7B	0.966 $\pm$ 0.016	1.000 $\pm$ 0.000
Unremovable	Mistral-7B	0.979 $\pm$ 0.011	1.000 $\pm$ 0.000

Appendix

Table 13: Effect of the LLR detector on baselines Replacing each method’s native detector with the LLR improves [email protected]%FPR on both models, with the largest gain for GaussMark.

Text Watermarking

Text watermarking embeds detectable signals in generated text to indicate provenance, identify sources, or carry messages.

See all 42 papers

ByteMark: A Unified Byte-Level Watermark for Detection Across Language Models
Oct 10, 2026Xiaopu Wang, Runze LiText Watermarking LLM Watermarking
Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks
Aug 13, 2026Xiaoyan Feng, Yanjun Zhang, He Zhang +2Data Provenance LLM Watermarking
Griffith University
Language Model Fingerprinting Requires Rethinking Watermark Teachers
Oct 3, 2026Jeongyeon Hwang, Anshul Nasery, Sewoong Oh +1Language Model Fingerprinting LLM Watermarking
Pohang University of Science and Technology · University of Washington

Oct 10, 2026J/K move · Enter open · S save

ByteMark: A Unified Byte-Level Watermark for Detection Across Language Models

Xiaopu Wang, Runze Li

Text Watermarking LLM Watermarking Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

As watermarking of LLM-generated text enters deployment, users increasingly need to check text whose source is unknown. Existing detectors read the watermark through a specific tokenizer or embedding pipeline, and even if all providers adopted the same watermarking method, differences in these representations would leave their watermarks incompatible, and the text alone does not reveal which representation produced it. We propose ByteMark, a unified watermarking scheme that defines its signal on the UTF-8 bytes of published text. Models with different tokenizers embed this signal under one public specification and key, and a single detector verifies the text from the text and key alone. When the tokenizer or embedding model of a baseline detector is replaced, its detection degrades sharply, whereas ByteMark, which uses neither, retains its detection rate; ByteMark also remains detectable when participating models continue or rewrite the text.

Abstract

Text Watermarking

Explore similar work

ByteMark: A Unified Byte-Level Watermark for Detection Across Language Models

Tracing Provenance and Detecting Tampering with Complementary LLM Watermarks

Language Model Fingerprinting Requires Rethinking Watermark Teachers