Auditing Cross-Lingual Fairness in Language Model Watermarking

Alexander Nemecek; Osama Zafar; Debargha Ganguly; Vikash Singh; Vipin Chaudhary; Erman Ayday

cs.CLAug 20, 2026

Auditing Cross-Lingual Fairness in Language Model Watermarking

Authors: Alexander Nemecek, Osama Zafar, Debargha Ganguly, Vikash Singh, Vipin Chaudhary, Erman Ayday

Organizations: Case Western Reserve University

Abstract

Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements. Multilingual deployment exposes evaluation-design choices that are inconsequential on English but determine conclusions cross-lingually. We propose an evaluation framework with four components: detection thresholds calibrated empirically per deployment context, a threshold-independent companion measurement that distinguishes calibration failures from detection failures, three disjoint quality measurement paradigms (distributional, paired-semantic, and reference-perplexity), and a generalized-entropy decomposition of cross-language disparity over a typological family partition. Applied to six watermarking schemes, three open-weight generators, eleven languages spanning four scripts and eight typological families, and both base and instruction-tuned regimes, the framework reveals failure modes that single-language single-paradigm evaluation cannot surface. Across detection and quality, observed disparity is predominantly between-family on the typological partition, indicating that cross-lingual fairness gaps in watermarking are structural to language properties rather than idiosyncratic to particular languages.

Figures & tables

ISO 639-3	Language	Script	Typological family	Joshi et al. tier
eng	English	Latin	Germanic	5
nld	Dutch	Latin	Germanic	4
fra	French	Latin	Romance	5
spa	Spanish	Latin	Romance	5
por	Portuguese	Latin	Romance	4
tur	Turkish	Latin	Turkic	4

Table 1: The eleven evaluation languages with script, typological family, and Joshi resource tier.

		AUC		TPR at $\tau_{g}$ (global)					TPR at $\tau_{l}$ (per-lang)
Scheme	Gen.	Mean	Min	Mean TPR	Min TPR	DI <.8	GE 2	Btw. %	Mean TPR	$\Delta$
Base regime (FLORES+ continuation prompts)
KGW	Mistral	0.999	0.997	0.992 [0.987, 0.996]	0.978	0	$<\!0.001$	83.6	0.991	$+0.001$
	Gemma	0.997	0.991	0.969 [0.959, 0.979]	0.934	0	$<\!0.001$	72.4	0.974	$-0.005$
	Qwen	0.996	0.987	0.965 [0.939, 0.986]	0.880	0	$<\!0.001$	85.1	0.966	$-0.001$
Unigram	Mistral	0.996	0.981	0.868 [0.802, 0.927]	0.638	3	0.008	61.3	0.971	$-0.103$

Table 2: Detection fairness by scheme, generator, and regime, subset all ,

\alpha=0.01

. Mean TPR shown with 95% bootstrap CIs (

B=1000

resamples of the 11-language vector). Min TPR is the realized minimum. DI <.8 counts languages with

\mathrm{TPR}_{l}<0.8\cdot\max_{l}\mathrm{TPR}_{l}

. GE 2 uses the typological partition (Germanic, Romance, Indic, Semitic, Sinitic, Japonic, Turkic, Austroasiatic); Btw. % is its between-family share.

\Delta=\mathrm{mean\,TPR}(\tau_{g})-\mathrm{mean\,TPR}(\tau_{l})

. Values below

10^{-3}

reported as

<\!0.001

; entries marked are undefined when the per-language mean is zero.

Panel of Figure 1 — Figure 1: Per-language quality preservation under three paradigms, base-FLORES (top) and instruct-AYA (bottom); $n=500$ matched pairs per cell. Left: MAUVE on mean-pooled XLM-R-large embeddings ( $1.0$ identical, $0.0$ fully separable). Center: BERTScore F1, rescaled against per-language FLORES non-pair baselines (negative: paired generations less similar than random non-pairs). Right: PPL preservation under XGLM-7.5B, $\exp(-|\log\mathrm{PPL}_{R}(\mathcal{X}^{+})-\log\mathrm{PPL}_{R}(\mathcal{X}^{-})|)\in(0,1]$ . Rows: (scheme, generator) cells; columns: eleven languages grouped by typological family.

Figure 2: Per-scheme disparity decomposition under three quality paradigms, base-FLORES (top) and instruct-AYA (bottom). Bar height is GE 2 over the per-language preservation vector, averaged across the three generators per scheme. Stacking shows the typological-partition decomposition into between-family (dark) and within-family (light) components; the percentage above is the between-family share. Absolute disparity is orders of magnitude smaller than under MAUVE or PPL but the between-family pattern persists. Y-axes differ across panels.

Scheme	MAUVE $\leftrightarrow$ BS	MAUVE $\leftrightarrow$ PPL	BS $\leftrightarrow$ PPL
KGW	+0.61 (+0.45)	+0.03 ( $-$ 0.80)	+0.03 ( $-$ 0.76)
Unigram	+0.56 (+0.23)	+0.14 ( $-$ 0.39)	+0.20 ( $-$ 0.67)
SynthID	+0.76 (+0.47)	+0.81 (+0.27)	+0.77 (+0.45)
EXPEdit	+0.68 (+0.26)	+0.75 (+0.39)	+0.70 (+0.46)
DIP	+0.46 ( $-$ 0.03)	+0.43 (+0.05)	+0.50 ( $-$ 0.05)
XSIR	+0.42 ( $-$ 0.09)	+0.10 ( $-$ 0.71)	+0.00 ( $-$ 0.76)

Table 3: Cross-paradigm rank agreement: per-cell Spearman

\rho

between per-language preservation vectors, mean across (model, regime) cells with per-cell minimum in parentheses (subset= all ;

n=6

cells per scheme,

n=36

pooled). Pairs use BERTScore F1 (rescaled) and PPL preservation under XGLM-7.5B; BS-raw gives nearly identical rankings (

\rho_{\text{BS-raw}\leftrightarrow\text{BS-rescaled}}=+0.87

). Negative minima identify cells where two paradigms produce opposite per-language orderings.

Figure 3: Joint per-language detection-quality landscape, MAUVE paradigm. Markers: per-language

(\mathrm{TPR}\,@\,\tau_{g},\mathrm{MAUVE})

pairs at

\alpha{=}0.01

. Circles: base-FLORES. Triangles: instruct-AYA. Color: typological family. Annotations

s_{B}

s_{I}

: mean pairwise Euclidean distance within each regime. BERTScore and PPL-preservation companion panels in Appendix E .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	Base regime (FLORES+ continuation prompts)						Instruct regime (AYA native-speaker prompts)
	Mistral		Gemma		Qwen		Mistral		Gemma		Qwen
Lang	NWM%	$\bar{\Delta}$	NWM%	$\bar{\Delta}$	NWM%	$\bar{\Delta}$	NWM%	$\bar{\Delta}$	NWM%	$\bar{\Delta}$	NWM%	$\bar{\Delta}$
eng	99.6	$-0.3$	98.8	$-2.0$	98.4	$-0.6$	92.6	$-0.9$	80.6	$-1.7$	93.2	$-2.0$
fra	99.2	$-1.1$	97.6	$-4.0$	89.2	$-0.2$	91.4	$+0.5$	86.2	$-1.7$	94.8	$-4.9$
spa	100.0	$-1.4$	96.8	$-2.6$	91.2	$+1.1$	95.2	$-1.3$	62.6	$+2.7$	92.0	$-5.0$
por	99.2	$-0.5$	96.8	$-1.0$	98.2	$-0.8$	96.2	$-2.0$	70.6	$+3.0$	96.4	$-4.3$

Appendix

Table 4: Per-cell language adherence on the eleven evaluation languages, base-FLORES (left) and instruct-AYA (right). NWM% is the GlotLID-v3 on-target rate for nowatermark generations;

\bar{\Delta}

is the mean shift in percentage points across the six watermarked schemes. mean rows are unweighted means across the eleven languages.

Regime	Subset	$n$ kept	$n$ skipped	Mean	Median	Min	Max
Base	on_target_only	231	0	1.003	1.000	0.996	1.081
Base	off_target_only	8	223	0.929	0.908	0.859	1.023
Instruct	on_target_only	231	0	1.000	1.000	0.964	1.057
Instruct	off_target_only	26	205	1.025	1.017	0.947	1.119

Appendix

Table 5: Length-distribution similarity between LID subsets and the all-subset baseline. Each row is a (regime, subset) pair; ratio

=

subset-mean length/all-subset mean length, computed per (regime, model, language, watermark) cell and summarized across cells. Cells require

n\geq 100

in both subsets to enter the summary;

n

skipped counts cells that fail this floor (typically off_target on high-adherence cells).

conf_floor	on_target_ratio	$\rho$ vs default	Mean on-target %	Mean $\|\Delta\|$ (pp)	Max $\|\Delta\|$ (pp)
0.3	0.7	0.9347	92.0	2.19	13.60
0.3	0.8	0.9468	90.7	0.91	13.00
0.3	0.9	0.8627	86.3	4.87	31.00
0.5	0.7	0.9910	91.1	1.26	11.80
0.5 †	0.8	1.0000	89.8	0.00	0.00
0.5	0.9	0.9317	85.4	4.43	31.20

Appendix

Table 6: LID categorization robustness: per-cell adherence under a

3\times 3

sweep of the GlotLID per-sentence confidence floor ( conf_floor ) and the per-text on-target sentence fraction ( on_target_ratio ).

\rho

is the Spearman rank-correlation between the per-cell on-target-rate vector under each configuration and the default (

\dagger

). Mean on-target % is the unweighted mean across the 462 cells of the evaluation grid. Mean/Max

|\Delta|

are the mean and maximum absolute per-cell shift in on-target rate (pp) relative to default.

Base regime (FLORES+ continuation prompts)
		Germanic		Romance			Indic	Semitic	Sinitic	Japonic	Turkic	Aus.
Scheme	Gen.	eng	nld	fra	spa	por	hin	arb	zho	jpn	tur	vie
KGW	Mistral	$0.000$	$+0.004$	$+0.004$	$+0.002$	$0.000$	$-0.008$	$0.000$	$-0.006$	$0.000$	$-0.004$	$+0.002$
	Gemma	$+0.006$	$+0.030$	$+0.008$	$+0.016$	$+0.038$	$-0.020$	$+0.044$	$-0.010$	$-0.064$	$0.000$	$+0.010$
	Qwen	$+0.028$	$+0.018$	$-0.002$	$-0.002$	$+0.002$	$-0.008$	$0.000$	$+0.008$	$0.000$	$-0.030$	$0.000$
Unigram	Mistral	$+0.074$	$+0.354$	$+0.218$	$+0.292$	$+0.142$	$-0.062$	$+0.004$	$+0.064$	$-0.018$	$+0.074$	$-0.012$

Appendix

Table 7: Per-language calibration gap

\Delta_{l}=\mathrm{TPR}_{l}(\tau_{l})-\mathrm{TPR}_{l}(\tau_{g})

, base-FLORES (top) and instruct-AYA (bottom), subset all ,

\alpha=0.01

. Positive: per-language calibration recovers detection. Negative: per-language calibration worsens detection. Zero: detection insensitive to calibration locality. Row means equal

-1\times

the

\Delta

column of Table 2 .

Panel of Figure 4 — Figure 4: Per-language detection diagnostics, subset all , $\alpha=0.01$ . Within each panel, left: TPR at the global empirical-FPR threshold $\tau_{g}$ ; right: threshold-free AUC. Rows are (scheme, generator) cells; columns are the eleven evaluation languages, grouped by typological family.

		TPR at $\tau_{g}$ (global)					TPR at $\tau_{l}$ (per-lang)
Scheme	Gen.	Mean TPR	Min TPR	DI <.8	GE 2	Btw. %	Mean TPR	$\Delta$
Base regime (FLORES+ continuation prompts)
KGW	Mistral	0.997	0.992	0	$<\!0.001$	61.1	0.997	$+0.000$
	Gemma	0.991	0.980	0	$<\!0.001$	93.2	0.993	$-0.002$
	Qwen	0.987	0.952	0	$<\!0.001$	57.8	0.986	$+0.000$
Unigram	Mistral	0.977	0.920	0	$<\!0.001$	54.9	0.988	$-0.011$

Appendix

Table 8: Detection fairness summary at

\alpha=0.05

, base-FLORES (top) and instruct-AYA (bottom), subset all . Format mirrors Table 2 ; AUC columns are omitted (threshold-independent, identical to Table 2 ) and bootstrap CIs on Mean TPR are omitted for compactness.

L=11

throughout. Values below

10^{-3}

reported as

<\!0.001

		Base regime (FLORES+ continuation)				Instruct regime (AYA native-speaker)
Scheme	Gen.	Mean TPR	Min TPR	DI <.8	$\Delta$ vs. all	Mean TPR	Min TPR	DI <.8	$\Delta$ vs. all
KGW	Mistral	0.995	0.983	0	$+0.003$	0.681	0.425	6	$+0.055$
	Gemma	0.973	0.939	0	$+0.004$	0.426	0.264	10	$+0.037$
	Qwen	0.987	0.963	0	$+0.022$	0.697	0.520	8	$+0.008$
Unigram	Mistral	0.885	0.654	2	$+0.017$	0.388	0.011	7	$+0.077$
	Gemma	0.987	0.974	0	$+0.987$	0.020	0.000	9	$-0.017$

Appendix

Table 9: Detection summary on the on-target subset,

\alpha=0.01

, global threshold

\tau_{g}

, base-FLORES (left) and instruct-AYA (right). Thresholds re-calibrated on each cell’s on-target unwatermarked pool;

L=11

throughout. The off-target subset is omitted: in the base regime no cell meets the

n_{\text{neg}}\geq 200

inclusion floor, and in the instruct regime only six (Mistral-generator) cells pass the relaxed

\alpha=0.05

floor with

L\leq 3

, insufficient for the fairness aggregates this appendix reports. The

\Delta

vs. all columns report change in Mean TPR relative to subset all (Table 2 ).

Scheme	Gen.	Min TPR	Bot.-q mean	Bot.-q langs	Min TPR	Bot.-q mean	Bot.-q langs
		Base regime (FLORES+)			Instruct regime (AYA)
KGW	Mistral	0.978	0.979	zho, hin	0.386	0.447	fra, por
	Gemma	0.934	0.943	arb, por	0.222	0.251	por, vie
	Qwen	0.880	0.894	tur, nld	0.510	0.564	por, nld
Unigram	Mistral	0.638	0.672	nld, spa	0.022	0.029	spa, fra
	Gemma	0.000	0.000	eng, fra	0.010	0.010	por, tur

Appendix

Table 10: Bottom-quintile-mean Rawlsian floor as a stability companion to Min TPR (Table 2 ), base-FLORES (left) and instruct-AYA (right), subset all ,

\alpha=0.01

, global threshold

\tau_{g}

. With

L=11

evaluation languages the bottom quintile is

\lfloor 0.2\cdot 11\rfloor=2

languages; the columns report the mean TPR of the two worst-performing languages per cell, with the languages listed minimum first (arbitrary tie-breaking). The Unigram-Gemma base cell is degenerate (all eleven per-language TPRs are zero); its bottom-quintile language identities are reported but uninformative.

Scheme	Gen.	Total	Btw. %	Total	Btw. %	Total	Btw. %	Total	Btw. %
		Base regime (FLORES+)				Instruct regime (AYA)
		$\mathrm{GE}_{2}$		$\mathrm{GE}_{0}$		$\mathrm{GE}_{2}$		$\mathrm{GE}_{0}$
KGW	Mistral	$<\!0.001$	83.6	$<\!0.001$	83.7	0.026	95.3	0.027	91.4
	Gemma	$<\!0.001$	72.4	$<\!0.001$	72.7	0.053	82.4	0.046	73.3
	Qwen	0.001	85.1	0.001	85.1	0.011	78.6	0.011	72.5
Unigram	Mistral	0.008	61.3	0.009	55.7	0.297	99.8	0.556	94.8

Appendix

Table 11: Robustness of the generalized-entropy decomposition to choice of inequality measure:

\mathrm{GE}_{2}

(half the squared coefficient of variation, more sensitive to inequality at the top of the per-language TPR distribution) versus

\mathrm{GE}_{0}

(mean log deviation, more sensitive to inequality at the bottom). Base-FLORES (left) and instruct-AYA (right), subset all ,

\alpha=0.01

, global threshold

\tau_{g}

; typological partition as in Table 2 . Total values below

10^{-3}

reported as

<\!0.001

. The Unigram-Gemma base

\mathrm{GE}_{0}

entry is obtained by replacing zero TPRs with a small

\epsilon

(per the strict-positivity requirement of

\mathrm{GE}_{0}

); the resulting

\mathrm{Total}=0

and

\mathrm{Btw.\,\%}=100.0

are regularization artifacts, marked † .

		Base regime (FLORES+)			Instruct regime (AYA)
			Btw. %			Btw. %
Scheme	Gen.	$\mathrm{GE}_{2}$	Family	Script	$\mathrm{GE}_{2}$	Family	Script
KGW	Mistral	$<\!0.001$	83.6	46.2	0.026	95.3	32.8
	Gemma	$<\!0.001$	72.4	62.2	0.053	82.4	29.8
	Qwen	0.001	85.1	26.2	0.011	78.6	26.7
Unigram	Mistral	0.008	61.3	28.8	0.297	99.8	46.3

Appendix

Table 12: Robustness of the

\mathrm{GE}_{2}

between-share to choice of partition: typological family (8 groups: Germanic, Romance, Indic, Semitic, Sinitic, Japonic, Turkic, Austroasiatic, with 5 of 11 languages in non-singleton groups) vs. script (4 groups: Latin pooling eng, nld, fra, spa, por, tur, vie; Devanagari

=

hin; Arabic

=

arb; Han pooling zho, jpn; with 9 of 11 languages in non-singleton groups). Base-FLORES (left) and instruct-AYA (right), subset all ,

\alpha=0.01

, global threshold

\tau_{g}

\mathrm{GE}_{2}

Total depends only on the per-language vector and is partition-independent (shown once per regime). Totals below

10^{-3}

reported as

<\!0.001

; for these cells the between-share values involve ratios of near-zero variance components and should not be over-interpreted.

		MAUVE				BERTScore F1 (rescaled)	PPL preservation (XGLM-7.5B)
Scheme	Gen	mean (min)	DI $<$ .8	GE 2	Btw. %	mean (min)	mean (min)	DI $<$ .8	GE 2	Btw. %
Base regime (FLORES+ continuation)
KGW	Mistral	0.947 (0.879) jpn	0	0.001	86.0	-0.041 (-0.079) vie	0.866 (0.793) spa	0	0.002	66.4
KGW	Gemma	0.855 (0.435) arb	1	0.013	98.9	-0.081 (-0.194) vie	0.899 (0.812) fra	0	0.002	91.0
KGW	Qwen	0.889 (0.471) hin	1	0.012	95.6	-0.034 (-0.091) tur	0.834 (0.686) hin	3	0.005	94.0
Unigram	Mistral	0.747 (0.420) arb	5	0.026	49.5	-0.070 (-0.134) arb	0.872 (0.811) spa	0	0.002	96.4

Appendix

Table 13: Quality fairness summary by scheme, generator, regime, and measurement, subset all . Each row reports mean per-language preservation with the per-cell minimum and floor language (ISO 639-3 superscript) in parentheses. DI

<

.8 counts languages with per-language preservation below

0.8\times\max_{l}

. GE 2 uses the typological partition (Germanic, Romance, Indic, Semitic, Sinitic, Japonic, Turkic, Austroasiatic); Btw. % is its between-family share. BERTScore mean (min) is reported on rescaled F1 [ 28 ] ; DI

<

.8 and GE 2 are omitted for BERTScore because neither registers signal on the bounded raw scale and both behave pathologically on the rescaled scale (§ 5.2 ).

Panel of Figure 5 — Figure 5: PPL preservation under mGPT-1.3B cross-reference. Format and color scale match Figure 1 .

Panel of Figure 6 — Figure 6: On-target minus all-subset preservation, base-FLORES regime (top) and instruct-AYA regime (bottom). Each cell is $\Delta=\text{preservation}_{\text{on-target}}-\text{preservation}_{\text{all}}$ for the indicated paradigm; rows are (scheme, generator), columns are languages. Positive (red): on-target restriction raises preservation; negative (blue): on-target restriction lowers it. Color scale fixed at $[-0.3,+0.3]$ ; values outside this range saturate. The EXPEdit-Mistral, Arabic, BERTScore F1 entry in the instruct panel is omitted (—) because the on-target sample size falls below the $n\geq 100$ paired-set floor.

Figure 7: Joint per-language detection–quality landscape, BERTScore-rescaled (top) and PPL-XGLM (bottom) paradigms. Format matches Figure 3 : markers are per-language

(\mathrm{TPR}\,@\,\tau_{g},\mathrm{quality})

pairs at

\alpha{=}0.01

n{=}33

per regime (11 languages

\times

3 generators); circles are base-FLORES, triangles are instruct-AYA; color encodes typological family;

s_{B}

s_{I}

are mean pairwise Euclidean distance within each regime.

Base regime (FLORES+ continuation, $n=33$ )
Covariate	KGW	Unigram	SynthID	EXPEdit	DIP	XSIR
Fertility (tokens/sent.)	0.016	0.052	0.008	0.834 ∗∗	0.238 ∗∗	0.001
NWM surprisal (XGLM)	0.066	0.074	0.058	0.261 ∗∗	0.156 ∗	0.003
Latin script	0.029	0.001	0.005	0.051	0.009	0.019
NWM adherence	0.600 ∗∗	0.006	0.415 ∗∗	0.001	0.461 ∗∗	0.091
Instruct regime (AYA native-speaker, $n=33$ )

Appendix

Table 14: Single-covariate

R^{2}

for per-language detection TPR at

\tau_{g}

\alpha{=}0.01

, subset all. Each fit regresses the per-language outcome vector on a single covariate over 33 (language, generator) points. Stars from the slope

p

-value:

{}^{*}p<0.05

{}^{**}p<0.01

Base regime (FLORES+ continuation, $n=33$ )
Covariate	KGW	Unigram	SynthID	EXPEdit	DIP	XSIR
Fertility (tokens/sent.)	0.392 ∗∗	0.249 ∗∗	0.027	0.005	0.234 ∗∗	0.093
NWM surprisal (XGLM)	0.026	0.023	0.018	0.001	0.007	0.009
Latin script	0.136 ∗	0.036	0.062	0.052	0.083	0.000
NWM adherence	0.031	0.007	0.101	0.074	0.219 ∗∗	0.072
Instruct regime (AYA native-speaker, $n=33$ )

Appendix

Table 15: Single-covariate

R^{2}

for per-language MAUVE preservation. Each fit regresses the per-language outcome vector on a single covariate over 33 (language, generator) points. Stars from the slope

p

-value:

{}^{*}p<0.05

{}^{**}p<0.01

. BERTScore-rescaled and PPL-XGLM paradigm-stability tables in § F.1 .

Base regime ( $n=33$ )
Covariate	KGW	Unigram	SynthID	EXPEdit	DIP	XSIR
Fertility (tokens/sent.)	0.000	0.013	0.024	0.054	0.039	0.095
NWM surprisal (XGLM)	0.036	0.016	0.000	0.032	0.039	0.094
Latin script	0.068	0.207 ∗∗	0.004	0.022	0.003	0.008
NWM adherence	0.239 ∗∗	0.004	0.378 ∗∗	0.259 ∗∗	0.298 ∗∗	0.188 ∗
Instruct regime ( $n=33$ )

Appendix

Table 16: Single-covariate

R^{2}

for BERTScore-rescaled preservation. Stars from the slope

p

-value:

{}^{*}p<0.05

{}^{**}p<0.01

Base regime ( $n=33$ )
Covariate	KGW	Unigram	SynthID	EXPEdit	DIP	XSIR
Fertility (tokens/sent.)	0.191 ∗	0.327 ∗∗	0.013	0.000	0.121 ∗	0.148 ∗
NWM surprisal (XGLM)	0.003	0.179 ∗	0.017	0.018	0.094	0.183 ∗
Latin script	0.022	0.001	0.126 ∗	0.105	0.052	0.056
NWM adherence	0.186 ∗	0.014	0.164 ∗	0.123 ∗	0.109	0.015
Instruct regime ( $n=33$ )

Appendix

Table 17: Single-covariate

R^{2}

for PPL-XGLM preservation. Stars from the slope

p

-value:

{}^{*}p<0.05

{}^{**}p<0.01

LLM Watermarking

Large language model watermarking embeds or detects signals in text and models to trace provenance, ownership, and unauthorized use.

See all 106 papers

Oct 20, 2025

Is Multilingual LLM Watermarking Truly Multilingual? Scaling Robustness to 100+ Languages via Back-Translation

Multilingual watermarking aims to make large language model (LLM) outputs traceable across languages, yet current methods still fall short. Despite claims of cross-lingual robustness, they are evaluated only on high-resource languages. We show that existing multilingual watermarking methods are not truly multilingual: they fail to remain robust under translation attacks in medium- and low-resource languages. We trace this failure to semantic clustering, which fails when the tokenizer vocabulary contains too few full-word tokens for a given language. To address this, we introduce STEAM, a detection method that uses Bayesian optimisation to search among 126 candidate languages for the back-translation that best recovers the watermark strength. It is compatible with any watermarking method, robust across different tokenizers and languages, non-invasive, and easily extendable to new languages. With average gains of +0.23 AUC and +37%p TPR@1%, STEAM provides a scalable approach toward fairer watermarking across the diversity of languages.

Asim Mohamed, Martin Gubri

Parameter Lab · École Polytechnique, IP Paris, CNRS

Multilingual Language Model Evaluation LLM Watermarking Robust Watermarking Low-Resource Language Processing

Ask PDF

May 30, 2026

Linguistics-Aware Non-Distortionary LLM Watermarking

Watermarking should identify language-model output without degrading quality or limiting verification to the model provider. Multilingual deployment makes this harder because morphology, segmentation, and script change where watermark evidence can enter naturally. We introduce LUNA, a linguistically adaptive watermark that combines model-free detection with single-token non-distortion under the standard random-key model. LUNA estimates normalized next-tag entropy from part-of-speech contexts in an external corpus and uses it to set the depth of a non-distortionary binary tournament sampler; the detector reconstructs the same schedule from text, a tokenizer, a tagger, and a secret key. We evaluate six typologically diverse languages and two domains against eight primary baselines. LUNA attains an AUROC of 0.9959 and the lowest mean absolute median perplexity shift of 0.045 across the twelve settings; its 95% bootstrap interval [0.022, 0.073] lies below all baseline intervals. LUNA also records the lowest mean Self-BLEU, Distinct-1, surprisal, and entropy shifts. It is the only method that simultaneously achieves AUROC > 0.99 and an absolute median perplexity shift below 0.1 in a majority of settings, reaching this regime in 9 of the 12 settings while no baseline reaches it in more than 2. Our code is available at: https://github.com/Shinwoo-Park/luna_watermark

Shinwoo Park, Hyejin Park, Hyeseon An +1

Yonsei University, Seoul, Republic of Korea · Rensselaer Polytechnic Institute, Troy, NY, USA

LLM Watermarking

Ask PDF

Apr 15, 2026

Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking

Watermarking is becoming the default mechanism for AI content authentication, with governance policies and frameworks referencing it as infrastructure for content provenance. Yet across text, image, and audio modalities, watermark signal strength, detectability, and robustness depend on statistical properties of the content itself, properties that vary systematically across languages, cultural visual traditions, and demographic groups. We examine how this content dependence creates modality-specific pathways to bias. Reviewing the major watermarking benchmarks across modalities, we find that, with one exception, none report performance across languages, cultural content types, or population groups. To address this, we propose three concrete evaluation dimensions for pluralistic watermark benchmarking: cross-lingual detection parity, culturally diverse content coverage, and demographic disaggregation of detection metrics. We argue that watermarking is part of the pluralistic alignment pipeline and should be held to the same evaluation standards. We connect this to governance frameworks currently mandating watermarking deployment without requiring fairness evaluation. Our position is that evaluation must precede deployment, and that the same bias auditing requirements applied to AI models should extend to the verification layer.

Alexander Nemecek, Osama Zafar, Yuqiao Xu +2

Case Western Reserve University

Watermark Removal Detection AI-Generated Image Watermarking

Ask PDF

Abstract

LLM Watermarking

Explore similar work

Is Multilingual LLM Watermarking Truly Multilingual? Scaling Robustness to 100+ Languages via Back-Translation

Linguistics-Aware Non-Distortionary LLM Watermarking

Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking