Learning to Watermark Speech Synthesis Against Model-Driven Reconstruction

Weizhi Liu; Yue Li; Hui Tian; Zhaoxia Yin

cs.CROct 3, 2026

Learning to Watermark Speech Synthesis Against Model-Driven Reconstruction

Authors: Weizhi Liu, Yue Li, Hui Tian, Zhaoxia Yin

Organizations: East China Normal University, Shanghai, China · Huaqiao University, Xiamen, China

Abstract

Modern TTS systems increasingly generate synthetic speech at scale for diverse users. This setting calls for content-level provenance that can verify the origin of released speech and attribute it to the requesting user, which generative watermarking can support by embedding multi-bit identifiers directly into synthesized speech. Once released, however, speech may undergo heterogeneous learned transformations during distribution and editing, with reconstruction objectives that can preserve speech utility while affecting watermark recoverability differently. We find that no single watermark carrier remains consistently reliable across reconstruction models, as its survival depends jointly on the embedded structure, reconstruction mechanism, and observation representation. To this end, we propose Thrive, a multi-bit generative speech watermarking framework for modern autoregressive TTS, covering both discrete-token and continuous-representation generation under reconstruction attacks. Specifically, Rise synchronizes watermark injection into intermediate representations with its continued integration into subsequent generation, while Care combines waveform and spectral experts using bit-wise reliability selection. Experiments on both autoregressive paradigms show that Thrive preserves synthesis fidelity, achieves 87.6% average recovery accuracy under reconstruction attacks, and supports source attribution over candidate sets of up to 10,000 identities.

Figures & tables

Figure 1: Comparison of generative speech watermarking paradigms. Existing approaches establish the watermark through a particular generation state, synthesis component, or model parameter, whereas Thrive coordinates watermark participation throughout generation and adapts recovery after model-driven reconstruction.

Figure 2: Controlled spectro-temporal pattern survival under reconstruction attacks. Mel residuals show reconstruction-dependent retention and deformation, while representation-space responses reveal that the same reconstructed pattern can appear differently across observation spaces.

Figure 3: Overview of Thrive . Rise synchronizes watermark embedding across representation and generation by injecting a representation residual while modulating selected generation layers with the same message. Care recovers complementary waveform and spectral predictions and performs calibrated bit-wise selection for robust watermark recovery after reconstruction.

Method (bps)	Objective Fidelity				Perceptual Quality		FAD		Recovery
Method (bps)	STOI $\uparrow$	PESQ $\uparrow$	SSIM $\uparrow$	MCD $\downarrow$	DNSMOS $\uparrow$	NISQA $\uparrow$	VGGish $\downarrow$	CLAP $\downarrow$	ACC $\uparrow$	TPR $\uparrow$
LJSpeech (In-Distribution Dataset)
Post-hoc Watermarking
AudioSeal (16)	0.998 $\pm$ 0.001	4.425 $\pm$ 0.073	0.978 $\pm$ 0.007	0.818 $\pm$ 0.160	3.202 $\pm$ 0.211	3.875 $\pm$ 0.625	0.077	0.004	1.000	1.000
WavMark (32)	1.000 $\pm$ 0.001	4.495 $\pm$ 0.084	0.973 $\pm$ 0.022	1.278 $\pm$ 0.455	3.132 $\pm$ 0.239	3.708 $\pm$ 0.723	0.435	0.013	1.000	1.000
TBWM (100)	0.986 $\pm$ 0.007	3.929 $\pm$ 0.214	0.937 $\pm$ 0.013	1.539 $\pm$ 0.138	3.213 $\pm$ 0.208	3.678 $\pm$ 0.621	0.855	0.019	1.000	1.000

Table 1: Fidelity and watermark recovery performance. Values are mean

\pm

standard deviation except FAD, which is computed at the corpus level. TPR is measured at 1% FPR. Among generative watermarking methods, the best results are highlighted in bold and the second-best are underlined .

\uparrow

and

\downarrow

indicate that higher and lower values are better, respectively.

	Post-hoc Watermarking			Generative Watermarking
	AudioSeal (16)	WavMark (32)	TBWM (100)	Groot (100)	RIWF (16)	HFGw (20)	Ours-Q (16)	Ours-D (16)
Clean	0.999 / 1.000	1.000 / 1.000	1.000 / 1.000	0.995 / 1.000	0.993 / 1.000	0.989 / 1.000	0.999 / 1.000	0.999 / 1.000
Signal Distortion
Gau. Noi.	0.808 / 0.380	0.561 / 0.022	0.725 / 0.986	0.995 / 1.000	0.951 / 0.959	0.926 / 0.987	0.983 / 0.998	0.993 / 1.000
Low-pass	0.978 / 0.995	1.000 / 1.000	0.993 / 1.000	0.985 / 1.000	0.525 / 0.004	0.914 / 0.976	0.999 / 1.000	0.999 / 1.000
Echo	0.969 / 0.987	0.859 / 0.971	0.947 / 1.000	0.986 / 1.000	0.593 / 0.016	0.983 / 1.000	0.958 / 0.973	0.962 / 0.979

Table 2: Provenance verification under common distortions and reconstruction attacks on LJSpeech. Entries are ACC / TPR@1%FPR and Ours-Q/D denote Qwen3-TTS/Dots.tts.

Figure 4: Source attribution across increasing candidate-set sizes. Results are averaged over common distortions in (a) and across reconstruction attacks in (b).

Variant	Quality		Recovery ACC
Variant	PESQ $\uparrow$	NISQA $\uparrow$	Clean $\uparrow$	Codec Avg. $\uparrow$	Vocoder Avg. $\uparrow$
Full Thrive	3.565	2.807	0.999	0.818	0.906
Embedding Ablation
Generation only	3.755	3.313	0.999 (0.000)	0.675 (-0.143)	0.674 (-0.232)
Representation only	4.643	3.477	0.489 (-0.510)	–	–
Static LoRA	4.156	3.451	0.489 (-0.510)	–	–

Table 3: Ablation of Thrive on Qwen3-TTS. Parentheses indicate ACC changes relative to the full model.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Representation Injection Path	Generation Integration Path
Qwen3-TTS	Outputs of post_upsample_1 and post_upsample_2 ; the feature residual is added at each named stage.	Full-decoder LoRA updates on 57 modules: pre_conv.conv (Conv1d), the q/k/v/o attention projections (Linear), and the gate/up/down MLP projections (Linear).
Dots.tts	Outputs of decoder.ups[0] and decoder.ups[1] ; the feature residual is added before the corresponding decoder.resblocks AMP aggregation.	LoRA updates on 48 modules: post_proj (Conv1d), dec_mi_layer.1.lstm (LSTM), decoder.conv_pre (Conv1d), decoder.ups[0] and decoder.ups[1] (ConvTranspose1d), and selected decoder.resblocks convolutions (Conv1d).

Appendix

Table 4: Backbone-specific placement of the RISE paths.

Method (bps)	Objective Fidelity				Perceptual Quality		FAD		Recovery
Method (bps)	STOI $\uparrow$	PESQ $\uparrow$	SSIM $\uparrow$	MCD $\downarrow$	DNSMOS $\uparrow$	NISQA $\uparrow$	VGGish $\downarrow$	CLAP $\downarrow$	ACC $\uparrow$	TPR $\uparrow$
LibriTTS (OOD Dataset)
Post-hoc Watermarking
AudioSeal (16)	0.999 $\pm$ 0.001	4.393 $\pm$ 0.100	0.979 $\pm$ 0.007	0.689 $\pm$ 0.151	3.182 $\pm$ 0.190	4.260 $\pm$ 0.681	0.791	0.009	1.000	1.000
WavMark (32)	1.000 $\pm$ 0.001	4.246 $\pm$ 0.209	0.947 $\pm$ 0.038	1.532 $\pm$ 0.481	3.191 $\pm$ 0.192	4.275 $\pm$ 0.710	1.731	0.027	1.000	1.000
TBWM (100)	0.990 $\pm$ 0.005	3.870 $\pm$ 0.252	0.933 $\pm$ 0.015	1.368 $\pm$ 0.149	3.188 $\pm$ 0.208	4.019 $\pm$ 0.679	1.167	0.045	1.000	1.000

Appendix

Table 5: Fidelity and watermark recovery performance on LibriTTS. Values are mean

\pm

standard deviation except FAD, which is computed at the corpus level. TPR is measured at 1% FPR.

Method (bps)	Objective Fidelity				Perceptual Quality		FAD		Recovery
Method (bps)	STOI $\uparrow$	PESQ $\uparrow$	SSIM $\uparrow$	MCD $\downarrow$	DNSMOS $\uparrow$	NISQA $\uparrow$	VGGish $\downarrow$	CLAP $\downarrow$	ACC $\uparrow$	TPR $\uparrow$
AISHELL-3 (OOD Dataset)
Post-hoc Watermarking
AudioSeal (16)	0.996 $\pm$ 0.005	4.059 $\pm$ 0.275	0.905 $\pm$ 0.048	2.301 $\pm$ 0.907	2.974 $\pm$ 0.359	3.231 $\pm$ 0.787	1.325	0.095	1.000	1.000
WavMark (32)	0.994 $\pm$ 0.011	4.051 $\pm$ 0.381	0.814 $\pm$ 0.112	2.795 $\pm$ 0.936	2.909 $\pm$ 0.444	3.159 $\pm$ 0.792	0.993	0.126	1.000	1.000
TBWM (100)	0.961 $\pm$ 0.020	3.316 $\pm$ 0.487	0.844 $\pm$ 0.086	2.287 $\pm$ 0.502	2.964 $\pm$ 0.318	3.193 $\pm$ 0.772	1.076	0.085	1.000	1.000

Appendix

Table 6: Fidelity and watermark recovery performance on AISHELL-3. Values are mean

\pm

standard deviation except FAD, which is computed at the corpus level. TPR is measured at 1% FPR.

	Post-hoc Watermarking			Generative Watermarking
	AudioSeal (16)	WavMark (32)	TBWM (100)	Groot (100)	RIWF (16)	HFGw (20)	Ours-Q (16)	Ours-D (16)
Clean	1.000 / 1.000	1.000 / 1.000	1.000 / 1.000	0.995 / 1.000	0.938 / 0.926	0.980 / 1.000	0.999 / 1.000	0.999 / 1.000
Signal Distortion
Gau. Noi.	0.853 / 0.576	0.581 / 0.037	0.707 / 0.962	0.997 / 1.000	0.793 / 0.327	0.901 / 0.958	0.973 / 0.992	0.998 / 1.000
Low-pass	0.986 / 0.999	1.000 / 1.000	0.993 / 1.000	0.995 / 1.000	0.748 / 0.191	0.908 / 0.968	0.999 / 1.000	0.999 / 1.000
Echo	0.975 / 0.993	0.885 / 0.992	0.956 / 1.000	0.989 / 1.000	0.823 / 0.441	0.957 / 0.999	0.969 / 0.987	0.912 / 0.841

Appendix

Table 7: Provenance verification under common distortions and reconstruction attacks on LibriTTS. Entries are ACC / TPR@1%FPR and Ours-Q/D denote Qwen3-TTS/Dots.tts.

	Post-hoc Watermarking			Generative Watermarking
	AudioSeal (16)	WavMark (32)	TBWM (100)	Groot (100)	RIWF (16)	HFGw (20)	Ours-Q (16)	Ours-D (16)
Clean	1.000 / 1.000	1.000 / 1.000	1.000 / 1.000	0.954 / 1.000	0.921 / 0.873	0.857 / 0.851	0.999 / 1.000	0.999 / 1.000
Signal Distortion
Gau. Noi.	0.859 / 0.602	0.582 / 0.037	0.670 / 0.830	0.957 / 1.000	0.860 / 0.607	0.814 / 0.690	0.974 / 0.992	0.998 / 1.000
Low-pass	0.975 / 0.993	1.000 / 1.000	0.990 / 1.000	0.962 / 1.000	0.722 / 0.136	0.795 / 0.608	0.999 / 1.000	0.999 / 1.000
Echo	0.952 / 0.961	0.878 / 0.988	0.970 / 1.000	0.920 / 1.000	0.819 / 0.425	0.843 / 0.803	0.964 / 0.981	0.963 / 0.980

Appendix

Table 8: Provenance verification under common distortions and reconstruction attacks on AISHELL-3. Entries are ACC / TPR@1%FPR and Ours-Q/D denote Qwen3-TTS/Dots.tts.

	GN $\rightarrow$ Echo	LP $\rightarrow$ GN	Echo $\rightarrow$ MP3	GN $\rightarrow$ EnCodec	Opus $\rightarrow$ MP3
Qwen3-TTS	0.897 / 0.776	0.972 / 0.991	0.953 / 0.964	0.903 / 0.802	0.997 / 1.000
Dots.tts	0.941 / 0.934	0.984 / 0.998	0.956 / 0.969	0.928 / 0.897	0.999 / 1.000
	Opus $\rightarrow$ EnCodec	HiFiCodec $\rightarrow$ DiffWave	FACodec $\rightarrow$ BigVGAN	VoiceCraft $\rightarrow$ HiFiCodec	Vocos $\rightarrow$ DiffWave
Qwen3-TTS	0.919 / 0.864	0.791 / 0.321	0.748 / 0.193	0.716 / 0.124	0.820 / 0.429
Dots.tts	0.972 / 0.991	0.801 / 0.354	0.743 / 0.179	0.768 / 0.247	0.752 / 0.201

Appendix

Table 9: Robustness of Thrive under compound attacks on LJSpeech. Attacks are applied from left to right, and entries report ACC / TPR@1%FPR.

Figure 5: Qualitative visualization of fidelity on Qwen3-TTS. Each row presents the watermarked waveform, Mel-spectrogram, and residual Mel-spectrogram produced by a different watermarking method.

Figure 6: Qualitative visualization of fidelity on Dots.tts. Each row presents the watermarked waveform, Mel-spectrogram, and residual Mel-spectrogram produced by a different watermarking method.

Explore similar work

GAMW: Generation-Aware Mel-Domain Watermarking for TTS Traceability
Oct 10, 2026Yihang Zhang, Weicai Yan, Qifei Zhang +2Audio Watermarking Robust Watermarking
Zhejiang University
NeuMark-Native: Robust Text-to-Speech-Native Watermarking Through Full Utilization of Neural Audio Codec Latent Space
Oct 4, 2026Annan Wu, Wen-Chin Huang, Tomoki TodaAudio Watermarking Robust Watermarking
Nagoya University, Japan
SSTMark: Robust Training-Free Semantic-Level Speech Watermarking
Jul 20, 2026Kuan-Lin Chu, Jun-Cheng Chen, Chun-Shien LuAudio Watermarking Robust Watermarking
CITI, Academia Sinica · Taiwan, ROC · IIS, Academia Sinica

Oct 10, 2026J/K move · Enter open · S save

GAMW: Generation-Aware Mel-Domain Watermarking for TTS Traceability

Yihang Zhang, Weicai Yan, Qifei Zhang, Yangyang Wu+1

Zhejiang University

Audio Watermarking Robust Watermarking TTS Synthesis

Open paper AskPDF arXiv

As text-to-speech (TTS) systems continue to advance, provenance tracking and security verification of synthesized speech have become increasingly important. Audio watermarking provides active provenance by embedding imperceptible, machine-recoverable information into speech. However, existing post-hoc methods typically modify the waveform after synthesis, leaving the watermark vulnerable to compound perturbations. To address this limitation, we propose GAMW, a watermark injection framework integrated into the TTS generation process. Before waveform synthesis, GAMW embeds a content-adaptive multi-bit watermark residual into the generated representation, and its decoding objective combines focal modulation with dynamic per-bit weighting. We train and evaluate GAMW separately on two TTS backbones. With a 16-bit payload, GAMW improves decoding accuracy over the strongest baseline by 9.2 and 23.5 percentage points under individual and compound perturbations, respectively, while maintaining competitive speech quality. Reliable clean-speech decoding is also retained at higher payloads. Code is available at https://github.com/ZConJugaTe/GAMW.

Abstract

Explore similar work

GAMW: Generation-Aware Mel-Domain Watermarking for TTS Traceability

NeuMark-Native: Robust Text-to-Speech-Native Watermarking Through Full Utilization of Neural Audio Codec Latent Space

SSTMark: Robust Training-Free Semantic-Level Speech Watermarking