Partial manipulation of speech recordings, where only localized segments of an utterance are altered, poses a significant challenge for content integrity verification, as reliable detection and localization of such edits becomes harder as the manipulated proportion decreases. Watermarking offers a proactive defense alternative by embedding auxiliary information prior to distribution; classical hash-based schemes achieve near-perfect detection and localization under ideal conditions, but the original content cannot be recovered once a segment is manipulated. Building on a prior self-embedding audio steganography framework, this work presents an initial exploration of proactive defense performance under ideal conditions, extending the investigation along three axes: frame-level localization, multi-bit least significant bit variants, and evaluation across multiple ultra-low-bitrate neural codec representations. By embedding a compact neural codec representation rather than a cryptographic hash, the framework additionally enables recovery of the manipulated regions, while supporting training-free detection and localization without spoofed examples. Experiments across four controlled manipulation types under ideal channel conditions show that the embedded payload, and hence an approximate reconstruction of the authentic content, is always fully recovered without bit errors. The results also indicate that the choice of neural codec is the dominant factor for detection and localization performance.
Figures & tables
Fig. 1: Overview of the proposed self-embedding audio watermarking framework. At the publisher side, the carrier signal is encoded by a neural codec and embedded into itself via a watermark embedder. A partial manipulation is applied to the watermarked signal. At the receiver side, the embedded bitstream is recovered by a watermark decoder and passed to a codec decoder to reconstruct the authentic waveform. Finally, a mismatch score between the received signal and its reconstruction is used jointly for utterance-level detection and frame-level localization of manipulated segments.
Codec
Bitrate (kbps)
Repetitions R (per second)
LSB-1
LSB-2
LSB-4
SNAC
0.98
15
30
61
SemantiCodec
0.65
22
44
89
TAAE
0.40
18
36
74
TABLE I: Number of repetitions ( R ) for each codec–bit-depth combination, computed for a 1s carrier.
Detection EER (%) ↓
Localization AUC ↑
Replacement
Structural
Replacement
Structural
Codec
Method
Direct
TTS
Del.
Ins. same
Ins. diff.
Mean
Direct
TTS
Del.
Ins. same
Ins. diff.
Mean
SNAC
LSB-1
8.56
10.68
18.57
18.51
19.41
15.15
0.912
0.889
0.662
0.868
0.864
0.839
LSB-2
8.80
10.61
18.17
18.39
19.05
15.00
0.912
0.889
0.662
0.868
0.864
0.839
LSB-4
8.80
10.75
18.50
18.43
19.21
15.14
0.912
0.889
0.662
0.868
0.864
0.839
SemantiCodec
LSB-1
8.40
9.78
17.47
18.13
18.55
14.47
0.911
0.892
0.661
0.868
0.862
0.839
TABLE II: Utterance-level detection EER (%) ↓ and frame-level localization AUC ↑ for replacement and structural manipulations. Del., Ins. same, and Ins. diff. denote deletion, same-speaker insertion, and different-speaker insertion, respectively.
Fig. 2: DTW cost matrices and optimal warping paths for each manipulation type. Left: alignment between the intact watermarked signal x~ and its self-reconstruction R(x~) , showing a near-diagonal path and low cost. Right: alignment between the manipulated watermarked signal x~sp and its self-reconstruction R(x~sp) , with the manipulated region highlighted in red.
Neural audio codecs are challenging transformations for audio watermarking because they re-encode, quantize, and resynthesize speech. This paper investigates continuous latent-space watermarking for codec robustness. Instead of adding a watermark only to the waveform or spectrogram, we embed a 32-bit message into the continuous latent representation of a codec-like speech autoencoder. The pipeline uses a SEANet-style encoder-decoder, a Conformer-based message embedder, RVQ-guided latent decomposition, and a latent-domain detector trained under signal-processing and neural-codec transformations. Rather than proposing a final universal watermarking baseline, we characterize the trade-offs that appear when the watermark carrier is moved before neural decoding. On 48 kHz speech, EnCodec-aware training improves EnCodec-24k bit accuracy from 78.8% to 95.6% and 97.1%, while PESQ decreases from 3.727 to 3.514 and 3.427.
Zi Hu, Houmin Sun, Linxi Li +4
Digital Innovation Research Center, Duke Kunshan University, Kunshan, China · OfSpectrum, Inc., Los Angeles, USA · University of Warwick, Coventry, United Kingdom +1
Existing waveform-domain audio watermarks are robust to many conventional distortions but can degrade substantially under neural codec resynthesis. We investigate whether continuous neural codec latents provide a more suitable embedding space using a restricted formulation built around frozen pretrained EnCodec. To test this, a feedforward embedder maps a multi-bit payload to an additive latent perturbation decoded through the unchanged codec decoder. Compared with AudioSeal and WavMark, our latent watermark formulation degrades more gradually under repeated and low-bitrate EnCodec resynthesis, transfers to unseen DAC, and retains high detection under most waveform distortions. Substantial EnCodec robustness emerges even without codec-resynthesis supervision, indicating that this behavior is inherent to our latent formulation and is further strengthened by codec-aware training. Learned perturbations are also preserved more strongly through codec cycling than equal-norm random controls, with preservation depending more on channel-specific allocation than temporal structure. End-to-end perceptual quality remains close to that of the frozen EnCodec reconstruction, indicating that much of the observed degradation originates from the codec carrier itself. Overall, these results show that continuous neural codec latents provide a promising embedding space for watermarks that remain robust to neural codec resynthesis.
Lovro Brulec, Sahil Karawade, Leonard Kinzinger
Munich Music Labs · Technical University of Munich
Audio watermarking is increasingly important for tracing generated speech. Several audio watermarking methods have been proposed to embed the watermark in various domains, such as waveform, timbre feature, or latent representations, for making the embedded watermark robust against traditional digital signal processing (DSP) attacks. On the other hand, modern neural codecs introduce a different threat from DSP attacks: they resynthesize speech through quantized acoustic representations and can remove the embedded watermark evi- dence that is not aligned with codec-preserved structure. In this paper, we propose NeuMark, a codec-latent audio watermarking framework that embeds watermark evidence into SpeechTok- enizer acoustic tokens to address this resynthesis threat. NeuMark uses cross-attention to inject a 16-bit message across residual vector quantization (RVQ) layers, distributing the watermark over codec-aligned latent structure. Experimental results show that NeuMark substantially improves robustness under neural- codec resynthesis while supporting both watermark detection and message recovery. We also analyze the trade-off between reconstruction-referenced transparency and original-referenced robustness.