Transferable Multi-Bit Watermarking Across Frozen Diffusion Models via Latent Consistency Bridges
Authors: Hong-Hanh Nguyen-Le, Van-Tuan Tran, Thuc D. Nguyen, Nhien-An Le-Khac
Organizations: School of Computer Science, University College Dublin, Ireland · School of Computer Science and Statistics, Trinity College Dublin, Ireland · Department of Knowledge Engineering, University of Science, VNU-HCMC, Vietnam
As generative AI advances, global governance frameworks increasingly mandate verifiable content provenance. However, existing watermarking techniques face a critical policy-to-technology disconnect: sampling-based methods require computationally prohibitive inversion, while fine-tuning approaches are tethered to specific model checkpoints, hindering standardized, cross-model oversight. To bridge this gap, we introduce DiffMark, a plug-and-play multi-bit watermarking framework. DiffMark embeds a persistent, learned perturbation into every denoising step of a frozen diffusion model, accumulating a recoverable signal in the final latent space. To enable efficient training through the frozen network, we utilize Latent Consistency Models (LCMs) as a differentiable training bridge. DiffMark achieves 64-bit extraction in a single 16.4 ms forward pass, which is a 45× speed-up over inversion baselines. By enabling per-image key flexibility and cross-architecture transferability without retraining, DiffMark provides the practical, scalable technical tooling necessary to operationalize user accountability and enforce emerging AI governance mandates.
Table 1: Quantitative comparison on DiffusionDB. “Plug&Play” indicates whether the method operates on a frozen, unmodified diffusion model. Detection accuracy is measured on clean (unattacked) watermarked images. Generation consistency metrics compare watermarked vs. unwatermarked images from the same noise. Best results among plug-and-play methods are bolded ; overall best are underlined.
Figure 2: Bit Error Rate (BER) distributions for fixed-key and random-key generation across methods. Each violin shows the per-image BER over 1,000 images; wider regions indicate higher density.
Figure 3: Cross-model transferability of DiffMark. Bit accuracy on four unseen SD-family models after training exclusively on SD 1.5, with no per-model fine-tuning.
Figure 4: Decode latency comparison (per image, L40S GPU). Latencies are averaged over 100 images; the x-axis is on a log scale.
Figure 5: Clean identification scaling. Top-1 identification accuracy as a function of database size (number of registered users) without any attack.
Attack
Type
StegaStamp
Stable Signature
AquaLoRA
Tree-Ring
RingID
Shallow Diffuse
DiffMark (Ours)
Bright
Distortion Attacks
1.00
1.00
0.96
0.74
1.00
1.00
1.00
Compress
1.00
1.00
0.99
0.74
1.00
1.00
1.00
Contrast
1.00
1.00
0.97
0.74
0.98
1.00
1.00
Erase
1.00
1.00
0.96
0.51
1.00
1.00
1.00
RCrop
0.39
1.00
0.91
0.03
0.01
0.00
0.00
Rotation
0.00
0.65
0.00
0.16
1.00
0.00
0.00
Table 2: Robustness evaluation across 13 attacks on DiffusionDB ([email protected]%FPR).
Figure 6: LCM Step Ablation
Figure 7: Bit Length Ablation.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Comparison of watermarking paradigms for DMs. (a) Sampling-based methods, (b) Fine-tuning-based methods modify model weights for fast multi-bit detection, and (c) DiffMark (Ours) injects a learned perturbation δ at every step of a frozen UNet, enabling fast single-pass multi-bit detection without weight modification.
Module
Component
Parameters
Encoder Eϕ
Bit embeddings ( 2L×de )
8,192
Spatial basis B ( de×h×w )
262,144
Refinement convolutions
∼ 24K
Variational heads ( μ , logσ2 )
∼ 580
Total
295,265
Decoder Dψ
Input conv + 5 downsample blocks
∼ 700K
Appendix
Table 3: Parameter breakdown of the encoder and decoder.
Parameter
Value
Training steps
10,000
Batch size
16
Optimizer
AdamW
Learning rate (encoder / decoder)
5×10−5 / 3×10−4
LR schedule
Warmup (500 steps) + linear decay to 10−6
Gradient clip (encoder / decoder)
5.0 / 1.0
Appendix
Table 4: Training hyperparameters.
Loss
Symbol
Weight
LCM reconstruction
wlcm
1.0
DDIM supervision
wddim
1.0
Magnitude constraint
wmag
5.0 (MSE variant)
KL regularization
β
0.001→0.05 (warmup, 1K steps)
Orthogonality
worth
0.1
PRVL
wprvl
1.5
Appendix
Table 5: Loss weights for main training.
Attack
ID
Mechanism
Strength Range
Rotation
Rotation
Rotate by angle
0∘→45∘
Resized Crop
RCrop
Crop + resize
scale 1.0→0.5
Random Erasing
Erase
Zero-fill random patch
0%→25% area
Brightness
Bright
Enhance brightness
factor 1.0→2.0
Contrast
Contrast
Enhance contrast
factor 1.0→2.0
Gaussian Blur
Blur
Gaussian blur kernel
size 0→20
Appendix
Table 6: Summary of distortion attacks used for robustness evaluation.
Attack
ID
Strength Range
Regeneration attacks
VAE
Regen-VAE
quality 1→7
Diffusion
Regen-Diff
steps 40→200
Rinsing (2 × )
Rinse-2xDiff
steps 20→100
Adversarial attacks
KL-VAE (grey-box)
Adv-KLVAE8
ϵ : 2/255→8/255
Appendix
Table 7: Summary of regeneration and adversarial attacks used for robustness evaluation.
Table 8: Quantitative comparison on DALL-E3 and MS-COCO. Best results among plug-and-play methods are bolded ; overall best are underlined.
Figure 9: Identification accuracy under attacks. Top-1 accuracy vs. database size for representative distortion (solid) and regeneration (dashed) attacks. Most attacks preserve >95% identification at 106 users.
Figure 10: Per-image BER as a function of Hamming distance from the training key s∗ .
Figure 11: Cross-model transferability of DiffMark under attack. [email protected]%FPR is reported for five SD-family models across eight distortion attacks (left) and six regeneration attacks (right). DiffMark is trained exclusively on SD 1.5; no per-model fine-tuning is applied to the four unseen models.
Figure 12: Qualitative cross-model transferability of DiffMark. Each column shares the same prompt; each row corresponds to a different SD-family model. DiffMark is trained exclusively on SD 1.5 and applied to four unseen models (SD 2.1, DreamShaper 8, Realistic Vision 5.1, OpenJourney v4) without any per-model fine-tuning.
Attack
Type
StegaStamp
Stable Signature
AquaLoRA
Tree-Ring
RingID
Shallow Diffuse
DiffMark (Ours)
DALL-E3
Bright
Distortion Attacks
1.00
1.00
0.89
0.64
1.00
1.00
1.00
Compress
1.00
0.98
0.93
0.78
1.00
1.00
1.00
Contrast
1.00
1.00
0.87
0.73
1.00
1.00
1.00
Erase
1.00
1.00
0.88
0.53
1.00
1.00
1.00
RCrop
0.37
1.00
0.82
0.03
0.00
0.00
0.00
Appendix
Table 9: Robustness comparison on DALL-E3 and MS-COCO under 13 attack types ([email protected]%FPR). Best results are bolded .
Figure 13: Qualitative examples of watermarked images generated by DiffMark with their corresponding input prompts. The embedded watermark is visually imperceptible.
Figure 14: Watermark signal analysis. Columns from left to right: watermarked image, clean image (same zT and prompt), difference map ( 10× amplified), and delta heatmap (L2 norm across latent channels). The watermark perturbation is imperceptible in pixel space while maintaining spatially uniform energy in latent space.
Figure 15: Training progression of the watermark signal. Top row: δ heatmaps; bottom row: difference maps. Left to right: early training (pre-PRVL), mid training (post-PRVL), and late training (near convergence). The PRVL regularizer enforces spatial uniformity in δ , while Lfreq pushes pixel-space differences toward high-frequency regions.
Figure 16: Per-channel δ decomposition. Left: overall L2 norm across channels. Right four panels: individual latent channels visualized with a diverging colormap. The watermark signal is distributed across all four channels with distinct spatial patterns.
Figure 17: Frequency domain analysis of the δ signal. (a) Average log power spectrum across all samples and channels, with DC at center. (b) Radial power profile showing suppressed low-frequency energy below the configured radius (red dashed line). (c) Single-example power spectrum for channel 0.
Figure 18: Difference map overlay on watermarked images. Top row: watermarked images. Bottom row: same images with the mean absolute difference rendered as a semi-transparent heatmap. Bright regions indicate stronger watermark perturbation, which concentrates at edges and textures.
Diffusion models generate high-quality images but pose serious risks like copyright violation and disinformation. Watermarking is a key defense for tracing and authenticating AI-generated content. However, existing methods rely on threshold-based detection, which only supports fuzzy matching and cannot recover structured watermark data bit-exactly, making them unsuitable for offline verification or applications requiring lossless metadata (e.g., licensing instructions). To address this problem, in this paper, we propose Gaussian Shannon, a watermarking framework that treats the diffusion process as a noisy communication channel and enables both robust tracing and exact bit recovery. Our method embeds watermarks in the initial Gaussian noise without fine-tuning or quality loss. We identify two types of channel interference, namely local bit flips and global stochastic distortions, and design a cascaded defense combining error-correcting codes and majority voting. This ensures reliable end-to-end transmission of semantic payloads. Experiments across three Stable Diffusion variants and seven perturbation types show that Gaussian Shannon achieves state-of-the-art bit-level accuracy while maintaining a high true positive rate, enabling trustworthy rights attribution in real-world deployment. The source code have been made available at: https://github.com/Rambo-Yi/Gaussian-Shannon
Yi Zhang, Hongbo Huang, Liang-Jie Zhang
Center for AI Services Computing, College of Computer Science and Software Engineering, Shenzhen University
Artificial Intelligence Generated Content (AIGC), particularly video generation with diffusion models, has been advanced rapidly. Invisible watermarking is a key technology for protecting AI-generated videos and tracing harmful content, and thus plays a crucial role in AI safety. Beyond post-processing watermarks which inevitably degrade video quality, recent studies have proposed distortion-free in-generation watermarking for video diffusion models. However, existing in-generation approaches are non-blind: they require maintaining all the message-key pairs and performing template-based matching during extraction, which incurs prohibitive computational costs at scale. Moreover, when applied to modern video diffusion models with causal 3D Variational Autoencoders (VAEs), their robustness against temporal disturbance becomes extremely weak. To overcome these challenges, we propose SIGMark, a Scalable In-Generation watermarking framework with blind extraction for video diffusion. To achieve blind-extraction, we propose to generate watermarked initial noise using a Global set of Frame-wise PseudoRandom Coding keys (GF-PRC), reducing the cost of storing large-scale information while preserving noise distribution and diversity for distortion-free watermarking. To enhance robustness, we further design a Segment Group-Ordering module (SGO) tailored to causal 3D VAEs, ensuring robust watermark inversion during extraction under temporal disturbance. Comprehensive experiments on modern diffusion models show that SIGMark achieves very high bit-accuracy during extraction under both temporal and spatial disturbances with minimal overhead, demonstrating its scalability and robustness. Our project is available at https://jeremyzhao1998.github.io/SIGMark-release/.
Rapid advancements in video diffusion models have enabled the creation of realistic videos, raising concerns about unauthorized use and driving the demand for techniques to protect model ownership. Existing watermarking methods suffer from two key limitations: they overlook temporal consistency due to conventional watermark decoders and degrade the visual quality of the generated videos. To address these issues, we introduce a robust watermarking method for latent video diffusion models named Latent Video Diffusion Watermarking (LVMark). We propose a novel watermark decoder tailored for generated videos by learning the consistency between adjacent frames. It ensures accurate message decoding, even under malicious attacks, by combining the low-frequency components of the three-dimensional wavelet domain with the color features of the video. Additionally, we train a latent decoder to maintain the visual fidelity of the generated video. Watermarks are embedded into layers with minimal impact on visual appearance using an importance-based weight modulation strategy. We optimize both the watermark decoder and the latent decoder of diffusion model, effectively balancing the trade-off between visual quality and bit accuracy. Our experiments show that our method embeds invisible watermarks into video diffusion models, ensuring robust decoding accuracy with 512-bit capacity, even under distortions.
Youngdong Jang, MinHyuk Jang, JaeHyeok Lee +4
Department of Artificial Intelligence, Korea University, Seoul 02841, Republic of Korea · Google DeepMind, Mountain View, CA 94043, USA