Beyond Detection: Multi-Scale Hidden-Code for Natural Image Deepfake Recovery and Factual Retrieval

Yuan-Chih Chen; Chun-Shien Lu

cs.CVFeb 26, 2026

Beyond Detection: Multi-Scale Hidden-Code for Natural Image Deepfake Recovery and Factual Retrieval

Authors: Yuan-Chih Chen, Chun-Shien Lu

Organizations: IIS, Academia Sinica, Taiwan, ROC

Abstract

Recent advances in image authenticity have primarily focused on deepfake detection and localization, leaving recovery of tampered contents for factual retrieval relatively underexplored. We propose a unified hidden-code recovery framework that enables both retrieval and restoration from post-hoc and in-generation watermarking paradigms. Our method encodes semantic and perceptual information into a compact hidden-code representation, refined through multi-scale vector quantization, and enhances contextual reasoning via conditional Transformer modules. To enable systematic evaluation for natural images, we construct ImageNet-S, a benchmark that provides paired image-label factual retrieval tasks. Extensive experiments on ImageNet-S demonstrate that our method exhibits promising retrieval and reconstruction performance while remaining fully compatible with diverse watermarking pipelines. This framework establishes a foundation for general-purpose image recovery beyond detection and localization.

Figures & tables

Figure 1: Overall workflow of the proposed method. In the multi-scale token watermarking stage, our goal is to protect an image $I$ by embedding image content–related information to produce a watermarked image $I_{w}$ . The original image $I$ is first converted into quantized multi-scale tokens $z_{s_{1}},\ldots,z_{s_{k}}$ using a VQ-VAE. These tokens are then flattened and transformed into a bit string $h$ . The bit string $h$ is embedded into $I$ as a content watermark through the watermark injection encoder $E_{w}$ , where the embedding length is constrained by the watermark capacity $|m|$ such that $|h|\leq|m|$ . When a tampered image $I_{d}$ is generated from the watermarked image $I_{w}$ by malicious manipulations (e.g., object removal or inpainting), our method enables deepfake recovery using the content-related information embedded in $I_{w}$ . The tampered image $I_{d}$ is decoded by the watermark decoder $D_{w}$ to extract the hidden quantized multi-scale tokens $h^{\prime}$ and by the localization decoder $D_{loc}$ to produce a deepfake localization map $M_{loc}$ . Finally, the multi-scale recovery transformer reconstructs the tampered image $I_{d}$ based on $h^{\prime}$ , $M_{loc}$ , and the quantized tokens $h_{d}$ of the tampered image.

Panel of Figure 2 — Figure 2: Visualization of the dropout effect during quantizer (VQ-VAE) training. From left to right, each column shows the reconstructed images using token maps $(z_{s_{1}},\ldots,z_{s_{k}})$ with $1\leq k\leq 10$ . Dropout encourages semantic robustness across scales, allowing lower-level representations (smaller $k$ ) to retain more global structure and meaningful content. In contrast, the model trained without dropout produces coarse, less informative reconstructions at lower scales.

Figure 3: Illustration of proposed conditional deepfake recovery Transformer. The predicted hidden codes $h^{\prime}$ are used as conditional tokens to guide the generation process, enabling context-aware reconstruction. The untampered regions $\tilde{z}_{s_{i}}$ of deepfaked image provide additional contextual cues, helping the model produce more detailed and consistent predictions at higher scales.

ImageNet	Same samples				Different samples		CLIP score ( $\uparrow$ )
Methods \ Metric	Top-1 label ( $\uparrow$ )	Top-5 label ( $\uparrow$ )	Top-1 image ( $\uparrow$ )	Top-5 image ( $\uparrow$ )	Top-1 label ( $\uparrow$ )	Top-5 label ( $\uparrow$ )
HiNet [ 11 ]	0.2393	0.3511	0.2017	0.2739	0.0742	0.0716	0.7255
RePaint [ 15 ]	0.2821	0.3868	0.2571	0.3373	0.0717	0.1693	0.7854
VQGAN [ 6 ] (codebook size $V$ = 1024)	0.3633	0.5516	0.2756	0.4243	0.1787	0.3699	0.7887
VQGAN [ 6 ] (codebook size $V$ = 16384)	0.5205	0.6895	0.4246	0.5925	0.2467	0.4535	0.8157
VAR [ 23 ]	0.6754	0.8125	0.5761	0.7400	0.4222	0.6547	0.8817

Table 1: Comparison of retrieval accuracy and CLIP score between our method and related approaches. Deepfake images are generated from the ImageNet-S dataset using Stable Diffusion. “Same samples” indicates that the original image is included in the evaluation dataset, whereas “Different samples” indicates that the original image is excluded, allowing retrieval only within the same class. “Top-

k

label” denotes the retrieval accuracy based on whether any of the top-

k

retrieved images from ImageNet-S share the same label as the original image. “Top-

k

image” denotes the retrieval accuracy based on whether the original image itself appears among the top-

k

retrieved images. Higher CLIP scores indicate better semantic consistency between the recovered and original images.

Figure 4: Qualitative comparison of tampered image recovery performance. The first two columns show the Original Image and its Tampering mask. The third column, Deepfaked Image, shows the tampered image generated using Stable Diffusion based on the mask. While general recovery methods (HINet [ 11 ] , RePaint [ 15 ] ) often fail to maintain semantic consistency or introduce severe distortions. The general quantization methods (VQGAN, VAR) with our watermarking scheme struggle with texture preservation, our method successfully reconstructs the masked region, restoring the identity and realistic appearance of the original subjects (birds and an ostrich).

Method	Type	Hiding Information	Smallest Hiding Unit	Hiding Domain	Bits per Pixel (BPP)	Total Hidden Information
HiNet [ 11 ]	Reversible module	Pixel-level content	$1\times 1$	Latent space	–	$H\times W\times C$
RePaint [ 15 ]	Inpainting model	–	–	–	–	–
Common self-recovery watermarking [ 1 ]	Fragile watermark	MSBs, block averages, or transform-domain coefficients	$2\times 2$ – $6\times 6$	Pixel (LSB)	2–4	$\text{bpp}\times H\times W\times C$
Bouarroudj et al. [ 4 ]	Fragile watermark	DWT coefficients	$4\times 4$	Frequency domain	0.5	$\text{bpp}\times H\times W\times C$
Ours	Robust watermark	Multi-scale quantized vectors	$16\times 16$	Latent space	–	$\|h\|$ ( $<\|m\|$ )

Table 2: Comparison of different types of image recovery and watermarking approaches. Traditional self-recovery watermarking methods embed large amounts of pixel or transform information into fragile regions such as LSBs, leading to high BPP and poor robustness. In contrast, our method encodes compact hidden-code representations through robust, multi-scale quantization in the latent space, achieving efficient and resilient recovery.

Methods	Clean ( $\uparrow$ )	JPEG( $\uparrow$ )	Gaussian Blur( $\uparrow$ )	Gaussian Noise ( $\uparrow$ )
		QF $70\%$	$5\times 5$	$\sigma=1$	$\sigma=5$
Bouarroudj et al. [ 4 ]	1.0000	0.5054	0.5042	0.5851	0.5047
Ours(P)	0.9988	0.9923	0.9983	0.9983	0.9577
Ours(I)	0.9999	0.9951	0.9971	0.9971	0.9951

Table 3: Comparison of bit accuracy between the self-recovery watermarking method [ 4 ] and our proposed approaches (Ours(P) denotes the use of post-hoc watermarking model–EditGuard and Ours(I) denotes the use of in-generation model–Gaussian shading) under various image degradations. Higher values indicate better robustness in bit preservation.

Figure 5: Illustration of single-scale versus multi-scale quantization. (a) Single-scale quantization: each token $x_{i}$ corresponds to a specific spatial position in the image, and the transformer predicts $x_{i}$ sequentially based on all preceding tokens $(x_{1},x_{2},\ldots,x_{i-1})$ . (b) Multi-scale quantization: each token map $z_{s_{i}}$ represents a group of tokens at a particular scale $s_{i}$ . The transformer predicts $z_{s_{i}}$ conditioned on all previous token maps $(z_{s_{1}},z_{s_{2}},\ldots,z_{s_{i-1}})$ .

Panel of Figure 6 — Figure 6: Visualization of the limitations of single-scale tokenization at varying image resolutions. (a) Original input image ( $256{\times}256$ ). (b) Reconstructed images using VQGAN, from left to right, correspond to image sizes of $256{\times}256$ , $128{\times}128$ , $96{\times}96$ , and $64{\times}64$ . Even at $128{\times}128$ , the reconstruction begins to lose fine details. This degradation occurs because downsampling increases the information contained within each patch while the codebook remains fixed, forcing each quantized vector to represent a larger and more complex region. Consequently, the quantized representation becomes insufficient to capture fine-grained details, resulting in visible information loss and reduced visual fidelity.

Methods	SSIM ( $\uparrow$ )	PSNR ( $\uparrow$ )	LPIPS ( $\downarrow$ )
HiNet [ 11 ]	0.7024	18.7884	0.0375
RePaint [ 15 ]	0.7832	18.9624	0.0436
VQGAN [ 6 ] ( $V$ = 1024)	0.7987	22.6042	0.0208
VQGAN [ 6 ] ( $V$ = 16384)	0.8029	23.1354	0.0199
VAR [ 23 ]	0.7993	16.5914	0.0249
hidden code (Ours)	0.8238	24.6575	0.0190

Table 4: Comparison of image quality across different recovery approaches. All deepfake images were generated from the ImageNet-S dataset using Stable Diffusion.

Methods	Gaussian Shading	Guassian Shading + ours
	Forgery BA ( $\uparrow$ )	Forgery BA ( $\downarrow$ )
Average attack ( $n=5$ , grey box setting)	0.8999	0.5142
Average attack ( $n=5$ , black box setting)	0.8000	0.5079
Average attack ( $n=10$ , grey box setting)	0.9126	0.5125
Average attack ( $n=10$ , black box setting)	0.8141	0.5047
Average attack ( $n=20$ , grey box setting)	0.9328	0.5119

Table 5: Comparison of Forgery Attack Success Rate (Bit Accuracy, BA) under Black-box and Grey-box Settings. We evaluate the success of forgery attacks (

\text{BA}\uparrow

) against a baseline method and against our proposed framework (

\text{BA}\downarrow

). The BA quantifies the attacker’s ability to guess the hidden code bit, where

\text{BA}\approx 0.5

signifies a failed attack (random guessing). The Average Attack uses

n

watermarked images to estimate the code. The Grey-box setting assumes the attacker has access to the average clean image, while the Black-box setting uses an average image from a different dataset. Our method consistently forces the attacker’s BA toward 0.5, demonstrating a significantly higher level of security compared to the baseline.

ImageNet	CLIP score( $\uparrow$ )						CLIP score( $\downarrow$ )
Methods \ Metric	Original	Crop	Color jitter	JPEG	Gaussian Blur	Gaussian Noise	Different image
		$50\%$	$30\%$	QF $70\%$	$5\times 5$	$\sigma=5$
VQGAN [ 6 ] (codebook size $V$ = 1024)	0.7939	0.7984	0.7906	0.7730	0.8307	0.7817	0.6787
VQGAN [ 6 ] (codebook size $V$ = 16384)	0.8208	0.8229	0.8061	0.7949	0.8572	0.8040	0.6571
VAR [ 23 ]	0.8845	0.8692	0.8385	0.8418	0.8582	0.8529	0.6552
hidden code (Ours)	0.9021	0.8885	0.8503	0.8765	0.8784	0.8948	0.6176

Table 6: CLIP Similarity Analysis (Semantic Consistency). We report the CLIP score between the reconstructed image (derived from the embedded hidden code) and the image variants to evaluate both robustness (higher scores for image degradations) and forgery resistance (lower scores for unrelated images). CLIP score (

\uparrow

) indicates better robustness, while CLIP score (

\downarrow

) for the “Different image” column indicates better forgery resistance.

Image Restoration

Image restoration uses computational methods to recover useful visual content from noisy, blurred, low-resolution, or otherwise degraded images.

See all 372 papers

Mar 25, 2026

High-Fidelity Face Content Recovery via Tamper-Resilient Versatile Watermarking

The proliferation of AIGC-driven face manipulation and deepfakes poses severe threats to media provenance, integrity, and copyright protection. Existing versatile watermarking systems typically rely on embedding explicit localization payloads, which introduces a fidelity--functionality trade-off: larger localization signals degrade visual quality and often reduce decoding robustness under strong generative edits. Moreover, these methods rarely support content recovery, limiting their forensic value when original evidence must be reconstructed. To address these challenges, we present VeriFi, a versatile watermarking framework that unifies copyright protection, pixel-level manipulation localization, and high-fidelity face content recovery. VeriFi makes three key contributions: (1) it embeds a compact semantic latent watermark that serves as a content-preserving prior, enabling faithful restoration even after severe manipulations; (2) it achieves fine-grained localization without dedicated payloads by correlating image features with decoded provenance signals; and (3) it introduces an AIGC attack simulator that combines latent-space mixing with seamless blending to enhance robustness against realistic deepfake pipelines. Extensive experiments on CelebA-HQ and FFHQ demonstrate that VeriFi consistently outperforms state-of-the-art baselines in watermark robustness, localization accuracy, and recovery quality, providing a practical and verifiable defense for deepfake forensics.

Peipeng Yu, Jinfeng Xie, Chengfu Ou +5

Jinan University, Guangzhou, Guangdong, China · University of Macau, Macau, China · Sun Yat-sen University, Guangzhou, Guangdong, China +1

Robust Watermarking Image Forgery Localization

Ask PDF

Oct 22, 2026

Spectra: Multi-Scale Implicit Decomposition for Deepfake Detection

Deepfakes increasingly undermine trust in digital media, yet existing detectors often struggle to generalize across unseen manipulation techniques, compression levels, and dataset shifts. Most approaches rely on RGB appearance, frequency-domain artifacts, or their combination, learning cues present in the final synthesized image. While effective in-distribution, such cues are frequently generator-specific and can diminish as synthesis models evolve. We argue that a complementary perspective can be obtained through implicit neural representations (INRs), which expose how an image is reconstructed rather than merely how it appears. By fitting an INR to an image and analyzing its internal activations, one can reveal reconstruction dynamics that are difficult to observe in the image domain alone and may capture more intrinsic differences between genuine and synthetic content. To this end, we propose Spectra, a deepfake detection framework that combines semantic visual representations with artifact-aware cues derived from INR reconstruction. Spectra extracts multi-level reconstruction heatmaps from image-specific INRs and jointly models them with semantic features through a bidirectional fusion mechanism. This enables direct interaction between image semantics and reconstruction artifacts, allowing the model to identify forgery cues that are less dependent on specific generation pipelines. Extensive experiments on multiple deepfake benchmarks demonstrate improved robustness and cross-dataset generalization, consistently outperforming detectors based solely on RGB appearance or semantic representations. Furthermore, we show that the proposed representation extends beyond deepfake detection to face presentation attack detection, highlighting its potential as a unified framework for facial attack detection.

Dhananjaya Jayasundara, Kartik Narayan, Vishal M. Patel

*All authors are with Johns Hopkins University

Deepfake Detection

Ask PDF

Aug 4, 2026

SpreadMark: Robust Image Watermarking via Spread-Spectrum Embedding

Invisible image watermarks are increasingly used for deepfake detection and provenance tracking, where they must survive not only incidental distortions but also deliberate removal. We revisit spread-spectrum embedding, a classical watermarking principle, inside a modern neural post-hoc watermarking architecture. Our starting point is a measurement: in existing encoder-decoder schemes each message bit occupies only a small fraction of the image, a shared contributing factor to their fragility, since removal then need only disturb the region a bit occupies. SpreadMark instead spreads each bit as a dense pseudo-random codeword over the whole image and recovers it by matched-filtering a learned cover-suppressed chip representation, with a parallel convolutional decoding path and sparsification-aware training. A conditional chip-space analysis shows that, under a codeword-independent perturbation model, dense spreading increases the budget required to disrupt matched-filter recovery. Evaluated on COCO and DIV2K against nine schemes, SpreadMark is the only evaluated method retaining high detection under both the regeneration and the latent-space sparsification settings we test, with competitive JPEG and additive-noise robustness. It keeps the embedded watermark imperceptible, maintaining high perceptual quality on both COCO and DIV2K.

Wei Song, Yuxin Cao, Zhenchang Xing +4

University of New South Wales, Australia · National University of Singapore, Singapore · CSIRO’s Data61, Australia

Dataset Watermarking Robust Watermarking

Ask PDF

Abstract

Image Restoration

Explore similar work

High-Fidelity Face Content Recovery via Tamper-Resilient Versatile Watermarking

Spectra: Multi-Scale Implicit Decomposition for Deepfake Detection

SpreadMark: Robust Image Watermarking via Spread-Spectrum Embedding