Beyond Detection: Multi-Scale Hidden-Code for Natural Image Deepfake Recovery and Factual Retrieval

Yuan-Chih Chen; Chun-Shien Lu

Figure 1: Overall workflow of the proposed method. In the multi-scale token watermarking stage, our goal is to protect an image

I

by embedding image content–related information to produce a watermarked image

I_{w}

. The original image

I

is first converted into quantized multi-scale tokens

z_{s_{1}},\ldots,z_{s_{k}}

using a VQ-VAE. These tokens are then flattened and transformed into a bit string

h

. The bit string

h

is embedded into

I

as a content watermark through the watermark injection encoder

E_{w}

, where the embedding length is constrained by the watermark capacity

|m|

such that

|h|\leq|m|

. When a tampered image

I_{d}

is generated from the watermarked image

I_{w}

by malicious manipulations (e.g., object removal or inpainting), our method enables deepfake recovery using the content-related information embedded in

I_{w}

. The tampered image

I_{d}

is decoded by the watermark decoder

D_{w}

to extract the hidden quantized multi-scale tokens

h^{\prime}

and by the localization decoder

D_{loc}

to produce a deepfake localization map

M_{loc}

. Finally, the multi-scale recovery transformer reconstructs the tampered image

I_{d}

based on

h^{\prime}

,

M_{loc}

, and the quantized tokens

h_{d}

of the tampered image.

Figure 2: Visualization of the dropout effect during quantizer (VQ-VAE) training. From left to right, each column shows the reconstructed images using token maps

(z_{s_{1}},\ldots,z_{s_{k}})

with

1\leq k\leq 10

. Dropout encourages semantic robustness across scales, allowing lower-level representations (smaller

k

) to retain more global structure and meaningful content. In contrast, the model trained without dropout produces coarse, less informative reconstructions at lower scales.

Figure 3: Illustration of proposed conditional deepfake recovery Transformer. The predicted hidden codes

h^{\prime}

are used as conditional tokens to guide the generation process, enabling context-aware reconstruction. The untampered regions

\tilde{z}_{s_{i}}

of deepfaked image provide additional contextual cues, helping the model produce more detailed and consistent predictions at higher scales.

ImageNet	Same samples				Different samples		CLIP score ( $\uparrow$ )
Methods \ Metric	Top-1 label ( $\uparrow$ )	Top-5 label ( $\uparrow$ )	Top-1 image ( $\uparrow$ )	Top-5 image ( $\uparrow$ )	Top-1 label ( $\uparrow$ )	Top-5 label ( $\uparrow$ )
HiNet [ 11 ]	0.2393	0.3511	0.2017	0.2739	0.0742	0.0716	0.7255
RePaint [ 15 ]	0.2821	0.3868	0.2571	0.3373	0.0717	0.1693	0.7854
VQGAN [ 6 ] (codebook size $V$ = 1024)	0.3633	0.5516	0.2756	0.4243	0.1787	0.3699	0.7887
VQGAN [ 6 ] (codebook size $V$ = 16384)	0.5205	0.6895	0.4246	0.5925	0.2467	0.4535	0.8157
VAR [ 23 ]	0.6754	0.8125	0.5761	0.7400	0.4222	0.6547	0.8817

Table 1: Comparison of retrieval accuracy and CLIP score between our method and related approaches. Deepfake images are generated from the ImageNet-S dataset using Stable Diffusion. “Same samples” indicates that the original image is included in the evaluation dataset, whereas “Different samples” indicates that the original image is excluded, allowing retrieval only within the same class. “Top-

k

label” denotes the retrieval accuracy based on whether any of the top-

k

retrieved images from ImageNet-S share the same label as the original image. “Top-

k

image” denotes the retrieval accuracy based on whether the original image itself appears among the top-

k

retrieved images. Higher CLIP scores indicate better semantic consistency between the recovered and original images.

Figure 4: Qualitative comparison of tampered image recovery performance. The first two columns show the Original Image and its Tampering mask. The third column, Deepfaked Image, shows the tampered image generated using Stable Diffusion based on the mask. While general recovery methods (HINet [ 11 ] , RePaint [ 15 ] ) often fail to maintain semantic consistency or introduce severe distortions. The general quantization methods (VQGAN, VAR) with our watermarking scheme struggle with texture preservation, our method successfully reconstructs the masked region, restoring the identity and realistic appearance of the original subjects (birds and an ostrich).

Method	Type	Hiding Information	Smallest Hiding Unit	Hiding Domain	Bits per Pixel (BPP)	Total Hidden Information
HiNet [ 11 ]	Reversible module	Pixel-level content	$1\times 1$	Latent space	–	$H\times W\times C$
RePaint [ 15 ]	Inpainting model	–	–	–	–	–
Common self-recovery watermarking [ 1 ]	Fragile watermark	MSBs, block averages, or transform-domain coefficients	$2\times 2$ – $6\times 6$	Pixel (LSB)	2–4	$\text{bpp}\times H\times W\times C$
Bouarroudj et al. [ 4 ]	Fragile watermark	DWT coefficients	$4\times 4$	Frequency domain	0.5	$\text{bpp}\times H\times W\times C$
Ours	Robust watermark	Multi-scale quantized vectors	$16\times 16$	Latent space	–	$\|h\|$ ( $<\|m\|$ )

Table 2: Comparison of different types of image recovery and watermarking approaches. Traditional self-recovery watermarking methods embed large amounts of pixel or transform information into fragile regions such as LSBs, leading to high BPP and poor robustness. In contrast, our method encodes compact hidden-code representations through robust, multi-scale quantization in the latent space, achieving efficient and resilient recovery.

Methods	Clean ( $\uparrow$ )	JPEG( $\uparrow$ )	Gaussian Blur( $\uparrow$ )	Gaussian Noise ( $\uparrow$ )
		QF $70\%$	$5\times 5$	$\sigma=1$	$\sigma=5$
Bouarroudj et al. [ 4 ]	1.0000	0.5054	0.5042	0.5851	0.5047
Ours(P)	0.9988	0.9923	0.9983	0.9983	0.9577
Ours(I)	0.9999	0.9951	0.9971	0.9971	0.9951

Table 3: Comparison of bit accuracy between the self-recovery watermarking method [ 4 ] and our proposed approaches (Ours(P) denotes the use of post-hoc watermarking model–EditGuard and Ours(I) denotes the use of in-generation model–Gaussian shading) under various image degradations. Higher values indicate better robustness in bit preservation.

Figure 5: Illustration of single-scale versus multi-scale quantization. (a) Single-scale quantization: each token

x_{i}

corresponds to a specific spatial position in the image, and the transformer predicts

x_{i}

sequentially based on all preceding tokens

(x_{1},x_{2},\ldots,x_{i-1})

. (b) Multi-scale quantization: each token map

z_{s_{i}}

represents a group of tokens at a particular scale

s_{i}

. The transformer predicts

z_{s_{i}}

conditioned on all previous token maps

(z_{s_{1}},z_{s_{2}},\ldots,z_{s_{i-1}})

.

Figure 6: Visualization of the limitations of single-scale tokenization at varying image resolutions. (a) Original input image (

256{\times}256

). (b) Reconstructed images using VQGAN, from left to right, correspond to image sizes of

256{\times}256

,

128{\times}128

,

96{\times}96

, and

64{\times}64

. Even at

128{\times}128

, the reconstruction begins to lose fine details. This degradation occurs because downsampling increases the information contained within each patch while the codebook remains fixed, forcing each quantized vector to represent a larger and more complex region. Consequently, the quantized representation becomes insufficient to capture fine-grained details, resulting in visible information loss and reduced visual fidelity.

Methods	SSIM ( $\uparrow$ )	PSNR ( $\uparrow$ )	LPIPS ( $\downarrow$ )
HiNet [ 11 ]	0.7024	18.7884	0.0375
RePaint [ 15 ]	0.7832	18.9624	0.0436
VQGAN [ 6 ] ( $V$ = 1024)	0.7987	22.6042	0.0208
VQGAN [ 6 ] ( $V$ = 16384)	0.8029	23.1354	0.0199
VAR [ 23 ]	0.7993	16.5914	0.0249
hidden code (Ours)	0.8238	24.6575	0.0190

Table 4: Comparison of image quality across different recovery approaches. All deepfake images were generated from the ImageNet-S dataset using Stable Diffusion.

Methods	Gaussian Shading	Guassian Shading + ours
	Forgery BA ( $\uparrow$ )	Forgery BA ( $\downarrow$ )
Average attack ( $n=5$ , grey box setting)	0.8999	0.5142
Average attack ( $n=5$ , black box setting)	0.8000	0.5079
Average attack ( $n=10$ , grey box setting)	0.9126	0.5125
Average attack ( $n=10$ , black box setting)	0.8141	0.5047
Average attack ( $n=20$ , grey box setting)	0.9328	0.5119

Table 5: Comparison of Forgery Attack Success Rate (Bit Accuracy, BA) under Black-box and Grey-box Settings. We evaluate the success of forgery attacks (

\text{BA}\uparrow

) against a baseline method and against our proposed framework (

\text{BA}\downarrow

). The BA quantifies the attacker’s ability to guess the hidden code bit, where

\text{BA}\approx 0.5

signifies a failed attack (random guessing). The Average Attack uses

n

watermarked images to estimate the code. The Grey-box setting assumes the attacker has access to the average clean image, while the Black-box setting uses an average image from a different dataset. Our method consistently forces the attacker’s BA toward 0.5, demonstrating a significantly higher level of security compared to the baseline.

ImageNet	CLIP score( $\uparrow$ )						CLIP score( $\downarrow$ )
Methods \ Metric	Original	Crop	Color jitter	JPEG	Gaussian Blur	Gaussian Noise	Different image
		$50\%$	$30\%$	QF $70\%$	$5\times 5$	$\sigma=5$
VQGAN [ 6 ] (codebook size $V$ = 1024)	0.7939	0.7984	0.7906	0.7730	0.8307	0.7817	0.6787
VQGAN [ 6 ] (codebook size $V$ = 16384)	0.8208	0.8229	0.8061	0.7949	0.8572	0.8040	0.6571
VAR [ 23 ]	0.8845	0.8692	0.8385	0.8418	0.8582	0.8529	0.6552
hidden code (Ours)	0.9021	0.8885	0.8503	0.8765	0.8784	0.8948	0.6176

Table 6: CLIP Similarity Analysis (Semantic Consistency). We report the CLIP score between the reconstructed image (derived from the embedded hidden code) and the image variants to evaluate both robustness (higher scores for image degradations) and forgery resistance (lower scores for unrelated images). CLIP score (

\uparrow

) indicates better robustness, while CLIP score (

\downarrow

) for the “Different image” column indicates better forgery resistance.

Beyond Detection: Multi-Scale Hidden-Code for Natural Image Deepfake Recovery and Factual Retrieval

Abstract

Figures & tables

Image Restoration

High-Fidelity Face Content Recovery via Tamper-Resilient Versatile Watermarking

Spectra: Multi-Scale Implicit Decomposition for Deepfake Detection

SpreadMark: Robust Image Watermarking via Spread-Spectrum Embedding

Abstract

Image Restoration

Explore similar work

High-Fidelity Face Content Recovery via Tamper-Resilient Versatile Watermarking

Spectra: Multi-Scale Implicit Decomposition for Deepfake Detection

SpreadMark: Robust Image Watermarking via Spread-Spectrum Embedding