Context-Aware Mutual Learning for Blind Image Inpainting and Beyond

Haoru Zhao; Yufeng Wang; Zhaorui Gu; Bing Zheng; Haiyong Zheng

cs.CVSep 13, 2026

Context-Aware Mutual Learning for Blind Image Inpainting and Beyond

Authors: Haoru Zhao, Yufeng Wang, Zhaorui Gu, Bing Zheng, Haiyong Zheng

Organizations: Department of Economic Crime Investigation, Shandong Police College, Jinan 250200, China · College of Electronic Engineering, Ocean University of China, Qingdao 266000, China · Sanya Oceanographic Institution, Ocean University of China, Sanya 572000, China

Abstract

Blind image inpainting, aiming to recover contaminated images in the case of unknown masks, is a challenging task. Motivated by the perspective of human vision and knowledge, blind image inpainting can be decomposed into two stages: mask estimation and image inpainting based on the estimated mask. The two-stage idea exhibits evident advantages in enhancing inpainting quality and augmenting the generalization capability of unknown real-world contamination by explicitly employing the estimated mask for image inpainting compared to one-stage scheme. This two-stage idea has also been intuitively implemented. However, existing two-stage methods excessively emphasize the unilateral relationship of mask estimation to image inpainting, and may overlook the mutual relations between them. Specifically, mask estimation can provide more contextual semantics for image inpainting to strengthen the understanding of semantics, and image inpainting can offer more contextual details (e.g., textures and edges) for mask estimation to improve the learning of details. In this work, we propose a novel Context-Aware Mutual Learning (CAML) framework for blind image inpainting that joints mask estimation and image inpainting to mutually exploit contextual information. In the CAML framework, we design the Inpainting-Guided Context-Mutual (IGCM) learner to acquire the complementary contextual details from image inpainting for assisting mask estimation, and the Estimation-Guided Context-Mutual (EGCM) learner to strengthen the understanding of contextual semantics from mask estimation for assisting image inpainting. Ablation studies validate the efficacy of our CAML. Extensive experiments show that our CAML achieves state-of-the-art performance on both blind image inpainting and additional vision tasks, i.e., snow removal, shadow removal, and watermark removal, indicating its superiority.

Figures & tables

Figure 1: Feature visualization and visual comparison on FFHQ (top: mask estimation, bottom: image inpainting). (a) The contaminated image (Input). (b) Visualized features after the last layer of both encoders in (c) Baseline, which comprises two basic encoder-decoder networks for mask estimation and image inpainting respectively without mutuation. (d) Our CAML leverages context-aware mutual learning of mask estimation and image inpainting to produce more accurate estimated mask (top) and visually realistic yet reasonable inpainted image (bottom). (e) Ground Truth.

Figure 2: Our CAML framework contains two core components (a) Inpainting-Guided Context-Mutual (IGCM) learner and (b) Estimation-Guided Context-Mutual (EGCM) learner. Context-Mutual Adaptive Transfer (CMAT) block and Context-Mutual Adaptive Fusion (CMAF) block are core operations designed for IGCM learner and EGCM learner, with details shown in Figure 3 and Figure 4 respectively. Top: mask estimation. Bottom: image inpainting.

Figure 3: Details of Context-Mutual Adaptive Transfer (CMAT) block. Adapter is a critical component of CMAT, where “Conv( $k$ , $s$ , $p$ )” denotes convolution with kernel size $k$ , stride $s$ , and padding $p$ ; “MaxPool( $k$ , $s$ , $p$ )” denotes max pooling with kernel size $k$ , stride $s$ , and padding $p$ .

Figure 4: Details of Context-Mutual Adaptive Fusion (CMAF) block.

Estimating Encoder	Output	Inpainting Encoder	Output
ResBlock(3, 64, 3, 1, 1)	256	Conv(3, 64, 7, 1, 3)+IN(64)+ReLU()	256
IGCM learner	256	EGCM learner	256
ResBlock(64, 128, 3, 2, 1)	128	Conv(64, 128, 3, 2, 1)+IN(128)+ReLU()	128
IGCM learner	128	EGCM learner	128
ResBlock(128, 256, 3, 2, 1)	64	Conv(128, 256, 3, 2, 1)+IN(256)+ReLU()	64
IGCM learner	64	EGCM learner	64

Table 1: Detailed settings of our estimating encoder, mask estimator, inpainting encoder, and inpainting generator. “ResBlock(

c_{in}

c_{out}

k

s

p

)” denotes residual block [ 18 ] with

c_{in}

input channels,

c_{out}

output channels, kernel size

k

, stride

s

, and padding

p

. “Conv(

c_{in}

c_{out}

k

s

p

)” denotes convolution with

c_{in}

input channels,

c_{out}

output channels, kernel size

k

, stride

s

, and padding

p

. “Upsample(

s

)” means nearest-neighbor upsampling with a scale factor of

s

. “IN(

n

)” means instance normalization [ 52 ] with

n

dimensions. “Output” means output size.

Metric	Method	FFHQ			CelebA-HQ			ImageNet
Metric	Method	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$
BCE $\downarrow$	VCN	$0.79$	$0.69$	$0.87$	$0.66$	$0.82$	$0.92$	$0.66$	$0.81$	$0.90$
BCE $\downarrow$	CAML	$\mathbf{0.66}$	$\mathbf{0.60}$	$\mathbf{0.54}$	$\mathbf{0.44}$	$\mathbf{0.61}$	$\mathbf{0.54}$	$\mathbf{0.55}$	$\mathbf{0.63}$	$\mathbf{0.76}$
PSNR $\uparrow$	PICNet ♯	$23.59$	$21.70$	$16.29$	$26.84$	$23.29$	$18.80$	$25.34$	$20.84$	$16.05$
	CTSDG ♯	$27.58$	$24.25$	$20.05$	$28.86$	$24.35$	$20.36$	$25.93$	$22.00$	$18.36$
	LaMa ♯	$26.30$	$23.41$	$19.91$	$30.61$	$26.07$	$21.89$	$25.90$	$22.11$	$18.64$

Table 2: Quantitative comparison results on FFHQ, CelebA-HQ, and ImageNet datasets.

\sharp

means the non-blind image inpainting methods with ground truth masks, and otherwise means the blind image inpainting methods.

Metric	Method	Paris StreetView			Places2
Metric	Method	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$
BCE $\downarrow$	VCN	$0.70$	$0.83$	$0.92$	$0.66$	$0.81$	$0.91$
BCE $\downarrow$	CAML	$\mathbf{0.66}$	$\mathbf{0.62}$	$\mathbf{0.55}$	$\mathbf{0.41}$	$\mathbf{0.60}$	$\mathbf{0.53}$
PSNR $\uparrow$	PICNet ♯	$26.73$	$23.26$	$19.03$	$27.25$	$22.61$	$17.98$
	CTSDG ♯	$\mathbf{30.02}$	$25.87$	$21.76$	$29.60$	$25.12$	$21.28$
	LaMa ♯	$27.70$	$24.84$	$21.32$	$29.13$	$25.17$	$21.64$

Table 3: Quantitative comparison results on Paris StreetView and Places2 datasets.

Figure 5: Qualitative comparison results of mask estimation for blind image inpainting: (a) Input contaminated images, (b) VCN, (c) our CAML, and (d) Ground Truth Mask (GT Mask). From left to right (one example for each dataset): FFHQ, CelebA-HQ, ImageNet, Paris StreetView, and Places2.

Figure 6: Qualitative comparison results of image inpainting for blind image inpainting. From top to bottom (one example for each dataset): FFHQ, CelebA-HQ, and Paris StreetView.

Figure 7: Qualitative comparison results of image inpainting for blind image inpainting on Places2.

Figure 8: Qualitative comparison results of image inpainting for blind image inpainting on ImageNet.

Method		FFHQ	CelebA-HQ	ImageNet	Paris StreetView	Places2
CAML	$>$ PICNet ♯	$89.75\%$	$81.25\%$	$78.50\%$	$84.00\%$	$85.25\%$
	$>$ CTSDG ♯	$87.25\%$	$72.75\%$	$87.25\%$	$89.75\%$	$86.50\%$
	$>$ LaMa ♯	$76.25\%$	$68.50\%$	$80.50\%$	$82.75\%$	$81.75\%$
	$>$ MISF ♯	$89.50\%$	$84.00\%$	$88.00\%$	$85.25\%$	$91.75\%$
	$>$ CoordFill ♯	$79.00\%$	$71.75\%$	$86.75\%$	$87.50\%$	$72.75\%$
	$>$ Repaint ♯	$90.00\%$	$85.50\%$	$87.25\%$	$85.50\%$	$90.50\%$

Table 4: User study results. Each entry shows the percentage of cases where the proposed method is judged as more realistic and reasonable than all existing competitors.

\sharp

means the non-blind image inpainting methods with ground truth masks, and otherwise means the blind image inpainting methods.

Dataset	Contaminated Pattern	VCN					TransCNN-HAE
Dataset	Contaminated Pattern	BCE $\downarrow$ $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
FFHQ	Graffiti	$0.756$	$21.04$	$0.797$	$19.05$	$0.232$	$22.77$	$0.813$	$14.90$	$0.182$
	Text	$0.751$	$21.52$	$0.848$	$21.41$	$0.157$	$23.98$	$0.905$	$21.64$	$0.144$
	Image Occlusion	$0.755$	$20.32$	$0.815$	$61.84$	$0.421$	$23.88$	$0.895$	$56.06$	$0.257$
CelebA-HQ	Graffiti	$0.715$	$19.08$	$0.746$	$21.11$	$0.298$	$21.19$	$0.814$	$13.81$	$0.246$
	Text	$0.708$	$23.10$	$0.887$	$10.26$	$0.174$	$24.58$	$0.911$	$10.81$	$0.158$

Table 5: Quantitative comparison results on unseen contaminated patterns across all available datasets.

\uparrow

indicates the higher the better, and

\downarrow

indicates the lower the better. Bold means the best results.

Figure 9: Qualitative comparison results of unseen contaminated patterns for blind image inpainting on FFHQ and Paris StreetView: (a) Input contaminated images, (b) VCN (left: estimated masks, right: inpainted images), (c) OmniWavNet, (d) TransCNN-HAE, (e) our CAML (left: estimated masks, right: inpainted images), and (f) Ground Truth (left: masks, right: images).

Figure 10: Qualitative comparison results of real-world cases from Real-World Masked Face Dataset: (a) Input occluded faces, (b) VCN (left: estimated masks, right: inpainted images), (c) OmniWavNet, (d) TransCNN-HAE, and (e) our CAML (left: estimated masks, right: inpainted images).

Number	Embedding	BCE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	Params $\downarrow$	FLOPs $\downarrow$
	Position						(M)	(G)
$0$ IGCM	-	$0.591$	$20.45$	$0.812$	$13.51$	$0.151$	$17.09$	$98.65$
$0$ EGCM
$1$ IGCM $1$ EGCM	Encoder 1st	$0.577$	$22.61$	$0.862$	$11.59$	$0.134$	$17.89$	$125.93$
	Encoder 2nd	$0.575$	$22.68$	$0.864$	$11.01$	$0.132$	$18.87$	$125.86$
	Encoder 3rd	$0.574$	$22.79$	$0.868$	$11.00$	$0.135$	$22.81$	$125.80$

Table 6: Ablation studies of the analysis of IGCM and EGCM learners with different numbers and embedding positions on FFHQ.

Method		BCE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
CMAT block	w/o $\mathbf{S}_{i}$	$0.572$	$22.67$	$0.861$	$10.68$	$0.130$
CAML	w/o CMAT block	$0.591$	$21.78$	$0.814$	$12.12$	$0.147$
CAML	w/ CMAT block	$\mathbf{0.568}$	$\mathbf{23.40}$	$\mathbf{0.888}$	$\mathbf{9.01}$	$\mathbf{0.112}$

Table 7: Ablation studies of validating the efficacy of CMAT block’s design and usage on FFHQ.

Figure 11: Visual examples for ablation study of validating the efficacy of CMAT block’s design and usage on FFHQ: (a) Input contaminated image, CMAT block (b) w/o $\mathbf{S}_{i}$ (top: estimated mask, bottom: inpainted image), CAML (c) w/o CMAT block and (d) our w/ CMAT block (top: estimated mask, bottom: inpainted image), and (e) Ground Truth (top: mask, bottom: image).

Method		PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
CMAF block	w/o $f^{c}_{I_{i}}$	$22.84$	$0.871$	$10.05$	$0.141$
	w/o $f^{nc}_{I_{i}}$	$20.13$	$0.718$	$25.22$	$0.275$
	Min	$9.30$	$0.029$	$214.58$	$0.743$
	Max	$22.90$	$0.875$	$10.90$	$0.125$
	Average	$21.76$	$0.860$	$12.04$	$0.141$
CAML	w/o CMAF block	$22.06$	$0.865$	$10.84$	$0.133$

Table 8: Ablation studies of validating the efficacy of CMAF block’s design and usage on FFHQ.

Figure 12: Visual examples for ablation study of validating the efficacy of CMAF block’s design and usage on FFHQ.

Method	BCE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
Multi-task	$0.702$	$21.73$	$0.845$	$11.74$	$0.147$
Simultaneous	$0.570$	$23.21$	$0.885$	$9.42$	$0.117$
II first	$\mathbf{0.567}$	$23.23$	$0.884$	$9.34$	$0.118$
ME first	$0.568$	$\mathbf{23.40}$	$\mathbf{0.888}$	$\mathbf{9.01}$	$\mathbf{0.112}$

Table 9: Ablation studies of validating the efficacy of iterative training on FFHQ.

Method	Snow100K			CSD
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DesnowNet	$23.12$	$0.780$	$0.25$	$20.63$	$0.777$	$0.30$
HDCWNet	$21.29$	$0.640$	$0.40$	$28.66$	$0.892$	$0.11$
DDMSNet	$29.05$	$0.891$	$0.10$	$24.97$	$0.905$	$0.08$
TKL	$30.50$	$0.890$	$0.09$	$33.35$	$0.954$	$0.03$
CAML	$\mathbf{31.30}$	$\mathbf{0.973}$	$\mathbf{0.06}$	$\mathbf{34.03}$	$\mathbf{0.984}$	$\mathbf{0.01}$

Table 10: Quantitative comparison of our CAML with existing state-of-the-art snow removal methods (DesnowNet, HDCWNet, DDMSNet, and TKL) on Snow100K and CSD.

Figure 13: Qualitative comparison results of snow removal on the real-world dataset: (a) Input snowy images, (b) DesnowNet, (c) HDCWNet, (d) DDMSNet, (e) TKL, and (f) our CAML.

Method	ISTD			SRD
(RMSE $\downarrow$ )	Shadow	Non-Shadow	All	Shadow	Non-Shadow	All
DSC	$9.76$	$6.14$	$6.67$	$10.89$	$4.99$	$6.23$
DHAN	$8.14$	$6.04$	$6.37$	$8.94$	$4.80$	$5.67$
AEF	$7.77$	$5.56$	$5.92$	$8.56$	$5.75$	$6.61$
ShadowFormer	$6.76$	$4.44$	$4.79$	$5.90$	$3.44$	$4.04$
CAML	$\mathbf{5.59}$	$\mathbf{4.36}$	$\mathbf{4.57}$	$\mathbf{5.16}$	$\mathbf{2.67}$	$\mathbf{3.88}$

Table 11: Quantitative comparison of our CAML with existing state-of-the-art shadow removal methods (DSC, DHAN, AEF, and ShadowFormer) on ISTD and SRD.

Figure 14: Qualitative comparison results of shadow removal: (a) Input shadowed images, (b) DSC, (c) DHAN, (d) AEF, (e) ShadowFormer, (f) our CAML, and (g) Ground Truth. Top: ISTD dataset. Bottom: SRD dataset.

Method	LOGO30k			CLWD
Method	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$
BVMR	$38.28$	$0.985$	$3.87$	$35.89$	$0.973$	$5.02$
SplitNet	$41.27$	$0.991$	$2.74$	$37.41$	$0.978$	$4.23$
SLBA	$41.09$	$0.995$	$3.39$	$38.28$	$0.981$	$3.76$
CAML	$\mathbf{42.13}$	$\mathbf{0.996}$	$\mathbf{2.70}$	$\mathbf{38.32}$	$\mathbf{0.993}$	$\mathbf{2.52}$

Table 12: Quantitative comparison of our CAML with existing state-of-the-art watermark removal methods (BVMR, SplitNet, and SLBA) on LOGO30k and CLWD.

Figure 15: Qualitative comparison results of watermark removal: (a) Input watermarked images, (b) BVMR, (c) SplitNet, (d) SLBA, (e) our CAML, and (f) Ground Truth. Top: LOGO30K dataset. Bottom: CLWD dataset.

Image Inpainting

Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.

See all 92 papers

Jul 27, 2026

Image Inpainting via Stochastic Dynamics

Image inpainting aims to recover missing regions while preserving structural consistency. We propose a non-parametric method without network training based on data-guided stochastic dynamics. Starting from a masked image, the missing pixels are evolved through a reverse-time stochastic differential equation with a kernel-weighted correction estimated directly from a reference dataset. This empirical correction guides the reconstruction toward high-density regions of the data distribution without training a neural network or fitting a parametric density model. Experiments on MNIST, Fashion-MNIST, and MVTec show that the proposed method outperforms Mean Fill, Telea, and Navier-Stokes inpainting in PSNR, SSIM, and visual quality. On CelebA, it remains competitive and produces plausible completions for structure-sensitive occlusions. These results demonstrate the effectiveness of empirical reference statistics as a non-parametric prior for image inpainting.

Jiaqi Kuang, Zihao Guo, Zhongmin Qian

Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou, China · Institute for Advanced Research, Great Bay University, Dongguan, Guangdong, China · Mathematical Institute, University of Oxford, Andrew Wiles Building, Radcliffe Observatory Quarter, Oxford, OX2 6GG, Oxfordshire, United Kingdom

Image Restoration Image Inpainting

Ask PDF

Nov 25, 2025

SONIC: Spectral Optimization of Noise for Inpainting with Consistency

We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting -- in practice their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue the missing ingredient for training-free generic model usage is proper optimization of the initial noise sample. We optimize the initial noise to approximately reproduce the unmasked image, in as few as tens of optimization steps, then use it with a conventional training-free inpainting method. Critically, we propose two core ideas that make this possible: (i) we perform linear approximation that avoids the costly and often impractical unrolling required to relate the initial noise sample to model output -- which potentially is why this relationship was previously overlooked; and (ii) perform spectral preconditioning by optimizing the initial noise sample in the spectral domain with Adam, which stabilizes the optimization. We demonstrate our method on various inpainting tasks, outperforming the state of the art. Project website: https://ubc-vision.github.io/sonic/

Seungyeon Baek, Erqun Dong, Shadan Namazifard +2

University of British Columbia, Canada · Google DeepMind, USA

Diffusion Model Inversion Image Inpainting Diffusion-Based Inverse Problems T2I Diffusion Models Diffusion Models

Ask PDF

Aug 13, 2026

SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting

Sketch-guided image inpainting provides intuitive structural control, yet real sketches often mix reliable global intent with locally crowded, displaced, incomplete, or deliberately unconventional strokes. Existing approaches typically either retain the input sketch as a fixed condition throughout denoising or refine it into a clean structure before RGB synthesis. The former assumes uniformly reliable strokes and can propagate local errors throughout generation; the latter must resolve ambiguous structure before emerging appearance and semantic context become available. We propose SketchSense, a framework that interprets imperfect sketch guidance by synchronously denoising interacting RGB and structure streams. Bidirectional Attention Fusion couples appearance generation with structural recovery, producing a refined structure that exposes the model's evolving sketch interpretation. A phrase-level objective aligns the semantic grounding of the two streams. Sketch-Aware Spatial Regulation further adapts sketch use to local generation states by modulating attention and the fusion process, while an optional signed prior injects preserve-versus-correct intent into feature representations and attention behavior. Experiments on natural and structurally complex imagery show substantial gains over existing methods in both restoration quality and structural fidelity.

Zian Yang

Fudan University

Diffusion Models for Image Restoration Diffusion Model Guidance Image Inpainting

Ask PDF

Abstract

Image Inpainting

Explore similar work

Image Inpainting via Stochastic Dynamics

SONIC: Spectral Optimization of Noise for Inpainting with Consistency

SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting