Context-Aware Mutual Learning for Blind Image Inpainting and Beyond

Haoru Zhao; Yufeng Wang; Zhaorui Gu; Bing Zheng; Haiyong Zheng

cs.CVSep 13, 2026

Context-Aware Mutual Learning for Blind Image Inpainting and Beyond

Authors: Haoru Zhao, Yufeng Wang, Zhaorui Gu, Bing Zheng, Haiyong Zheng

Organizations: Department of Economic Crime Investigation, Shandong Police College, Jinan 250200, China · College of Electronic Engineering, Ocean University of China, Qingdao 266000, China · Sanya Oceanographic Institution, Ocean University of China, Sanya 572000, China

Abstract

Blind image inpainting, aiming to recover contaminated images in the case of unknown masks, is a challenging task. Motivated by the perspective of human vision and knowledge, blind image inpainting can be decomposed into two stages: mask estimation and image inpainting based on the estimated mask. The two-stage idea exhibits evident advantages in enhancing inpainting quality and augmenting the generalization capability of unknown real-world contamination by explicitly employing the estimated mask for image inpainting compared to one-stage scheme. This two-stage idea has also been intuitively implemented. However, existing two-stage methods excessively emphasize the unilateral relationship of mask estimation to image inpainting, and may overlook the mutual relations between them. Specifically, mask estimation can provide more contextual semantics for image inpainting to strengthen the understanding of semantics, and image inpainting can offer more contextual details (e.g., textures and edges) for mask estimation to improve the learning of details. In this work, we propose a novel Context-Aware Mutual Learning (CAML) framework for blind image inpainting that joints mask estimation and image inpainting to mutually exploit contextual information. In the CAML framework, we design the Inpainting-Guided Context-Mutual (IGCM) learner to acquire the complementary contextual details from image inpainting for assisting mask estimation, and the Estimation-Guided Context-Mutual (EGCM) learner to strengthen the understanding of contextual semantics from mask estimation for assisting image inpainting. Ablation studies validate the efficacy of our CAML. Extensive experiments show that our CAML achieves state-of-the-art performance on both blind image inpainting and additional vision tasks, i.e., snow removal, shadow removal, and watermark removal, indicating its superiority.

Figures & tables

Figure 1: Feature visualization and visual comparison on FFHQ (top: mask estimation, bottom: image inpainting). (a) The contaminated image (Input). (b) Visualized features after the last layer of both encoders in (c) Baseline, which comprises two basic encoder-decoder networks for mask estimation and image inpainting respectively without mutuation. (d) Our CAML leverages context-aware mutual learning of mask estimation and image inpainting to produce more accurate estimated mask (top) and visually realistic yet reasonable inpainted image (bottom). (e) Ground Truth.

Figure 2: Our CAML framework contains two core components (a) Inpainting-Guided Context-Mutual (IGCM) learner and (b) Estimation-Guided Context-Mutual (EGCM) learner. Context-Mutual Adaptive Transfer (CMAT) block and Context-Mutual Adaptive Fusion (CMAF) block are core operations designed for IGCM learner and EGCM learner, with details shown in Figure 3 and Figure 4 respectively. Top: mask estimation. Bottom: image inpainting.

Figure 3: Details of Context-Mutual Adaptive Transfer (CMAT) block. Adapter is a critical component of CMAT, where “Conv( $k$ , $s$ , $p$ )” denotes convolution with kernel size $k$ , stride $s$ , and padding $p$ ; “MaxPool( $k$ , $s$ , $p$ )” denotes max pooling with kernel size $k$ , stride $s$ , and padding $p$ .

Figure 4: Details of Context-Mutual Adaptive Fusion (CMAF) block.

Estimating Encoder	Output	Inpainting Encoder	Output
ResBlock(3, 64, 3, 1, 1)	256	Conv(3, 64, 7, 1, 3)+IN(64)+ReLU()	256
IGCM learner	256	EGCM learner	256
ResBlock(64, 128, 3, 2, 1)	128	Conv(64, 128, 3, 2, 1)+IN(128)+ReLU()	128
IGCM learner	128	EGCM learner	128
ResBlock(128, 256, 3, 2, 1)	64	Conv(128, 256, 3, 2, 1)+IN(256)+ReLU()	64
IGCM learner	64	EGCM learner	64

Table 1: Detailed settings of our estimating encoder, mask estimator, inpainting encoder, and inpainting generator. “ResBlock(

c_{in}

c_{out}

k

s

p

)” denotes residual block [ 18 ] with

c_{in}

input channels,

c_{out}

output channels, kernel size

k

, stride

s

, and padding

p

. “Conv(

c_{in}

c_{out}

k

s

p

)” denotes convolution with

c_{in}

input channels,

c_{out}

output channels, kernel size

k

, stride

s

, and padding

p

. “Upsample(

s

)” means nearest-neighbor upsampling with a scale factor of

s

. “IN(

n

)” means instance normalization [ 52 ] with

n

dimensions. “Output” means output size.

Metric	Method	FFHQ			CelebA-HQ			ImageNet
Metric	Method	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$
BCE $\downarrow$	VCN	$0.79$	$0.69$	$0.87$	$0.66$	$0.82$	$0.92$	$0.66$	$0.81$	$0.90$
BCE $\downarrow$	CAML	$\mathbf{0.66}$	$\mathbf{0.60}$	$\mathbf{0.54}$	$\mathbf{0.44}$	$\mathbf{0.61}$	$\mathbf{0.54}$	$\mathbf{0.55}$	$\mathbf{0.63}$	$\mathbf{0.76}$
PSNR $\uparrow$	PICNet ♯	$23.59$	$21.70$	$16.29$	$26.84$	$23.29$	$18.80$	$25.34$	$20.84$	$16.05$
	CTSDG ♯	$27.58$	$24.25$	$20.05$	$28.86$	$24.35$	$20.36$	$25.93$	$22.00$	$18.36$
	LaMa ♯	$26.30$	$23.41$	$19.91$	$30.61$	$26.07$	$21.89$	$25.90$	$22.11$	$18.64$

Table 2: Quantitative comparison results on FFHQ, CelebA-HQ, and ImageNet datasets.

\sharp

means the non-blind image inpainting methods with ground truth masks, and otherwise means the blind image inpainting methods.

Metric	Method	Paris StreetView			Places2
Metric	Method	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$	$0$ - $20\%$	$20$ - $40\%$	$40$ - $60\%$
BCE $\downarrow$	VCN	$0.70$	$0.83$	$0.92$	$0.66$	$0.81$	$0.91$
BCE $\downarrow$	CAML	$\mathbf{0.66}$	$\mathbf{0.62}$	$\mathbf{0.55}$	$\mathbf{0.41}$	$\mathbf{0.60}$	$\mathbf{0.53}$
PSNR $\uparrow$	PICNet ♯	$26.73$	$23.26$	$19.03$	$27.25$	$22.61$	$17.98$
	CTSDG ♯	$\mathbf{30.02}$	$25.87$	$21.76$	$29.60$	$25.12$	$21.28$
	LaMa ♯	$27.70$	$24.84$	$21.32$	$29.13$	$25.17$	$21.64$

Table 3: Quantitative comparison results on Paris StreetView and Places2 datasets.

Figure 5: Qualitative comparison results of mask estimation for blind image inpainting: (a) Input contaminated images, (b) VCN, (c) our CAML, and (d) Ground Truth Mask (GT Mask). From left to right (one example for each dataset): FFHQ, CelebA-HQ, ImageNet, Paris StreetView, and Places2.

Figure 6: Qualitative comparison results of image inpainting for blind image inpainting. From top to bottom (one example for each dataset): FFHQ, CelebA-HQ, and Paris StreetView.

Figure 7: Qualitative comparison results of image inpainting for blind image inpainting on Places2.

Figure 8: Qualitative comparison results of image inpainting for blind image inpainting on ImageNet.

Method		FFHQ	CelebA-HQ	ImageNet	Paris StreetView	Places2
CAML	$>$ PICNet ♯	$89.75\%$	$81.25\%$	$78.50\%$	$84.00\%$	$85.25\%$
	$>$ CTSDG ♯	$87.25\%$	$72.75\%$	$87.25\%$	$89.75\%$	$86.50\%$
	$>$ LaMa ♯	$76.25\%$	$68.50\%$	$80.50\%$	$82.75\%$	$81.75\%$
	$>$ MISF ♯	$89.50\%$	$84.00\%$	$88.00\%$	$85.25\%$	$91.75\%$
	$>$ CoordFill ♯	$79.00\%$	$71.75\%$	$86.75\%$	$87.50\%$	$72.75\%$
	$>$ Repaint ♯	$90.00\%$	$85.50\%$	$87.25\%$	$85.50\%$	$90.50\%$

Table 4: User study results. Each entry shows the percentage of cases where the proposed method is judged as more realistic and reasonable than all existing competitors.

\sharp

means the non-blind image inpainting methods with ground truth masks, and otherwise means the blind image inpainting methods.

Dataset	Contaminated Pattern	VCN					TransCNN-HAE
Dataset	Contaminated Pattern	BCE $\downarrow$ $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
FFHQ	Graffiti	$0.756$	$21.04$	$0.797$	$19.05$	$0.232$	$22.77$	$0.813$	$14.90$	$0.182$
	Text	$0.751$	$21.52$	$0.848$	$21.41$	$0.157$	$23.98$	$0.905$	$21.64$	$0.144$
	Image Occlusion	$0.755$	$20.32$	$0.815$	$61.84$	$0.421$	$23.88$	$0.895$	$56.06$	$0.257$
CelebA-HQ	Graffiti	$0.715$	$19.08$	$0.746$	$21.11$	$0.298$	$21.19$	$0.814$	$13.81$	$0.246$
	Text	$0.708$	$23.10$	$0.887$	$10.26$	$0.174$	$24.58$	$0.911$	$10.81$	$0.158$

Table 5: Quantitative comparison results on unseen contaminated patterns across all available datasets.

\uparrow

indicates the higher the better, and

\downarrow

indicates the lower the better. Bold means the best results.

Figure 9: Qualitative comparison results of unseen contaminated patterns for blind image inpainting on FFHQ and Paris StreetView: (a) Input contaminated images, (b) VCN (left: estimated masks, right: inpainted images), (c) OmniWavNet, (d) TransCNN-HAE, (e) our CAML (left: estimated masks, right: inpainted images), and (f) Ground Truth (left: masks, right: images).

Figure 10: Qualitative comparison results of real-world cases from Real-World Masked Face Dataset: (a) Input occluded faces, (b) VCN (left: estimated masks, right: inpainted images), (c) OmniWavNet, (d) TransCNN-HAE, and (e) our CAML (left: estimated masks, right: inpainted images).

Number	Embedding	BCE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	Params $\downarrow$	FLOPs $\downarrow$
	Position						(M)	(G)
$0$ IGCM	-	$0.591$	$20.45$	$0.812$	$13.51$	$0.151$	$17.09$	$98.65$
$0$ EGCM
$1$ IGCM $1$ EGCM	Encoder 1st	$0.577$	$22.61$	$0.862$	$11.59$	$0.134$	$17.89$	$125.93$
	Encoder 2nd	$0.575$	$22.68$	$0.864$	$11.01$	$0.132$	$18.87$	$125.86$
	Encoder 3rd	$0.574$	$22.79$	$0.868$	$11.00$	$0.135$	$22.81$	$125.80$

Table 6: Ablation studies of the analysis of IGCM and EGCM learners with different numbers and embedding positions on FFHQ.

Method		BCE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
CMAT block	w/o $\mathbf{S}_{i}$	$0.572$	$22.67$	$0.861$	$10.68$	$0.130$
CAML	w/o CMAT block	$0.591$	$21.78$	$0.814$	$12.12$	$0.147$
CAML	w/ CMAT block	$\mathbf{0.568}$	$\mathbf{23.40}$	$\mathbf{0.888}$	$\mathbf{9.01}$	$\mathbf{0.112}$

Table 7: Ablation studies of validating the efficacy of CMAT block’s design and usage on FFHQ.

Figure 11: Visual examples for ablation study of validating the efficacy of CMAT block’s design and usage on FFHQ: (a) Input contaminated image, CMAT block (b) w/o $\mathbf{S}_{i}$ (top: estimated mask, bottom: inpainted image), CAML (c) w/o CMAT block and (d) our w/ CMAT block (top: estimated mask, bottom: inpainted image), and (e) Ground Truth (top: mask, bottom: image).

Method		PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
CMAF block	w/o $f^{c}_{I_{i}}$	$22.84$	$0.871$	$10.05$	$0.141$
	w/o $f^{nc}_{I_{i}}$	$20.13$	$0.718$	$25.22$	$0.275$
	Min	$9.30$	$0.029$	$214.58$	$0.743$
	Max	$22.90$	$0.875$	$10.90$	$0.125$
	Average	$21.76$	$0.860$	$12.04$	$0.141$
CAML	w/o CMAF block	$22.06$	$0.865$	$10.84$	$0.133$

Table 8: Ablation studies of validating the efficacy of CMAF block’s design and usage on FFHQ.

Figure 12: Visual examples for ablation study of validating the efficacy of CMAF block’s design and usage on FFHQ.

Method	BCE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
Multi-task	$0.702$	$21.73$	$0.845$	$11.74$	$0.147$
Simultaneous	$0.570$	$23.21$	$0.885$	$9.42$	$0.117$
II first	$\mathbf{0.567}$	$23.23$	$0.884$	$9.34$	$0.118$
ME first	$0.568$	$\mathbf{23.40}$	$\mathbf{0.888}$	$\mathbf{9.01}$	$\mathbf{0.112}$

Table 9: Ablation studies of validating the efficacy of iterative training on FFHQ.

Method	Snow100K			CSD
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DesnowNet	$23.12$	$0.780$	$0.25$	$20.63$	$0.777$	$0.30$
HDCWNet	$21.29$	$0.640$	$0.40$	$28.66$	$0.892$	$0.11$
DDMSNet	$29.05$	$0.891$	$0.10$	$24.97$	$0.905$	$0.08$
TKL	$30.50$	$0.890$	$0.09$	$33.35$	$0.954$	$0.03$
CAML	$\mathbf{31.30}$	$\mathbf{0.973}$	$\mathbf{0.06}$	$\mathbf{34.03}$	$\mathbf{0.984}$	$\mathbf{0.01}$

Table 10: Quantitative comparison of our CAML with existing state-of-the-art snow removal methods (DesnowNet, HDCWNet, DDMSNet, and TKL) on Snow100K and CSD.

Figure 13: Qualitative comparison results of snow removal on the real-world dataset: (a) Input snowy images, (b) DesnowNet, (c) HDCWNet, (d) DDMSNet, (e) TKL, and (f) our CAML.

Method	ISTD			SRD
(RMSE $\downarrow$ )	Shadow	Non-Shadow	All	Shadow	Non-Shadow	All
DSC	$9.76$	$6.14$	$6.67$	$10.89$	$4.99$	$6.23$
DHAN	$8.14$	$6.04$	$6.37$	$8.94$	$4.80$	$5.67$
AEF	$7.77$	$5.56$	$5.92$	$8.56$	$5.75$	$6.61$
ShadowFormer	$6.76$	$4.44$	$4.79$	$5.90$	$3.44$	$4.04$
CAML	$\mathbf{5.59}$	$\mathbf{4.36}$	$\mathbf{4.57}$	$\mathbf{5.16}$	$\mathbf{2.67}$	$\mathbf{3.88}$

Table 11: Quantitative comparison of our CAML with existing state-of-the-art shadow removal methods (DSC, DHAN, AEF, and ShadowFormer) on ISTD and SRD.

Figure 14: Qualitative comparison results of shadow removal: (a) Input shadowed images, (b) DSC, (c) DHAN, (d) AEF, (e) ShadowFormer, (f) our CAML, and (g) Ground Truth. Top: ISTD dataset. Bottom: SRD dataset.

Method	LOGO30k			CLWD
Method	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$
BVMR	$38.28$	$0.985$	$3.87$	$35.89$	$0.973$	$5.02$
SplitNet	$41.27$	$0.991$	$2.74$	$37.41$	$0.978$	$4.23$
SLBA	$41.09$	$0.995$	$3.39$	$38.28$	$0.981$	$3.76$
CAML	$\mathbf{42.13}$	$\mathbf{0.996}$	$\mathbf{2.70}$	$\mathbf{38.32}$	$\mathbf{0.993}$	$\mathbf{2.52}$

Table 12: Quantitative comparison of our CAML with existing state-of-the-art watermark removal methods (BVMR, SplitNet, and SLBA) on LOGO30k and CLWD.

Figure 15: Qualitative comparison results of watermark removal: (a) Input watermarked images, (b) BVMR, (c) SplitNet, (d) SLBA, (e) our CAML, and (f) Ground Truth. Top: LOGO30K dataset. Bottom: CLWD dataset.

Image Inpainting

Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.

See all 92 papers

Image Inpainting via Stochastic Dynamics
Jul 27, 2026Jiaqi Kuang, Zihao Guo, Zhongmin QianImage Restoration Image Inpainting
Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou, China · Institute for Advanced Research, Great Bay University, Dongguan, Guangdong, China · Mathematical Institute, University of Oxford, Andrew Wiles Building, Radcliffe Observatory Quarter, Oxford, OX2 6GG, Oxfordshire, United Kingdom
SONIC: Spectral Optimization of Noise for Inpainting with Consistency
Nov 25, 2025Seungyeon Baek, Erqun Dong, Shadan Namazifard +2Diffusion Model Inversion Image Inpainting
University of British Columbia, Canada · Google DeepMind, USA
SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting
Aug 13, 2026Zian YangDiffusion Models for Image Restoration Diffusion Model Guidance
Fudan University

Jul 27, 2026J/K move · Enter open · S save

Image Inpainting via Stochastic Dynamics

Jiaqi Kuang, Zihao Guo, Zhongmin Qian

Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou, China · Institute for Advanced Research, Great Bay University, Dongguan, Guangdong, China · Mathematical Institute, University of Oxford, Andrew Wiles Building, Radcliffe Observatory Quarter, Oxford, OX2 6GG, Oxfordshire, United Kingdom

Image Restoration Image Inpainting

Open paper AskPDF arXiv

Image inpainting aims to recover missing regions while preserving structural consistency. We propose a non-parametric method without network training based on data-guided stochastic dynamics. Starting from a masked image, the missing pixels are evolved through a reverse-time stochastic differential equation with a kernel-weighted correction estimated directly from a reference dataset. This empirical correction guides the reconstruction toward high-density regions of the data distribution without training a neural network or fitting a parametric density model. Experiments on MNIST, Fashion-MNIST, and MVTec show that the proposed method outperforms Mean Fill, Telea, and Navier-Stokes inpainting in PSNR, SSIM, and visual quality. On CelebA, it remains competitive and produces plausible completions for structure-sensitive occlusions. These results demonstrate the effectiveness of empirical reference statistics as a non-parametric prior for image inpainting.

Abstract

Image Inpainting

Explore similar work

Image Inpainting via Stochastic Dynamics

SONIC: Spectral Optimization of Noise for Inpainting with Consistency

SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting