How Do Inpainting Artifacts Propagate to Language?

Pratham Yashwante; Davit Abrahamyan; Shresth Grover; Sukruth Rao

cs.CVFeb 24, 2026

How Do Inpainting Artifacts Propagate to Language?

Authors: Pratham Yashwante, Davit Abrahamyan, Shresth Grover, Sukruth Rao

Organizations: UC San Diego

Abstract

We study how visual artifacts introduced by diffusion-based inpainting affect language generation in vision-language models. We use a two-stage diagnostic setup in which masked image regions are reconstructed and then provided to captioning models, enabling controlled comparisons between captions generated from original and reconstructed inputs. Across multiple datasets, we analyze the relationship between reconstruction fidelity and downstream caption quality. We observe consistent associations between pixel-level and perceptual reconstruction metrics and both lexical and semantic captioning performance. Additional analysis of intermediate visual representations and attention patterns shows that inpainting artifacts lead to systematic, layer-dependent changes in model behavior. Together, these results provide a practical diagnostic framework for examining how visual reconstruction quality influences language generation in multimodal systems.

Figures & tables

Panel of Figure 1 — Figure 1: Qualitative examples illustrating captioning errors induced by center-region inpainting. Incorrect semantic attributes introduced by inpainting are highlighted in red , while the correct interpretation is shown in green .

Figure 2: Degradation–reconstruction–captioning framework used to evaluate how inpainting artifacts propagate into downstream language outputs.

Figure 3: Masking examples illustrating center-region degradations on (A) Flickr and (B) RefCOCOg.

Figure 4: Correlations between reconstruction fidelity metrics and caption quality metrics on Flickr, RefCOCOg, and TRUCE. Points correspond to Stable Diffusion inpainting variants under three masking strategies: cm (hard center mask), gc (Gaussian-blurred center), and ld (low-dimensional center degradation). Caption quality is evaluated using BLIP for Flickr, and Qwen2.5-VL for RefCOCOg and TRUCE.

Figure 5: Layer-wise attention drift and entropy under inpainting on Flickr. Drift increases with depth and is higher for center-masked reconstructions.

Figure 6: UMAP density visualization of ViT CLS embeddings for SD3 inpainting variants on Flickr. Smoother masking strategies preserve closer alignment with original representations.

Figure 7: Cosine similarity between original and reconstructed visual representations across domains.

Figure 8: Spatial localization of attention drift under center inpainting. Drift is consistently higher in reconstructed regions and increases with depth.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Representative examples from each dataset used in this study. These examples illustrate the diversity of visual modalities and caption styles across datasets.

Metric	$r_{\text{full}}$ range	$\boldsymbol{\mu_{\text{LOO}}}$	$\boldsymbol{\sigma_{\text{LOO}}}$	Sign flips
MSE	$[-0.951,-0.906]$	$\approx-0.88$	$\leq 0.19$	0
LPIPS	$[-0.914,-0.857]$	$\approx-0.85$	$\leq 0.11$	0
PSNR	$[0.662,0.761]$	$\approx 0.69$	$\leq 0.13$	0
SSIM	$[-0.166,-0.022]$	$\approx-0.07$	$\geq 0.19$	$\geq 1$

Appendix

Table 1: Leave-one-out (LOO) correlation stability on Flickr. For each reconstruction metric, we report the range of full-data Pearson correlations with caption metrics, the mean and standard deviation of LOO correlations, and the number of sign reversals across all LOO splits.

Method	MSE	PSNR	SSIM	LPIPS
SD1.5-cm	0.0246	16.59	0.582	0.218
SD1.5-gc	0.0231	16.88	0.589	0.204
SD1.5-ld	0.0231	16.86	0.588	0.205
SD2-cm	0.0240	16.68	0.592	0.223
SD2-gc	0.0225	16.98	0.599	0.211
SD2-ld	0.0226	16.94	0.598	0.212

Appendix

Table 2: Pixel-level reconstruction metrics on Flickr. MSE

\downarrow

, PSNR

\uparrow

, SSIM

\uparrow

, and LPIPS

\downarrow

are averaged over 4k samples. Lower MSE/LPIPS and higher PSNR/SSIM indicate better reconstruction fidelity.

BLIP
Dataset	B1	$\%\Delta$	B2	$\%\Delta$	B3	$\%\Delta$	B4	$\%\Delta$	MET.	$\%\Delta$	R-L	$\%\Delta$	sup.	$\%\Delta$	unsup.	$\%\Delta$	SBERT	$\%\Delta$
SD1.5-cm	0.602	-3.68	0.423	-5.79	0.289	-7.67	0.195	-10.14	0.303	-3.81	0.453	-3.63	0.753	-2.09	0.716	-2.16	0.653	-2.83
SD1.5-gc	0.604	-3.36	0.425	-5.35	0.291	-7.01	0.195	-10.14	0.305	-3.17	0.454	-3.40	0.756	-1.69	0.720	-1.64	0.658	-2.08
SD1.5-ld	0.608	-2.72	0.430	-4.23	0.296	-5.43	0.200	-7.83	0.306	-2.86	0.457	-2.81	0.756	-1.69	0.722	-1.37	0.659	-1.93
SD2-cm	0.610	-2.40	0.434	-3.34	0.298	-4.47	0.201	-7.37	0.312	-0.95	0.461	-1.87	0.765	-0.52	0.726	-0.82	0.668	-0.59
SD2-gc	0.610	-2.40	0.431	-3.99	0.296	-5.43	0.201	-7.37	0.309	-1.90	0.460	-2.13	0.764	-0.65	0.727	-0.69	0.669	-0.45

Appendix

Table 3: Captioning performance (absolute and relative) across inpainting variants and original datasets for Flickr. Lexical metrics: BLEU-1–4, METEOR (MET.), ROUGE-L (R-L). Semantic metrics: supervised SimCSE (sup.), unsupervised SimCSE (unsup.), SBERT cosine similarity. Abbreviations — cm: center mask; gc: Gaussian center; ld: low-dimensional mask.

\%\Delta=(\text{Inpainted}-\text{Original})/\text{Original}\times 100

. Negative = performance drop.

Figure 10: Effect of classifier-free guidance scale on reconstruction fidelity. LPIPS (left axis, lower is better) and SSIM (right axis, higher is better) are shown as a function of guidance over 100 samples. Reconstruction quality improves as guidance increases from 5.0 to approximately 7.0-7.5, after which both metrics exhibit increased variance and non-monotonic behavior, indicating diminishing or unstable gains at higher guidance values.

Variant	MSE	PSNR	SSIM	LPIPS
SD1.5-cm	0.03232	16.09	0.583	0.259
SD1.5-gc	0.02918	15.99	0.544	0.275
SD1.5-ld	0.02843	16.48	0.629	0.253
SD2-cm	0.02757	17.05	0.598	0.240
SD2-gc	0.02529	17.11	0.590	0.244
SD2-ld	0.02480	17.45	0.657	0.211

Appendix

Table 4: Pixel-level reconstruction metrics across inpainting variants for RefCOCOg. MSE

\downarrow

, PSNR

\uparrow

, SSIM

\uparrow

, and LPIPS

\downarrow

are averaged over the evaluation set. Lower MSE/LPIPS and higher PSNR/SSIM indicate better reconstruction fidelity.

Variant	BLEU ${}_{\text{avg}}$	METEOR	SBERT
SD1.5-cm	0.516	0.240	0.595
SD1.5-gc	0.572	0.279	0.580
SD1.5-ld	0.594	0.313	0.590
SD2-cm	0.579	0.301	0.603
SD2-gc	0.613	0.304	0.623
SD2-ld	0.614	0.345	0.653

Appendix

Table 5: Captioning performance across inpainting variants for RefCOCOg (QWEN). Reported metrics include BLEU average (BLEU-1–4), METEOR, and SBERT cosine similarity. Higher values indicate better caption quality and semantic alignment.

Variant	MSE	PSNR	SSIM	LPIPS
SD1.5-cm	0.0544	15.24	0.871	0.177
SD1.5-gc	0.0439	15.97	0.879	0.171
SD2-cm	0.0360	17.20	0.892	0.161
SD2-gc	0.0235	19.67	0.925	0.128
SD3-cm	0.0158	20.10	0.945	0.096
SD3-gc	0.0122	21.87	0.951	0.086

Appendix

Table 6: Pixel-level reconstruction metrics across inpainting variants for TRUCE. MSE

\downarrow

, PSNR

\uparrow

, SSIM

\uparrow

, and LPIPS

\downarrow

are averaged over all samples. Lower MSE/LPIPS and higher PSNR/SSIM indicate better reconstruction fidelity.

Variant	B1	B2	B3	B4	MET.	R-L	SBERT
SD1.5-cm	0.275	0.105	0.042	0.0260	0.195	0.190	0.47
SD1.5-gc	0.295	0.115	0.046	0.0280	0.210	0.205	0.48
SD2-cm	0.285	0.110	0.044	0.0270	0.205	0.200	0.49
SD2-gc	0.315	0.125	0.049	0.0300	0.225	0.220	0.52
SD3-cm	0.305	0.120	0.047	0.0290	0.220	0.215	0.51
SD3-gc	0.290	0.108	0.043	0.0265	0.208	0.202	0.50

Appendix

Table 7: Captioning performance on the TRUCE dataset (QWEN). Reported metrics include BLEU-1–4, METEOR (MET.), ROUGE-L (R-L), and SBERT cosine similarity. Abbreviations — cm: center mask; gc: Gaussian center.

Figure 11: Relationship between reconstruction fidelity and captioning performance across Stable Diffusion variants and BLIP model using the ROCOv2 dataset

Variant	MSE	PSNR	SSIM	LPIPS
SD1.5-cm	2217.34	16.74	0.677	0.275
SD1.5-gc	2012.18	17.41	0.690	0.251
SD1.5-ld	2132.21	17.25	0.685	0.256
SD2-cm	1417.41	17.97	0.711	0.209
SD2-gc	1336.09	18.34	0.715	0.195
SD2-ld	1320.51	18.39	0.716	0.195

Appendix

Table 8: Pixel-level reconstruction metrics across inpainting variants for ROCO V2. MSE

\downarrow

, PSNR

\uparrow

, SSIM

\uparrow

, and LPIPS

\downarrow

are averaged over the evaluation set. Lower MSE/LPIPS and higher PSNR/SSIM indicate better reconstruction fidelity.

BLIP
Variant	BLEU ${}_{\text{avg}}$	MET.	R-L	sup.	unsup.	SBERT
SD1.5-cm	0.022	0.041	0.086	0.322	0.250	0.227
SD1.5-gc	0.024	0.044	0.090	0.326	0.257	0.228
SD1.5-ld	0.024	0.043	0.088	0.319	0.253	0.226
SD2-cm	0.026	0.047	0.094	0.366	0.296	0.260
SD2-gc	0.026	0.046	0.094	0.365	0.288	0.251

Appendix

Table 9: Captioning performance across inpainting variants for the ROCO V2 dataset. Reported metrics include BLEU

{}_{\text{avg}}

(average of BLEU-1 and BLEU-2), METEOR (MET.), ROUGE-L (R-L), supervised SimCSE (sup.), unsupervised SimCSE (unsup.), and SBERT cosine similarity.

Figure 12: Relationship between reconstruction fidelity and captioning performance across Stable Diffusion variants and LAION model using the GTZAN dataset

Figure 13: Relationship between reconstruction fidelity and captioning performance across Stable Diffusion variants and BLIP model using the X-ray dataset

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 454 papers

A Language-Informed Loss for Visual Generation
Oct 16, 2026Suchir Agarwal, Keshigeyan Chandrasegaran, Ziyu Chen +4T2I Generation Vision-Language Models
Stanford University
Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
Oct 2, 2026Arun Josephraj Arokiaraj, Chenyi Huang, Zekun Wu +1VLM Robustness Adversarial Attacks on VLMs
Department of Computer Science, University College London · Holistic AI
TextLDM: Language Modeling with Continuous Latent Diffusion
May 8, 2026Jiaxiu Jiang, Jingjing Ren, Wenbo Li +10Diffusion Transformer Latent Diffusion Models
Joy Future Academy · HIT · HKUST(GZ)

Oct 16, 2026J/K move · Enter open · S save

A Language-Informed Loss for Visual Generation

Suchir Agarwal, Keshigeyan Chandrasegaran, Ziyu Chen, Kyle Sargent+3

Stanford University

T2I Generation Vision-Language Models

Open paper AskPDF arXiv

End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced "lily"), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84

\to

53.01) against a pixel-only baseline on the GPIC dataset. Our code, encoders, and generative models are available at https://suchiragarwal.com/lilie.

Abstract

VLM Robustness

Explore similar work

A Language-Informed Loss for Visual Generation

Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally

TextLDM: Language Modeling with Continuous Latent Diffusion