PixARG: Pixel-Asynchronous Retrieval-Augmented Diffusion for Fast and Realistic Image Generation

Zewei Xin; Qinya Li; Chaoyue Niu; Fan Wu

cs.CVOct 10, 2026

PixARG: Pixel-Asynchronous Retrieval-Augmented Diffusion for Fast and Realistic Image Generation

Authors: Zewei Xin, Qinya Li, Chaoyue Niu, Fan Wu

Organizations: Shanghai Jiao Tong University, Shanghai, China

Abstract

While diffusion models achieve impressive image generation capabilities, they suffer from high computational costs, implausible artifacts, and limited adaptability to dynamic or private data due to static, centralized training. To address these, we propose PixARG, a novel retrieval-augmented generative framework that integrates dynamic real-world image retrieval into the diffusion pipeline to simultaneously enhance efficiency and visual realism. Specifically, PixARG constructs a pixel-level affinity map between the user's text prompt and the retrieved image to guide a heterogeneous noise initialization. By modulating noise intensity across spatial regions, this mechanism preserves semantically relevant structural priors while perturbing less relevant areas, thereby significantly accelerating convergence. To accommodate this spatially non-uniform noise and ensure synchronized generation, we introduce a pixel-wise asynchronous denoising scheduler and a spatiotemporal denoiser conditioned on the affinity map. Furthermore, we derive an affinity-adaptive training objective that elegantly balances deterministic restoration with stochastic regeneration across pixels. Experiments on LAION and COYO datasets across offline, online, and user-specific settings show PixARG enhances realism and semantic alignment while drastically reducing diffusion steps.

Figures & tables

Figure 1. Comparison among diffusion methods. Traditional diffusion generates images from pure noise, and feature-level RAG diffusion incorporates retrieved image features into the denoising network to support new concepts. In contrast, our pixel-level RAG retrieves prompt-relevant pixel information from real images and injects it into the initial noise. This introduces realistic texture, structure and other low-level cues to enhance fidelity, while also accelerates generation through an information-enriched initialization that shortens the effective denoising trajectory. Comparison among diffusion methods. Traditional diffusion generates images from pure noise, and feature-level RAG diffusion incorporates retrieved image features into the denoising network to support new concepts. In contrast, our pixel-level RAG retrieves prompt-relevant pixel information from real images and injects it into the initial noise. This introduces realistic texture, structure and other low-level cues to enhance fidelity, while also accelerates generation through an information-enriched initialization that shortens the effective denoising trajectory.

Figure 2. Overview of PixARG. Based on the semantic relevance between the retrieved image $v$ and the text prompt, PixARG selectively applies spatially heterogeneous noise to obtain a visual sketch $S_{T}$ as the initial latent for generation. Text-aligned regions are perturbed with weaker noise to preserve structural details, while less relevant areas receive stronger noise to encourage regeneration. During sampling, pixel-wise asynchronous denoising allows pixels with different initial noise levels to denoise at adaptive rates, completing the generation within the same steps.

With DDIM ( Song et al., 2020 ) Sampler
Method	Diffusion Steps↓	Time (s/img)↓	Basic Metrics		Realism & Diversity		RichHF-18K				Human Preference Metrics
Method	Diffusion Steps↓	Time (s/img)↓	CLIP↑	FID↓	Precision(%)↑	Recall(%)↑	Plausibility↑	Alignment↑	Aesthetics↑	Overall↑	HPSv2↑	MPS↑	PickScore↑
SD 1.4 ( Rombach et al., 2022 )	50	0.883	33.10	11.49	57.07	64.73	0.6892	0.6079	0.6753	0.5828	24.00	9.58	19.85
ReDi ( Zhang et al., 2023 )	40	0.707	26.71	13.87	58.71	44.46	0.6589	0.5978	0.6488	0.5646	21.22	8.23	18.84
DiT4Edit ( Feng et al., 2025 )	50	2.767	30.92	9.44	64.34	59.65	0.6800	0.6008	0.6586	0.5739	21.86	9.18	19.47
RDM ( Blattmann et al., 2022 )	50	0.890	32.98	11.31	58.40	61.30	0.6871	0.6075	0.6724	0.5815	24.01	9.54	19.82

Table 1. Quantitative results on the SD 1.4 backbone under DDIM and PNDM samplers on the LAION dataset. Inference time measures the total duration from prompt to image, encompassing all stages such as retrieval, encoding, and generation.

Method	Steps↓	Basic Metrics		Realism & Diversity		RichHF-18K				Human Preference Metrics
Method	Steps↓	CLIP↑	FID↓	Precision(%)↑	Recall(%)↑	Plausibility↑	Alignment↑	Aesthetics↑	Overall↑	HPSv2↑	MPS↑	PickScore↑
SD 1.4 ( Rombach et al., 2022 )	50	33.77	9.33	64.35	70.44	0.5994	0.5502	0.6028	0.5189	22.90	8.57	19.58
ReDi ( Zhang et al., 2023 )	40	27.63	14.57	61.86	49.29	0.5398	0.5337	0.5582	0.4860	18.08	6.34	17.96
DiT4Edit ( Feng et al., 2025 )	50	28.15	9.20	68.11	64.89	0.5953	0.5504	0.5998	0.5168	19.11	7.14	18.45
RDM ( Blattmann et al., 2022 )	50	33.23	11.90	63.21	69.83	0.6073	0.5501	0.6059	0.5218	22.82	8.40	19.47
KNN-Diffusion ( Sheynin et al., 2022 )	50	33.16	10.99	62.52	69.03	0.6039	0.5480	0.6043	0.5195	21.40	8.29	19.35

Table 2. Quantitative comparison on the SD 1.4 backbone under DDIM sampler on the COYO dataset.

Figure 3. Performance comparison of diffusion models with varying parameter scales. Combined with PixARG, the smaller SD 1.4 outperforms SDXL Turbo.

Panel of Figure 4 — Figure 4. Our PixARG achieves consistent performance gains across all retrieval similarity intervals, with metrics evaluating alignment, human preference, and realism.

	Basic Metrics		Realism & Diversity		RichHF-18K
	CLIP↑	FID↓	Precision(%)↑	Recall(%)↑	Plausibility↑	Alignment↑	Aesthetics↑	Overall↑
Ret.	32.34	4.40	76.68	69.24	0.7145	0.6017	0.6700	0.5887
Sync.	33.30	8.64	69.31	59.49	0.6869	0.6030	0.6637	0.5783
MSE	33.38	6.94	66.84	61.30	0.6958	0.6093	0.6734	0.5852
Ours	33.43	6.78	69.01	61.35	0.6964	0.6103	0.6726	0.5860
Rank-2	33.35	6.98	69.13	59.78	0.6973	0.6098	0.6722	0.5859

Table 3. Ablation study under the DDIM sampler on the LAION dataset, using 15 diffusion steps for all settings. “Ret.” denotes the Rank-1 retrieved image without diffusion; “Sync.” adds synchronous denoising; “MSE” replaces the reconstruction objective with MSE loss; and “Rank-

k

” variants take the

k

-th retrieved image instead of the default Rank-1 image.

Panel of Figure 5 — Figure 5. Ablation study on the retrieval ratio under DDIM sampler on a 5k subset of LAION.

Figure 6. Online evaluation results using Google Image Search as the retrieval backend under the DDIM sampler.

User Name	Pairs	Method	Basic	Realism & Diversity		RichHF-18K
User Name	( $N$ )	Method	CLIP↑	Precision(%)↑	Reccall(%)↑	Plausibility↑	Alignment↑	Aesthetics↑	Overall↑
		SD 1.4	31.19	45.66	78.61	0.6227	0.5666	0.6001	0.5399
earthmoonsun	1736	PixARG (Ours)	31.52	58.96	81.50	0.6303	0.5676	0.5967	0.5411
		SD 1.4	32.80	50.23	76.92	0.6480	0.5421	0.6200	0.5384
ethan_kahn	1109	PixARG (Ours)	33.01	70.59	77.38	0.6609	0.5435	0.6305	0.5431
		SD 1.4	28.81	62.50	71.88	0.6429	0.6177	0.6282	0.5621

Table 4. Performance on user-specific datasets.

N

denotes the total image-text pairs per user, where 20% are used as query prompts and 80% constitute the retrieval database.

Backbone	Steps	Basic Metrics		Realism & Diversity		RichHF-18K
Backbone	Steps	CLIP↑	FID↓	Precision(%)↑	Recall(%)↑	Overall↑
PixArt- $\alpha$ ( Chen et al., 2023 )	15	31.55	18.75	48.16	46.73	0.6183
+ Ours	8	31.63	12.58	57.86	54.72	0.6037
SDXL ( Podell et al., 2023 )	50	32.24	14.22	65.33	62.75	0.5248
+ Ours	15	32.32	7.31	68.29	68.70	0.5287

Table 5. Evaluation on other diffusion backbones. “+ Ours” indicates the integration of PixARG.

Figure 7. Visualization of retrieval items and attention maps on LAION dataset ( Schuhmann et al., 2022 ) . The text-visual affinity maps are highly consistent with the model’s attention maps, providing effective visual prior for efficient generation.

Figure 8. Visual comparison with other methods under the same settings as Tab. 1 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9. Distribution of the mean CLIP similarity over the top 10% pixels in each image–text similarity map on a 5k subset of the LAION dataset.

Panel of Figure 10 — Figure 10. Normalization function $f(\cdot)$ under various slope $k$ and center $c$ . The black line indicates that $f(m)=m$ .

Panel of Figure 11 — Figure 11. Ablation on the slope parameter $k$ of the normalization function $f(\cdot)$ on a 5k subset of the LAION dataset.

Panel of Figure 12 — Figure 12. Ablation on the center parameter $c$ of the normalization function $f(\cdot)$ on a 5k subset of the LAION dataset.

Retrieval + Img2Img: generation-only img2img baselines using the same retrieved image input as PixARG.
Method	Steps $\downarrow$	Basic Metrics		Realism & Diversity		RichHF-18K		Human Preference Metrics
Method	Steps $\downarrow$	CLIP $\uparrow$	FID $\downarrow$	Precision $\uparrow$	Recall $\uparrow$	Plausibility $\uparrow$	Overall $\uparrow$	HPSv2 $\uparrow$	MPS $\uparrow$	PickScore $\uparrow$
SD 1.4	50	33.10	11.49	57.07	64.73	0.6892	0.5828	24.00	9.58	19.85
RePaint ( Lugmayr et al., 2022 )	50	30.02	10.61	73.05	45.31	0.6200	0.5453	22.21	9.18	19.48
LEDITS++ ( Brack et al., 2024 )	50	32.72	7.15	69.85	60.54	0.6898	0.5846	23.70	9.73	19.93
SDEdit (Tab. 3, Sync.) ( Meng et al., 2021 )	15	33.30	8.64	69.31	59.49	0.6869	0.5783	23.99	9.82	19.94

Appendix

Table 6. Comparison of PixARG with representative retrieval- and generation-oriented methods under retrieval-augmented diffusion settings using the SD 1.4 backbone on the LAION dataset. DPM++ denotes DPM-Solver++.

Method	LPIPS $\uparrow$	DINO $\downarrow$	CLIP-I $\downarrow$	CLIP-T $\uparrow$	FID $\downarrow$
Text2Img	0.778	0.516	0.801	33.10	11.49
Ret.+Img2Img	0.456	0.718	0.886	33.30	8.64
PixARG (Ours)	0.650	0.588	0.838	33.43	6.78

Appendix

Table 7. Comparison of retrieval dependency and generation quality between PixARG and Ret.+Img2Img using the SD 1.4 backbone. LPIPS, DINO, and CLIP-I measure the similarity between generated and retrieved images, where higher LPIPS and lower DINO/CLIP-I indicate reduced retrieval dependency. CLIP-T and FID evaluate generation quality, where higher CLIP-T and lower FID indicate better generation performance.

Method	Step	Basic Metrics		Realism & Diversity		RichHF-18K		Human Preference Metrics
Method	Step	CLIP $\uparrow$	FID $\downarrow$	Precision $\uparrow$	Recall $\uparrow$	Plausibility $\uparrow$	Overall $\uparrow$	HPSv2 $\uparrow$	MPS $\uparrow$	PickScore $\uparrow$
SD 1.4	10	32.69	13.52	58.93	57.13	0.6660	0.5687	22.39	9.14	19.48
PixARG	10	33.36	8.40	69.99	53.43	0.6886	0.5793	23.97	9.80	19.92
SD 1.4	20	33.03	11.84	56.88	62.65	0.6796	0.5778	23.60	9.49	19.76
PixARG	20	33.40	6.83	67.81	60.95	0.7057	0.5924	24.59	10.00	20.10
SD 1.4	30	33.07	11.85	56.21	64.45	0.6817	0.5786	23.83	9.51	19.79

Appendix

Table 8. Step-matched comparison between PixARG and SD 1.4 on LAION with different denoising steps.

Figure 13. Ablation of retrieval-guided initialization priors on a 5K LAION subset.

Figure 14. Visualization of PixARG’s retrieval and generation results and the corresponding signal-to-noise ratio. The generated results inherit the layout and structural details contained in the affinity-highlighted regions of the retrieved samples while creatively reinterpreting them, maintaining flexibility without losing the underlying structural guidance. Owing to the heterogeneous noise design, each pixel receives its own initial noise level and denoising speed, leading to pixel-wise signal-to-noise ratios that span a certain range.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 461 papers

AsyncPatch Diffusion: spatially-flexible image generation
Jun 5, 2026Samuele Papa, Valentin De Bortoli, Guillaume Couairon +3Diffusion Model Guidance Image Generation
Google DeepMind · University of Amsterdam · The Netherlands Cancer Institute
Parallel Rollout Approximation for Pixel-Space Autoregressive Image Generation
Jun 26, 2026Jiayi Xu, Di He, Guolin KeConditional Image Generation Autoregressive Image Generation
Peking University · DP Technology
LADR: Locality-Aware Dynamic Rescue for Efficient Text-to-Image Generation with Diffusion Large Language Models
Mar 13, 2026Chenglin Wang, Yucheng Zhou, Shawn Chen +2T2I Generation Diffusion Language Model Inference
East China Normal University · University of Macau · Zhejiang University +1

Jun 5, 2026J/K move · Enter open · S save

AsyncPatch Diffusion: spatially-flexible image generation

Samuele Papa, Valentin De Bortoli, Guillaume Couairon, Daniel Sýkora+2

Google DeepMind · University of Amsterdam · The Netherlands Cancer Institute

Diffusion Model Guidance Image Generation Diffusion-Based Image Generation Diffusion Models Diffusion-Based Denoising

Open paper AskPDF arXiv

Standard diffusion models corrupt an entire sample with a single shared noise level, forcing all spatial regions to follow the same denoising trajectory. We introduce AsyncPatch Diffusion, a joint-diffusion framework that assigns distinct noise levels to different input dimensions, such as image pixels, or latent tokens. We show how this asynchronous corruption defines a valid generative process while supporting a richer family of spatially heterogeneous denoising trajectories, and prove the first valid ELBO for this process. We show that a single pretrained model can perform spatially adaptive generation, where different regions are denoised on different schedules. A key challenge is training: naive independent noise-level sampling overemphasizes highly heterogeneous configurations and underrepresents homogeneous noise levels, that are crucial during sampling. We address this with a controlled noise-level sampler that regulates both the average corruption level and its spatial variability. AsyncPatch achieves generation quality comparable to conventional diffusion on ImageNet 256 and LSUN, while being natively suited for inpainting without task-specific fine-tuning. We further introduce input guidance, which uses clean or partially corrupted regions to guide the generation of unknown regions, improving local consistency and texture matching. Finally, we demonstrate adaptive generation strategies including uncertainty-guided acceleration and autoregressive sampling.

Abstract

Image Generation

Explore similar work

AsyncPatch Diffusion: spatially-flexible image generation

Parallel Rollout Approximation for Pixel-Space Autoregressive Image Generation

LADR: Locality-Aware Dynamic Rescue for Efficient Text-to-Image Generation with Diffusion Large Language Models