Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusion

Ali Zia; Muhammad Umer Ramzan; Usman Ali; Muhammad Faheem; Abdelwahed Khamis; Shahnawaz Qureshi

cs.CVMar 10, 2026

Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusion

Authors: Ali Zia, Muhammad Umer Ramzan, Usman Ali, Muhammad Faheem, Abdelwahed Khamis, Shahnawaz Qureshi

Organizations: School of Computing, Engineering and Mathematical Sciences La Trobe University. · Gujranwala Institute of Future Technologies(GIFT) University. · Data61, CSIRO. · Sino-Pak Centre for Artificial Intelligence Pak-Austria Fachhochschule Institute of Applied Sciences and Technology.

Abstract

Translating freehand sketches into photorealistic images remains a fundamental challenge in image synthesis, particularly due to the abstract, sparse, and stylistically diverse nature of sketches. Existing approaches, including GAN-based and diffusion-based models, often struggle to reconstruct fine-grained details, maintain spatial alignment, or adapt across different sketch domains. In this paper, we propose a component-aware, self-refining framework for sketch-to-image generation that addresses these challenges through a novel two-stage architecture. A Self-Attention-based Autoencoder Network (SA2N) first captures localised semantic and structural features from component-wise sketch regions, while a Coordinate-Preserving Gated Fusion (CGF) module integrates these into a coherent spatial layout. Finally, a Spatially Adaptive Refinement Revisor (SARR), built on a modified StyleGAN2 backbone, enhances realism and consistency through iterative refinement guided by spatial context. Extensive experiments across both facial (CelebAMask-HQ, CUFSF) and non-facial (Sketchy, ChairsV2, ShoesV2) datasets demonstrate the robustness and generalizability of our method. The proposed framework consistently outperforms state-of-the-art GAN and diffusion models, achieving significant gains in image fidelity, semantic accuracy, and perceptual quality. On CelebAMask-HQ, our model improves over prior methods by 21% (FID), 58% (IS), 41% (KID), and 20% (SSIM). These results, along with higher efficiency and visual coherence across diverse domains, position our approach as a strong candidate for applications in forensics, digital art restoration, and general sketch-based image synthesis.

Figures & tables

Fig. 1 : Illustration of the proposed sketch to image generation architecture. (Top) Component-based Face Representation Learning . The self-attention mechanism is applied to each component of the sketch face to refine feature representations in the autoencoder. (Bottom) CGF based Adversarial Face Generation . The feature descriptors of each facial component are converted into feature maps, and these feature maps undergo the training process in AFIG module. The bidirectional arrow connecting the discriminator represents joint fine-tuning of the trained component encoders during the second training stage. Finally, the generated image passed through a SARR module to fine-tune the details and quality of the generated image using identity-preserving loss.

Panel of Figure 2 — Fig. 2 : Qualitative comparison between our method and state-of-the-art approaches for sketch-to-image translation. Each column represents different sketch inputs and their corresponding outputs, generated by various methods.

Method	FID $\downarrow$	IS $\uparrow$	KID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
CelebAMask-HQ dataset [ 31 ]
ControlNet [ 20 ]	172.93	1.97	135.25	0.55	26.31	0.35
T2I-Adapter [ 21 ]	508.00	1.00	114.76	0.31	16.27	0.48
Pix2PixHD [ 14 ]	172.93	1.28	92.95	0.61	25.31	0.38
CycleGAN [ 36 ]	148.78	1.31	105.99	0.53	28.02	0.28
pSp [ 18 ]	99.16	1.74	73.58	0.66	29.32	0.17

TABLE I : A comparison of the quantitative results of our approach with other baseline techniques across three different datasets. The best scores are highlighted in bold black, while the second-best scores are highlighted in bold blue.

Panel of Figure 4 — Fig. 3 : Zero-shot comparison of DFD and Ground Truth (GT) with our method, trained on the CelebAMask-HQ dataset and tested on diverse sketch types.

Sketch Type	FID $\downarrow$	IS $\uparrow$	KID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
DFD [ 9 ]
Hand Drawn Sketch	191.01	1.70	94.65	0.52	25.768	0.368
Line Sketch	130.88	1.49	38.87	0.67	27.570	0.259
Photoshop Sketch	115.63	1.51	59.15	0.65	28.027	0.278
Ours
Hand Drawn Sketch	131.26	1.63	63.12	0.67	27.558	0.259

TABLE II : The comparison of our framework with DFD across different sketch types highlights the domain invariance of our approach.

Panel of Figure 6 — Fig. 4 : Qualitative results for sketch-to-image translation on three non-facial datasets: Sketchy Database [ 43 ] , ShoesV2, and ChairsV2. Each block shows four rows: Sketches, Ground Truth (GT), Generated Images, and CycleGAN outputs.

Method	FID $\downarrow$	IS $\uparrow$	KID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Sketchy Database [ 43 ]
ControlNet [ 20 ]	198.32	1.82	152.47	0.53	24.12	0.462
T2I-Adapter [ 21 ]	247.66	1.65	178.13	0.50	22.35	0.507
Pix2PixHD [ 14 ]	172.14	1.91	129.86	0.61	25.87	0.389
CycleGAN [ 36 ]	215.43	1.75	141.73	0.55	25.54	0.378
Ours	131.72	2.70	115.60	0.76	28.38	0.320

TABLE III : The comparison of our framework with other sketch-to-image translation methods on non-facial datasets including the Sketchy database [ 43 ] , QMUL ChairsV2 [ 44 , 45 ] , and QMUL ShoesV2 [ 44 , 45 ] datasets.

Fig. 5 : The pipeline of the Image‑to‑Image comparison module built on FaceNet‑512.

Model	Top-3 Hit Score (%)	LFW Score (%)
Facenet512	94.20	99.65
ArcFace	86.80	99.41
Dlib	84.60	99.38
VGG-Face	88.50	98.78

TABLE IV : Experimental Results of DeepFace Models on the LFW Dataset

	SA	AFIG	GM	SARR	FID $\downarrow$	IS $\uparrow$	KID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
1					115.8407	1.53477	66.75375	0.6691	18.0213	0.2822
2	✓				102.196	1.66641	61.48586	0.7414	17.3119	0.2735
3	✓	✓			91.876	1.9865	53.5971	0.7536	23.5864	0.1466
4	✓	✓		✓	61.5866	1.86954	45.1064	0.7753	29.3125	0.1498
5	✓	✓	✓		71.444	1.7006	46.2568	0.7698	29.6437	0.1549
6	✓			✓	63.2837	1.795	44.6407	0.7665	28.5526	0.1395

TABLE V : Performance of our framework across various evaluation metrics under different ablation experiments.

Panel of Figure 11 — Fig. 6 : Qualitative comparisons of ablation experiments on our architecture.

Method	CelebA	CUFSF	Sketchy	Chairs	Shoes
CycleGAN	0.42	0.39	0.36	0.34	0.33
Pix2PixHD	0.47	0.44	0.40	0.38	0.37
pSp	0.53	0.49	0.46	0.43	0.42
DFD	0.61	0.59	0.55	0.52	0.50
Ours	0.74	0.71	0.69	0.67	0.66

TABLE VI : Mean Opinion Score (MOS) across datasets (higher is better).

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

Feb 16, 2026

SketchingReality: From Freehand Scene Sketches To Photorealistic Images

Recent years have witnessed remarkable progress in generative AI, with natural language emerging as the most common conditioning input. As underlying models grow more powerful, researchers are exploring increasingly diverse conditioning signals, such as depth maps, edge maps, camera parameters, and reference images, to give users finer control over generation. Among different modalities, sketches are a natural and long-standing form of human communication, enabling rapid expression of visual concepts. Previous literature has largely focused on edge maps, often misnamed 'sketches', yet algorithms that effectively handle true freehand sketches, with their inherent abstraction and distortions, remain underexplored. We pursue the challenging goal of balancing photorealism with sketch adherence when generating images from freehand input. A key obstacle is the absence of ground-truth, pixel-aligned images: by their nature, freehand sketches do not have a single correct alignment. To address this, we propose a modulation-based approach that prioritizes semantic interpretation of the sketch over strict adherence to individual edge positions. We further introduce a novel loss that enables training on freehand sketches without requiring ground-truth pixel-aligned images. We show that our method outperforms existing approaches in both semantic alignment with freehand sketch inputs and in the realism and overall quality of the generated images.

Ahmed Bourouis, Mikhail Bessmeltsev, Yulia Gryaditskaya

University of Surrey, UK · Universit´e de Montréal, Canada · Adobe Research, UK

Conditional Image Generation Controllable Image Generation

Ask PDF

Aug 13, 2026

SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting

Sketch-guided image inpainting provides intuitive structural control, yet real sketches often mix reliable global intent with locally crowded, displaced, incomplete, or deliberately unconventional strokes. Existing approaches typically either retain the input sketch as a fixed condition throughout denoising or refine it into a clean structure before RGB synthesis. The former assumes uniformly reliable strokes and can propagate local errors throughout generation; the latter must resolve ambiguous structure before emerging appearance and semantic context become available. We propose SketchSense, a framework that interprets imperfect sketch guidance by synchronously denoising interacting RGB and structure streams. Bidirectional Attention Fusion couples appearance generation with structural recovery, producing a refined structure that exposes the model's evolving sketch interpretation. A phrase-level objective aligns the semantic grounding of the two streams. Sketch-Aware Spatial Regulation further adapts sketch use to local generation states by modulating attention and the fusion process, while an optional signed prior injects preserve-versus-correct intent into feature representations and attention behavior. Experiments on natural and structurally complex imagery show substantial gains over existing methods in both restoration quality and structural fidelity.

Zian Yang

Fudan University

Diffusion Models for Image Restoration Diffusion Model Guidance Image Inpainting

Ask PDF

Feb 20, 2026

Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation

Sketches offer designers a concise yet expressive medium for early-stage fashion ideation by specifying structure, silhouette, and spatial relationships, while textual descriptions complement sketches to convey material, color, and stylistic details. Effectively combining textual and visual modalities requires adherence to the sketch visual structure when leveraging the guidance of localized attributes from text. We present LOcalized Text and Sketch with multi-level guidance (LOTS), a framework that enhances fashion image generation by combining global sketch guidance with multiple localized sketch-text pairs. LOTS employs a Multi-level Conditioning Stage to independently encode local features within a shared latent space while maintaining global structural coordination. Then, the Diffusion Pair Guidance stage integrates both local and global conditioning via attention-based guidance within the diffusion model's multi-step denoising process. To validate our method, we develop Sketchy, the first fashion dataset where multiple text-sketch pairs are provided per image. Sketchy provides high-quality, clean sketches with a professional look and consistent structure. To assess robustness beyond this setting, we also include an "in the wild" split with non-expert sketches, featuring higher variability and imperfections. Experiments demonstrate that our method strengthens global structural adherence while leveraging richer localized semantic guidance, achieving improvement over state-of-the-art. The dataset, platform, and code are publicly available.

Ziyue Liu, Davide Talon, Federico Girella +5

University of Verona, 37129 Verona, Italy · Polytechnic Institute of Turin, 10129 Turin, Italy · Fondazione Bruno Kessler, 38123 Povo, Italy +1

Image Generation Conditional Image Generation Controllable Image Generation T2I Generation Diffusion-Based Image Generation

Ask PDF

Abstract

Image Generation

Explore similar work

SketchingReality: From Freehand Scene Sketches To Photorealistic Images

SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting

Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation