Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusion

Ali Zia; Muhammad Umer Ramzan; Usman Ali; Muhammad Faheem; Abdelwahed Khamis; Shahnawaz Qureshi

cs.CVMar 10, 2026

Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusion

Authors: Ali Zia, Muhammad Umer Ramzan, Usman Ali, Muhammad Faheem, Abdelwahed Khamis, Shahnawaz Qureshi

Organizations: School of Computing, Engineering and Mathematical Sciences La Trobe University. · Gujranwala Institute of Future Technologies(GIFT) University. · Data61, CSIRO. · Sino-Pak Centre for Artificial Intelligence Pak-Austria Fachhochschule Institute of Applied Sciences and Technology.

Abstract

Translating freehand sketches into photorealistic images remains a fundamental challenge in image synthesis, particularly due to the abstract, sparse, and stylistically diverse nature of sketches. Existing approaches, including GAN-based and diffusion-based models, often struggle to reconstruct fine-grained details, maintain spatial alignment, or adapt across different sketch domains. In this paper, we propose a component-aware, self-refining framework for sketch-to-image generation that addresses these challenges through a novel two-stage architecture. A Self-Attention-based Autoencoder Network (SA2N) first captures localised semantic and structural features from component-wise sketch regions, while a Coordinate-Preserving Gated Fusion (CGF) module integrates these into a coherent spatial layout. Finally, a Spatially Adaptive Refinement Revisor (SARR), built on a modified StyleGAN2 backbone, enhances realism and consistency through iterative refinement guided by spatial context. Extensive experiments across both facial (CelebAMask-HQ, CUFSF) and non-facial (Sketchy, ChairsV2, ShoesV2) datasets demonstrate the robustness and generalizability of our method. The proposed framework consistently outperforms state-of-the-art GAN and diffusion models, achieving significant gains in image fidelity, semantic accuracy, and perceptual quality. On CelebAMask-HQ, our model improves over prior methods by 21% (FID), 58% (IS), 41% (KID), and 20% (SSIM). These results, along with higher efficiency and visual coherence across diverse domains, position our approach as a strong candidate for applications in forensics, digital art restoration, and general sketch-based image synthesis.

Figures & tables

Fig. 1 : Illustration of the proposed sketch to image generation architecture. (Top) Component-based Face Representation Learning . The self-attention mechanism is applied to each component of the sketch face to refine feature representations in the autoencoder. (Bottom) CGF based Adversarial Face Generation . The feature descriptors of each facial component are converted into feature maps, and these feature maps undergo the training process in AFIG module. The bidirectional arrow connecting the discriminator represents joint fine-tuning of the trained component encoders during the second training stage. Finally, the generated image passed through a SARR module to fine-tune the details and quality of the generated image using identity-preserving loss.

Panel of Figure 2 — Fig. 2 : Qualitative comparison between our method and state-of-the-art approaches for sketch-to-image translation. Each column represents different sketch inputs and their corresponding outputs, generated by various methods.

Method	FID $\downarrow$	IS $\uparrow$	KID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
CelebAMask-HQ dataset [ 31 ]
ControlNet [ 20 ]	172.93	1.97	135.25	0.55	26.31	0.35
T2I-Adapter [ 21 ]	508.00	1.00	114.76	0.31	16.27	0.48
Pix2PixHD [ 14 ]	172.93	1.28	92.95	0.61	25.31	0.38
CycleGAN [ 36 ]	148.78	1.31	105.99	0.53	28.02	0.28
pSp [ 18 ]	99.16	1.74	73.58	0.66	29.32	0.17

TABLE I : A comparison of the quantitative results of our approach with other baseline techniques across three different datasets. The best scores are highlighted in bold black, while the second-best scores are highlighted in bold blue.

Panel of Figure 4 — Fig. 3 : Zero-shot comparison of DFD and Ground Truth (GT) with our method, trained on the CelebAMask-HQ dataset and tested on diverse sketch types.

Sketch Type	FID $\downarrow$	IS $\uparrow$	KID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
DFD [ 9 ]
Hand Drawn Sketch	191.01	1.70	94.65	0.52	25.768	0.368
Line Sketch	130.88	1.49	38.87	0.67	27.570	0.259
Photoshop Sketch	115.63	1.51	59.15	0.65	28.027	0.278
Ours
Hand Drawn Sketch	131.26	1.63	63.12	0.67	27.558	0.259

TABLE II : The comparison of our framework with DFD across different sketch types highlights the domain invariance of our approach.

Panel of Figure 6 — Fig. 4 : Qualitative results for sketch-to-image translation on three non-facial datasets: Sketchy Database [ 43 ] , ShoesV2, and ChairsV2. Each block shows four rows: Sketches, Ground Truth (GT), Generated Images, and CycleGAN outputs.

Method	FID $\downarrow$	IS $\uparrow$	KID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
Sketchy Database [ 43 ]
ControlNet [ 20 ]	198.32	1.82	152.47	0.53	24.12	0.462
T2I-Adapter [ 21 ]	247.66	1.65	178.13	0.50	22.35	0.507
Pix2PixHD [ 14 ]	172.14	1.91	129.86	0.61	25.87	0.389
CycleGAN [ 36 ]	215.43	1.75	141.73	0.55	25.54	0.378
Ours	131.72	2.70	115.60	0.76	28.38	0.320

TABLE III : The comparison of our framework with other sketch-to-image translation methods on non-facial datasets including the Sketchy database [ 43 ] , QMUL ChairsV2 [ 44 , 45 ] , and QMUL ShoesV2 [ 44 , 45 ] datasets.

Fig. 5 : The pipeline of the Image‑to‑Image comparison module built on FaceNet‑512.

Model	Top-3 Hit Score (%)	LFW Score (%)
Facenet512	94.20	99.65
ArcFace	86.80	99.41
Dlib	84.60	99.38
VGG-Face	88.50	98.78

TABLE IV : Experimental Results of DeepFace Models on the LFW Dataset

	SA	AFIG	GM	SARR	FID $\downarrow$	IS $\uparrow$	KID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
1					115.8407	1.53477	66.75375	0.6691	18.0213	0.2822
2	✓				102.196	1.66641	61.48586	0.7414	17.3119	0.2735
3	✓	✓			91.876	1.9865	53.5971	0.7536	23.5864	0.1466
4	✓	✓		✓	61.5866	1.86954	45.1064	0.7753	29.3125	0.1498
5	✓	✓	✓		71.444	1.7006	46.2568	0.7698	29.6437	0.1549
6	✓			✓	63.2837	1.795	44.6407	0.7665	28.5526	0.1395

TABLE V : Performance of our framework across various evaluation metrics under different ablation experiments.

Panel of Figure 11 — Fig. 6 : Qualitative comparisons of ablation experiments on our architecture.

Method	CelebA	CUFSF	Sketchy	Chairs	Shoes
CycleGAN	0.42	0.39	0.36	0.34	0.33
Pix2PixHD	0.47	0.44	0.40	0.38	0.37
pSp	0.53	0.49	0.46	0.43	0.42
DFD	0.61	0.59	0.55	0.52	0.50
Ours	0.74	0.71	0.69	0.67	0.66

TABLE VI : Mean Opinion Score (MOS) across datasets (higher is better).

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

SketchingReality: From Freehand Scene Sketches To Photorealistic Images
Feb 16, 2026Ahmed Bourouis, Mikhail Bessmeltsev, Yulia GryaditskayaConditional Image Generation Controllable Image Generation
University of Surrey, UK · Universit´e de Montréal, Canada · Adobe Research, UK
SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting
Aug 13, 2026Zian YangDiffusion Models for Image Restoration Diffusion Model Guidance
Fudan University
Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation
Feb 20, 2026Ziyue Liu, Davide Talon, Federico Girella +5Image Generation Conditional Image Generation
University of Verona, 37129 Verona, Italy · Polytechnic Institute of Turin, 10129 Turin, Italy · Fondazione Bruno Kessler, 38123 Povo, Italy +1

Feb 16, 2026J/K move · Enter open · S save

SketchingReality: From Freehand Scene Sketches To Photorealistic Images

Ahmed Bourouis, Mikhail Bessmeltsev, Yulia Gryaditskaya

University of Surrey, UK · Universit´e de Montréal, Canada · Adobe Research, UK

Conditional Image Generation Controllable Image Generation

Open paper AskPDF arXiv

Recent years have witnessed remarkable progress in generative AI, with natural language emerging as the most common conditioning input. As underlying models grow more powerful, researchers are exploring increasingly diverse conditioning signals, such as depth maps, edge maps, camera parameters, and reference images, to give users finer control over generation. Among different modalities, sketches are a natural and long-standing form of human communication, enabling rapid expression of visual concepts. Previous literature has largely focused on edge maps, often misnamed 'sketches', yet algorithms that effectively handle true freehand sketches, with their inherent abstraction and distortions, remain underexplored. We pursue the challenging goal of balancing photorealism with sketch adherence when generating images from freehand input. A key obstacle is the absence of ground-truth, pixel-aligned images: by their nature, freehand sketches do not have a single correct alignment. To address this, we propose a modulation-based approach that prioritizes semantic interpretation of the sketch over strict adherence to individual edge positions. We further introduce a novel loss that enables training on freehand sketches without requiring ground-truth pixel-aligned images. We show that our method outperforms existing approaches in both semantic alignment with freehand sketch inputs and in the realism and overall quality of the generated images.

Abstract

Image Generation

Explore similar work

SketchingReality: From Freehand Scene Sketches To Photorealistic Images

SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting

Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation