SketchingReality: From Freehand Scene Sketches To Photorealistic Images

Ahmed Bourouis; Mikhail Bessmeltsev; Yulia Gryaditskaya

cs.CVFeb 16, 2026

SketchingReality: From Freehand Scene Sketches To Photorealistic Images

Authors: Ahmed Bourouis, Mikhail Bessmeltsev, Yulia Gryaditskaya

Organizations: University of Surrey, UK · Universit´e de Montréal, Canada · Adobe Research, UK

Abstract

Recent years have witnessed remarkable progress in generative AI, with natural language emerging as the most common conditioning input. As underlying models grow more powerful, researchers are exploring increasingly diverse conditioning signals, such as depth maps, edge maps, camera parameters, and reference images, to give users finer control over generation. Among different modalities, sketches are a natural and long-standing form of human communication, enabling rapid expression of visual concepts. Previous literature has largely focused on edge maps, often misnamed 'sketches', yet algorithms that effectively handle true freehand sketches, with their inherent abstraction and distortions, remain underexplored. We pursue the challenging goal of balancing photorealism with sketch adherence when generating images from freehand input. A key obstacle is the absence of ground-truth, pixel-aligned images: by their nature, freehand sketches do not have a single correct alignment. To address this, we propose a modulation-based approach that prioritizes semantic interpretation of the sketch over strict adherence to individual edge positions. We further introduce a novel loss that enables training on freehand sketches without requiring ground-truth pixel-aligned images. We show that our method outperforms existing approaches in both semantic alignment with freehand sketch inputs and in the realism and overall quality of the generated images.

Figures & tables

Figure 2: Method overview. Modulation network (a). During training, the latent features $z_{t}$ are generated by applying a noising process to the clean latent $z_{0}$ , which is obtained by encoding the ground-truth image using the VAE encoder. During inference, $z_{t}$ is instead sampled from a standard Gaussian distribution and then progressively denoised by the diffusion model. Text-conditioned diffusion model generates time-dependent noise $\epsilon_{t}$ , which we modulate relying on semantic sketch features. For details, please refer to Sec. 3.2 and Tab. 4 . Attention supervision (b). Attention supervision allows us to train on a combination of freehand sketches and sketches algorithmically generated from reference images. It bypasses the need for pixel-aligned ground-truth images — which are unavailable for freehand sketches — and helps the modulation network focus on sketch semantics. For details, please refer to Sec. 3.4 .

Method	Setup	FID $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$
	Zero-shot	135.595	1.136	0.773
	$\mathcal{L}_{\text{noise}}$ only (syn)	136.821	1.141	0.771
	$\mathcal{L}_{\text{noise}}$ only (free)	139.872	1.042	0.789
CntrlNet SD2.1 ( Zhang et al., 2023 )	$\mathcal{L}_{\text{noise}}$ + $\mathcal{L}_{\text{attn}}$	135.891	1.196	0.768
	Zero-shot	144.329	-0.203	0.813
	$\mathcal{L}_{\text{noise}}$ only	138.479	0.318	0.779

Table 1: Comparison on 475 test sketches from the FS-COCO dataset ( Chowdhury et al., 2022 ) with baseline methods with different fine-tuning setups. The used metrics are summarized in Sec. 4.3 .

Figure 3: Visual comparison between our method and baselines in a zero-shot setting (using weights of pretrained models) and with the ones fine-tuned with the proposed attention loss on a mix of freehand and synthetic sketches.

Figure 4: Quantitative evaluation of the role of sketch representation. Please refer to Sec. 4.5.1 for the details.

Figure 5: Qualitative and quantitative evaluation when we (i) remove the attention loss $\mathcal{L}_{\text{attn}}$ ( Eq. 5 ), ‘WO attention loss’, and (ii) when we remove $\mathcal{L}_{\text{var}}$ ( Eq. 3 ), ‘WO variance loss’. Please refer to Sec. 4.5.2 for the detailed discussion. Captions are taken as-is from the FSCOCO dataset.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative results comparison between our method and baselines. Please see Sec. 4.4.2 for the detailed discussion. Sketch and text are both passed in as inputs.

Figure 7: Qualitative results comparison between our method and baselines. Please see Sec. 4.4.2 for the detailed discussion. Sketch and text are both passed in as inputs.

Baseline	Photorealism (%)			Sketch similarity (%)			Best trade-off (%)
Baseline	Ours	Base	Und.	Ours	Base	Und.	Ours	Base	Und.
SDXL T2I s=0.8 $\tau$ =0.2	47.2	34.9	17.9	46.4	37.1	16.5	66.0	20.6	13.4
SDXL T2I s=0.8 $\tau$ =0.4	47.4	39.2	13.4	55.0	35.8	9.2	71.0	22.0	7.0
ControlNet SDXL	40.4	45.2	14.4	54.0	37.0	9.0	54.4	32.2	13.3
ControlNet SD2	70.0	21.2	8.8	64.8	30.8	4.4	88.8	10.0	1.2

Appendix

Table 2: Pairwise user study preferences. Please see Sec. A.2 for the details.

	Method	Setup	FID $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$
#1	CntrlNet SDXL ( Zhang et al., 2023 )	Zero-shot	174.462	0.027	0.825
#2		ControlNet branch LoRA	177.243	0.046	0.813
#3		ControlNet branch LoRA + main branch LoRA	189.566	0.043	0.779
#4	Ours SD2.1	Modulation network training	121.973	1.291	0.739
#5	Ours SDXL	Modulation network training	117.025	1.331	0.708
#6	Ours SDXL	Modulation network training + backbone LoRA fine-tuning	138.213	1.162	0.759

Appendix

Table 3: Comparison on 475 test sketches from the FS-COCO dataset ( Chowdhury et al., 2022 ) , averaged over three seeds. (#4) Ours SD2.1 is the main model used throughout the paper, where we train only the modulation network and the last three layers of the sketch encoder with all the proposed losses as described in the main paper. All remaining variants use the SDXL backbone. (#5) Ours SDXL (Modulation network training) differs from (#4) Ours SD2.1 (Modulation network training) only in the backbone, while the training setup remains the same. (#6) Ours SDXL (Modulation network training+backbone LoRA fine-tuning) adds additional LoRA fine-tuning of the backbone SDXL model. For ControlNet SDXL, LoRA fine-tuning is applied either to the ControlNet branch alone (#2) or to both the ControlNet and main diffusion branches (#3). The used metrics are summarized in Sec. 4.3 . Analysis is provided in Sec. A.3 .

Figure 8: Qualitative comparison of our method using the SD2.1 backbone, the SDXL backbone, and the SDXL backbone with LoRA fine-tuning of the backbone. We also compare against ControlNet (SDXL backbone) in three configurations: zero-shot, LoRA applied to the control branch, and LoRA applied to both the diffusion and control branches. All methods are conditioned on the input sketches and textual captions shown in the figure. The results are shown for two different seeds.

Figure 9: Qualitative results comparison between our method and Flux.1 Kontext [dev]. Our method is conditioned on sketches and textual captions, shown in the figure. We prompted the FluxKontext model with “Convert a sketch to a photorealistic image of …” , where instead of “…” we use the same prompt as we pass to our model. The results are shown for two different seeds.

Figure 10: A visual comparison between our method and a specialized single object sketch-based image generation method Koley et al. (2024) on the sketches from the QuickDraw dataset Jongejan et al. (2017) .

Figure 11: A visual comparison between our method and a public implementation Chen (2023) of the method that combines ControlNet with paint-with-words guidance Balaji et al. (2022) .

Path/Block	Layer/Operation	Input Ch.	Output Ch.	Spatial Change
Sketch Path	DoubleConv	512	256	-
	DoubleConv	256	128	-
	Interpolate (bilinear)	128	128	Match latent/noise size
Noise $\epsilon$ Path	DoubleConv	4	16	-
	MaxPool2d (2)	16	16	/2
	DoubleConv	16	32	-

Appendix

Table 4: Architecture of the proposed modulation network.

Figure 12: Parameter preferences by model by question.

Figure 13: Visualization of image generated with ControlNet SD2 for different values of control parameter. Each row shows results for a specific sketch and text prompt using two random seeds. Text captions are shown below each sketch group.

Figure 14: Visualization of image generated with ControlNet SDXL for different values of control parameter. Each row shows results for a specific sketch and text prompt using two random seeds. Text captions are shown below each sketch group.

Figure 15: Visualization of image generated with T2I with SDXL model for different values of control parameters. Each row shows results for a specific sketch and text prompt, generated using two random seeds. Text captions are shown below each sketch group.

Figure 16: Visualization of image generated with ControlNext for different values of control parameter. Each row shows results for a specific sketch and text prompt using two random seeds. Text captions are shown below each sketch group.

Baseline	Param	FID $\downarrow$	$CLIP_{sim}\uparrow$	LPIPS $\downarrow$
	w0.2	141.46	0.911	0.780
	w0.4	138.62	0.927	0.774
	s0.6	135.60	1.136	0.773
	w0.8	137.35	0.765	0.773
	w1.0	143.42	0.668	0.786
ControlNet SD2	w1.2	147.53	0.468	0.791

Appendix

Table 5: Detailed performance comparison of all baseline models with different parameters. The settings highlighted in bold are the versions selected for the main paper.

Method	FID $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$
Freehand only	142.27	0.955	0.750
Ours (full)	121.973	1.291	0.739

Appendix

Table 6: Ablation: Comparing training on freehand sketches alone versus training on a combination of freehand and synthetic sketches, with evaluation performed on freehand sketches from the FS-COCO dataset Chowdhury et al. (2022) .

Figure 17: Qualitative results comparison between our method, our method trained on only synthetic sketches, Controlnet, and T2I adapter

Method	FID $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$
ControlNet SD2.1	124.57	1.247	0.458
ControlNet SDXL	112.69	1.286	0.432
T2I-Adapter SDXL	102.43	1.383	0.379
Ours (synthetic)	106.74	1.280	0.423
Ours (full)	109.08	1.193	0.484

Appendix

Table 7: Quantitative comparison of our method trained on synthetic sketches ( Ours (synthetic) ) against baselines trained similarly, evaluated on synthetic sketches, as the ones shown in Fig. 17 . The best results are highlighted in bold, and the second-best results are underlined.

Figure 18: Qualitative comparison between our method, our method trained on only synthetic sketches, ControlNet, and T2I Adapter tested on freehand sketches.

Method	FID $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$
ControlNet SD2.1	135.59	1.136	0.773
ControlNet SDXL	174.46	0.027	0.825
T2I-Adapter SDXL	159.82	0.213	0.819
Ours (synthetic)	132.17	0.817	0.793
Ours (full)	121.97	1.291	0.739

Appendix

Table 8: Quantitative comparison of our method trained on synthetic sketches only ( Ours (synthetic) ) with edge-conditioned baselines tested on freehand sketches

Figure 19: Qualitative and quantitative evaluation when training with $10\%$ , $20\%$ , $30\%$ , $40\%$ , $50\%$ and $100\%$ of timesteps corresponding to high noise regimes. Modulation is performed for the same number of time steps as the respective training. Please refer to Sec. D.3 for the detailed discussion.

Method	Setup	FID $\downarrow$	CLIP $\uparrow$	LPIPS $\downarrow$
	Zero-shot	135.595	1.136	0.773
	$\mathcal{L}_{\text{noise}}$ only (syn)	136.821	1.141	0.771
	$\mathcal{L}_{\text{noise}}$ only (free)	139.872	1.042	0.789
	$\mathcal{L}_{\text{noise}}$ + $\mathcal{L}_{\text{attn}}$	135.891	1.196	0.768
CntrlNet SD2.1 ( Zhang et al., 2023 )	$\mathcal{L}_{\text{noise}}$ + $\mathcal{L}_{\text{attn}}$ + Mod.	134.771	1.192	0.767
Ours	Full	121.973	1.291	0.739

Appendix

Table 9: Comparison on 475 test sketches from the FS-COCO dataset ( Chowdhury et al., 2022 ) with ControlNet versions.

Conditional Image Generation

Conditional image generation creates images guided by inputs such as text, sketches, layouts, masks, reference images, or physical measurements.

See all 160 papers

Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusion
Mar 10, 2026Ali Zia, Muhammad Umer Ramzan, Usman Ali +3Image Generation Conditional Image Generation
School of Computing, Engineering and Mathematical Sciences La Trobe University. · Gujranwala Institute of Future Technologies(GIFT) University. · Data61, CSIRO. +1
Geometrically Consistent Multi-View Scene Generation from Freehand Sketches
Apr 15, 2026Ahmed Bourouis, Savas Ozkan, Andrea Maracani +23D Scene Generation Multi-View Consistency
Samsung Research, UK · SketchX, University of Surrey, UK
SketchFlow: Zero-Shot Vector Sketch Generation via GMM Prior Flow in CLIP Latent Space
Aug 21, 2026Jin Zhou, Hongliang Yang, Pengfei Xu +1Flow-Based Generative Modeling Image Generation
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, CSSE, Shenzhen University, Shenzhen, China

Mar 10, 2026J/K move · Enter open · S save

Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusion

Ali Zia, Muhammad Umer Ramzan, Usman Ali, Muhammad Faheem+2

School of Computing, Engineering and Mathematical Sciences La Trobe University. · Gujranwala Institute of Future Technologies(GIFT) University. · Data61, CSIRO.+1

Image Generation Conditional Image Generation Generative Adversarial Networks

Open paper AskPDF arXiv

Translating freehand sketches into photorealistic images remains a fundamental challenge in image synthesis, particularly due to the abstract, sparse, and stylistically diverse nature of sketches. Existing approaches, including GAN-based and diffusion-based models, often struggle to reconstruct fine-grained details, maintain spatial alignment, or adapt across different sketch domains. In this paper, we propose a component-aware, self-refining framework for sketch-to-image generation that addresses these challenges through a novel two-stage architecture. A Self-Attention-based Autoencoder Network (SA2N) first captures localised semantic and structural features from component-wise sketch regions, while a Coordinate-Preserving Gated Fusion (CGF) module integrates these into a coherent spatial layout. Finally, a Spatially Adaptive Refinement Revisor (SARR), built on a modified StyleGAN2 backbone, enhances realism and consistency through iterative refinement guided by spatial context. Extensive experiments across both facial (CelebAMask-HQ, CUFSF) and non-facial (Sketchy, ChairsV2, ShoesV2) datasets demonstrate the robustness and generalizability of our method. The proposed framework consistently outperforms state-of-the-art GAN and diffusion models, achieving significant gains in image fidelity, semantic accuracy, and perceptual quality. On CelebAMask-HQ, our model improves over prior methods by 21% (FID), 58% (IS), 41% (KID), and 20% (SSIM). These results, along with higher efficiency and visual coherence across diverse domains, position our approach as a strong candidate for applications in forensics, digital art restoration, and general sketch-based image synthesis.

Abstract

Conditional Image Generation

Explore similar work

Component-Aware Sketch-to-Image Generation Using Self-Attention Encoding and Coordinate-Preserving Fusion

Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

SketchFlow: Zero-Shot Vector Sketch Generation via GMM Prior Flow in CLIP Latent Space