Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

Ahmed Bourouis; Savas Ozkan; Andrea Maracani; Yi-Zhe Song; Mete Ozay

cs.CVApr 15, 2026

Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

Authors: Ahmed Bourouis, Savas Ozkan, Andrea Maracani, Yi-Zhe Song, Mete Ozay

Organizations: Samsung Research, UK · SketchX, University of Surrey, UK

Abstract

We tackle a new problem: generating geometrically consistent multi-view scenes from a single freehand sketch. Freehand sketches are the most geometrically impoverished input one could offer a multi-view generator. They convey scene intent through abstract strokes while introducing spatial distortions that actively conflict with any consistent 3D interpretation. No prior method attempts this; existing multi-view approaches require photographs or text, while sketch-to-3D methods need multiple views or costly per-scene optimisation. We address three compounding challenges; absent training data, the need for geometric reasoning from distorted 2D input, and cross-view consistency, through three mutually reinforcing contributions: (i) a curated dataset of $\sim$ 9k sketch-to-multiview samples, constructed via an automated generation and filtering pipeline; (ii) Parallel Camera-Aware Attention Adapters (CA3) that inject geometric inductive biases into the video transformer; and (iii) a Sparse Correspondence Supervision Loss (CSL) derived from Structure-from-Motion reconstructions. Our framework synthesizes all views in a single denoising process without requiring reference images, iterative refinement, or per-scene optimization. Our approach significantly outperforms state-of-the-art two-stage baselines, improving realism (FID) by over 60% and geometric consistency (Corr-Acc) by 23%, while providing up to a 3.7 $\times$ inference speedup.

Figures & tables

Figure 2: Method overview. (a) A freehand sketch is encoded then denoised by a video DiT augmented with our CA3 . All $N$ views are generated in a single forward denoising process. (b) Each DiT block contains a self-attention with LoRa and a parallel CA3 to inject relative camera geometry. (c) During training, SfM correspondences from pseudo ground-truth views supervise the CA3 query–key projections via a sparse InfoNCE loss. Dashed boxes: frozen modules.

Figure 3: S2WV dataset generation pipeline. (a) Multi-seed image generation from sketch and caption. (b) Segmentation of sketch and generated images. (c) Best seed selection via mIoU. (d) Multi-view synthesis from the selected image. (e) Dataset curation yields 9,222 samples. More details in Sec. 3.2

Figure 4: SfM correspondences across views. Each correspondence image shows two views with colored lines connecting matched keypoints identified by Structure-from-Motion (COLMAP). The view angles (azimuth, elevation) for each pair are indicated above the correspondence visualizations. These correspondences serve as supervision targets for our camera-aware attention adapters ( Sec. 3.4 ).

Method	#Stages	Time/sample	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	CLIP-I $\uparrow$	Corr-Acc $\uparrow$
SEVA [ 60 ]	2-stage	${\sim}$ 3.1 min	11.310	0.265	0.705	46.34	0.756	0.161
ViewCrafter [ 54 ]	2-stage	${\sim}$ 35 min	11.148	0.338	0.737	48.22	0.773	0.136
Ours	1-stage	${\sim}$ 50 s	12.169	0.302	0.632	18.49	0.828	0.199

Table 1: Quantitative comparison on S2MV test set (477 samples,

N{=}33

views each, 3 seeds). Our single-stage method is compared against two-stage baselines that first translate the sketch to a photograph via FLUX.2-dev.

Figure 5: Qualitative comparison of multi-view generation from sketches. For each example we show the input sketch with its text prompt (left) and eight azimuth views ( $0^{\circ}$ – $315^{\circ}$ at $45^{\circ}$ intervals) generated by each method. GT : ground-truth views; Ours : our single-pass sketch-to-multiview method; VC : ViewCrafter; SEVA : Stable Video Diffusion. Additional comparisons are provided in the supplementary material ( Appendix 0.A ).

	Frontal				Multi-view ( $N{=}33$ )
Method	FID $\downarrow$	CLIP-I $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	FID $\downarrow$	CLIP-I $\uparrow$	Corr-Acc $\uparrow$
FLUX $+$ Qwen (3 seeds)	47.64	0.776	0.731	0.338	51.77	0.804	0.174
Ours	36.33	0.781	0.722	0.353	19.12	0.822	0.191

Table 2: Two-stage FLUX

+

Qwen vs. ours on S2MV. “Frontal” evaluates the single front view (mIoU

=

sketch fidelity); “Multi-view” evaluates all

N

views. Ours wins on both stages despite seeing only a sketch at inference.

	Configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	CLIP-I $\uparrow$	Corr-Acc $\uparrow$	T. params
(a)	Full model (CA3 $+$ CSL $+$ LoRA)	12.169	0.302	0.632	18.49	0.828	0.199	${\sim}$ 41.3M
(b)	w/o CSL ( $\lambda_{\text{corr}}{=}0$ )	12.073	0.287	0.664	20.37	0.817	0.175	${\sim}$ 41.3M
(c)	w/o CA3 (LoRA only)	5.026	0.266	0.819	266.06	0.632	0.183	${\sim}$ 5.9M
(d)	w/o LoRA (CA3 only)	12.211	0.304	0.644	19.27	0.823	0.188	${\sim}$ 35.4M
(e)	w/o frame replication	12.198	0.304	0.652	42.54	0.786	0.174	${\sim}$ 41.3M

Table 3: Ablation study. We remove each contribution independently. Row (a) is the full model. Removing any single component degrades both per-view quality and geometric consistency, confirming that the contributions are mutually reinforcing.

Figure 6: Attention correspondence visualisation. Query pixel (red dot) in the front view; heatmaps show attention at layer 20 over three target viewpoints. Left: with correspondence supervision. Right: without (ablation Tab. 3 row b).

Figure 7: Generalisation to unseen sketch domains. 12 views per sketch at four elevations. (Top) TU-Berlin object sketches [ 11 ] : consistent multi-view output despite a distinctly different sketch style. (Bottom) InkScenes [ 43 ] : robust to dense, textured scene compositions unseen during training.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Additional qualitative comparison. Same format as Fig. 5: input sketch with text prompt (left) and eight azimuth views generated by each method.

Figure 9: Additional qualitative comparison. Same format as Fig. 5: input sketch with text prompt (left) and eight azimuth views generated by each method.

Method	Passes	Time/sample	Speedup
FLUX $+$ SEVA	3 (FLUX $+$ 2 $\times$ SEVA)	${\sim}$ 3.1 min	$3.7\times$ slower
FLUX $+$ ViewCrafter	$N$ (FLUX $+$ $N{-}1$ $\times$ VC)	${\sim}$ 35 min	$42\times$ slower
Ours	1	${\sim}$ 50 s	—

Appendix

Table 4: Inference efficiency. Wall-clock time per sample on a single A100 GPU (

N{=}33

views).

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 190 papers

Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning
Jul 18, 2026Yuqi Zhang, Yadan Luo, Xiangyu Sun +33D Asset Generation 3D Scene Generation
The University of Queensland · Shanghai AI Laboratory · East China Normal University
SketchingReality: From Freehand Scene Sketches To Photorealistic Images
Feb 16, 2026Ahmed Bourouis, Mikhail Bessmeltsev, Yulia GryaditskayaConditional Image Generation Controllable Image Generation
University of Surrey, UK · Universit´e de Montréal, Canada · Adobe Research, UK
Sketch2MinSurf: Vision-Language Guided Generation of Editable Minimal Surfaces from Hand-Drawn Sketches
May 20, 2026Wenda Wang, Anqi Liu, Junqi Yang +43D Shape Generation
School of Architecture, Tsinghua University · Department of Architecture, National University of Singapore

Jul 18, 2026J/K move · Enter open · S save

Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning

Yuqi Zhang, Yadan Luo, Xiangyu Sun, Fengyi Zhang+2

The University of Queensland · Shanghai AI Laboratory · East China Normal University

3D Asset Generation 3D Scene Generation 3D Scene Representation 3D Scene Layout Generation 3D Diffusion Models Image-to-3D Scene Generation

Open paper AskPDF arXiv

High-quality 3D scene assets are critical for embodied applications such as robotic manipulation, navigation, and simulation. Despite their strong object priors, recent single-image 3D generation models such as SAM3D remain insufficient for real-world scenes, where severe occlusions, redundant observations, and cross-view inconsistencies make reliable scene generation challenging. We introduce Scene-SAM3D, a training-free framework that extends SAM3D from single-view object generation to calibrated multi-view scene asset generation. Scene-SAM3D selects a compact set of complementary views, reducing observation redundancy while providing additional evidence for regions occluded in individual views. Based on the selected views, it performs step-efficient latent velocity fusion to integrate multi-view evidence and suppress cross-view conflicts in canonical space. Finally, a lightweight rigid-object Gaussian optimization refines the scene layout within 200 iterations while preserving the generated object geometry. Experiments on Replica and ScanNet++ demonstrate consistent improvements at both instance and scene levels, with our method reducing scene-level CD by 43.8% on Replica and 30.9% on ScanNet++, while cutting flow-model sampling FLOPs and wall-time latency by nearly 20% under the same multi-view setting. Code will be released at https://github.com/xibi777/Scene-SAM3D.

Abstract

3D Scene Generation

Explore similar work

Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning

SketchingReality: From Freehand Scene Sketches To Photorealistic Images

Sketch2MinSurf: Vision-Language Guided Generation of Editable Minimal Surfaces from Hand-Drawn Sketches