Existing sketch generation methods often rely on image guidance and are largely limited to individual objects or simplified scenes, leaving text-guided complex scene sketching underexplored. We present SketchDirector, an agentic framework that mimics the human sketching workflow for automatic multi-concept scene sketching. Given a scene-level text prompt, SketchDirector first employs a large language model~(LLM) to decompose the scene into multiple draft assets, where each is represented by a set of Bézier curves. Then, an agentic reasoning loop iteratively plans and executes their spatial arrangement, guided by self-evolving memory. Finally, a detail refinement module leverages diffusion priors to enhance sketch quality while preserving geometric structure. To support future research and applications in multi-concept scene sketch generation, we further introduce the MCSketch8K dataset, comprising editable vector assets, compositional layouts, and object-level manipulation annotations. Extensive experiments show that SketchDirector outperforms state-of-the-art baselines in semantic and compositional quality while effectively preserving overall contour organization during refinement.
Figures & tables
Figure 2: Motivation of multi-concept scene sketch generation. (A) Widely used in animation, concept art, and design. (B) Artists adjust sketch refinement from draft to detailed sketch preserving structure for different goals.
Figure 3: SketchDirector pipeline . Given a scene description, SketchDirector first generates object-level draft assets. The agent then progressively composes the scene through executable operations from the action library, maintaining scene programs and spatial asset attributes. A closed-loop process iteratively adjusts the layout, supported by self-evolving memory that incorporates short-term failure feedback and long-term experiences. Finally, detail refinement employs curve contour resampling and progressive joint optimization to enhance sketch details while preserving the established scene layout and object structures.
Figure 4: Example agentic reasoning trajectory in SketchDirector . A real log illustrating how agents interleave layout planning, scene state updates, geometric-semantic verification, memory evolution, and correction.
Figure 5: A success example of planning and execution in SketchDirector . Each operation updates both the executable program and the represented scene state, yielding a traceable trajectory from the initial drafts to the final layout.
Figure 6: Qualitative comparison with existing methods. Compared with LLM-based methods , our sketch generation produces scene sketches with more explicit object organization, coherent layouts, and better visual-semantic alignment. Moreover, compared with optimization-based methods , our refinement improves sketch details while faithfully preserving the object layouts and spatial relationships in the draft.
Figure 7: Scene sketch diversity generated by SketchDirector. Our object-level reasoning enables flexible scene layouts from textual descriptions, by producing diverse spatial arrangements (the top row) and object configurations under similar semantic descriptions (the bottom row).
Method
Sem. ↑
Compo. ↑
Compl. ↑
Aes. ↑
Size ↑
Time ↓
SketchAgent
0.742
0.598
0.589
0.714
0.693
∼ 20s
OpenAI GPT-5.5
0.911
0.730
0.811
0.813
0.820
∼ 20s
VectorFusion
0.466
0.252
0.338
0.417
0.412
∼ 17min
SVGDreamer
0.690
0.546
0.620
0.650
0.695
∼ 24min
DiffSketcher
0.567
0.374
0.381
0.567
0.537
∼ 13min
SketchDirector
0.916
0.772
0.821
0.836
0.821
∼ 6min
Table 1: Comparison with baseline methods on multi-concept scene sketching. SketchDirector achieves the best performance across all five metrics.
Figure 8: User study results. SketchDirector preference over baseline methods across three criterias.
Figure 9: Qualitative comparison of VLM backbones. Visual results highlight differences in object-level fidelity, scene composition, and visual details across backbone models.
Method
Sem. ↑
Comp. ↑
Compl. ↑
Aes. ↑
Size ↑
Iter. ↓
Ver. ↑
Gemini-3.1-pro
0.858
0.723
0.723
0.819
0.707
5.332
0.535
Claude-Opus-4-7
0.879
0.681
0.791
0.739
0.721
5.843
0.477
Qwen3.7-Plus
0.629
0.477
0.526
0.552
0.507
10.743
0.436
OpenAI GPT-5.5
0.926
0.795
0.811
0.871
0.785
1.656
0.803
Table 2: Quantitative comparison of VLM backbones. Scores and reasoning iterations evaluate the layout reasoning quality and efficiency across backbone models.
Figure 10: Qualitative ablation of agentic layout reasoning. Visual results illustrate how individual components affect scene composition and object-level spatial coherence.
Method
Sem. ↑
Comp. ↑
Compl. ↑
Aes. ↑
Size ↑
Iter. ↓
Ver. ↑
w/o Plan
0.676
0.503
0.571
0.65
0.468
12.375
0.357
w/o Geo. Ver.
0.698
0.550
0.570
0.660
0.589
1.857
0.633
w/o Sem. Ver.
0.603
0.512
0.645
0.558
0.547
1.438
0.468
w/o Short Mem.
0.737
0.658
0.629
0.654
0.681
2.875
0.635
w/o Long Mem.
0.781
0.679
0.697
0.762
0.729
1.600
0.650
SketchDirector
0.839
0.765
0.732
0.768
0.742
1.375
0.873
Table 3: Quantitative ablation of agentic layout reasoning. Performance comparisons across metrics and reasoning iterations demonstrate the contributions of individual components to layout quality and reasoning efficiency.
Figure 11: Comparison of point-based and curve-based contour resampling method. Curve-based contour resampling preserves object shapes and structural details, providing a more stable initialization for subsequent refinement.
Figure 12: Qualitative ablation of progressive joint optimization. Removing local supervision, global supervision, or progressive optimization leads to degraded object fidelity and scene-level coherence.
Figure 13: Limitations of SketchDirector . (A) Challenges in modeling complex physical and semantic interactions. (B) Object hallucination during refinement.
Sketching in 3D space enables expressive reasoning about shape, structure, and spatial relationships, yet generating 3D sketches through natural language remains a major challenge. In this work, we introduce 3DrawAgent, a training-free, language-driven framework for 3D sketch generation that leverages large language models (LLMs) to sequentially draw 3D Bezier curves under geometric feedback. Unlike prior 2D sketch agents, our method introduces a relative experience optimization strategy that adapts the recently proposed Group Reward Policy Optimization (GRPO) paradigm. Instead of relying on explicit ground-truth supervision, we construct pairwise comparisons among generated sketches, with each pair consisting of a relatively better and a worse result based on CLIP-based perceptual rewards and LLM-based fine-grained qualitative assessment. These experiences are then used to iteratively refine the prior knowledge of 3D drawing, enabling black-box reinforcement of the model's 3D awareness. This design allows our model to self-improve its spatial understanding and drawing quality without parameter updates. Experiments show that 3DrawAgent can generate complex and coherent 3D Bezier sketches from diverse textual prompts, exhibit emergent geometric reasoning, and generalize to novel shapes, establishing a new paradigm for advancing the field of training-free 3D sketch intelligence.
Hongcan Xiao, Xinyue Xiao, Yilin Wang +2
Beijing University of Posts and Telecommunications · Jiangnan University · HaoHan Data
We tackle a new problem: generating geometrically consistent multi-view scenes from a single freehand sketch. Freehand sketches are the most geometrically impoverished input one could offer a multi-view generator. They convey scene intent through abstract strokes while introducing spatial distortions that actively conflict with any consistent 3D interpretation. No prior method attempts this; existing multi-view approaches require photographs or text, while sketch-to-3D methods need multiple views or costly per-scene optimisation. We address three compounding challenges; absent training data, the need for geometric reasoning from distorted 2D input, and cross-view consistency, through three mutually reinforcing contributions: (i) a curated dataset of ∼9k sketch-to-multiview samples, constructed via an automated generation and filtering pipeline; (ii) Parallel Camera-Aware Attention Adapters (CA3) that inject geometric inductive biases into the video transformer; and (iii) a Sparse Correspondence Supervision Loss (CSL) derived from Structure-from-Motion reconstructions. Our framework synthesizes all views in a single denoising process without requiring reference images, iterative refinement, or per-scene optimization. Our approach significantly outperforms state-of-the-art two-stage baselines, improving realism (FID) by over 60% and geometric consistency (Corr-Acc) by 23%, while providing up to a 3.7× inference speedup.
Ahmed Bourouis, Savas Ozkan, Andrea Maracani +2
Samsung Research, UK · SketchX, University of Surrey, UK
We develop a method for producing vector sketches one part at a time. To do this, we train a multi-modal language model-based agent using a novel multi-turn process-reward reinforcement learning following supervised fine-tuning. Our approach is enabled by a new dataset we call ControlSketch-Part, containing rich part-level annotations for sketches, obtained using a novel, generic automatic annotation pipeline that segments vector sketches into semantic parts and assigns paths to parts with a structured multi-stage labeling process. Our results indicate that incorporating structured part-level data and providing agent with the visual feedback through the process enables interpretable, controllable, and locally editable text-to-vector sketch generation.