SketchForge: Training-Free Scene-to-Vector Sketching via Decoupled Structure and Texture Optimization
Authors: Quang-Thinh Nguyen, Duy-Khang Do, Hien Trinh, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
Organizations: University of Science, Ho Chi Minh, Vietnam · Vietnam National University, Ho Chi Minh, Vietnam · University of Dayton, Dayton, Ohio, US
Converting a raster image into a resolution-independent, stroke-editable vector sketch while faithfully preserving both structural contours and tonal texture remains challenging: raster-space rendering pipelines are difficult to edit at the stroke level, while generation-from-scratch vectorization methods struggle to remain faithful to an arbitrary input image. We present SketchForge, a training-free framework that converts an input raster image into a resolution-independent SVG sketch through per-instance test-time optimization of Bezier curves, requiring no additional training or fine-tuning. Strokes are seeded via a contour and attention-guided initialization, then refined by a vector-native differentiable rendering pipeline that decouples structural contour strokes from free-form texture strokes under dedicated objectives, composited via multiply-blend layering. Extensive experiments show that SketchForge consistently outperforms existing vector-based baselines in structural fidelity and generation quality. Project page: https://khangcntn123.github.io/SketchForge
Figures & tables
Figure 1. Qualitative comparison on sketch vectorization. ControlSketch ( Arar et al., 2025 ) and SwiftSketch ( Arar et al., 2025 ) produce sparse, abstract strokes; CLIPasso ( Vinker et al., 2022 ) yields spatially disordered strokes; CLIPascene ( Vinker et al., 2023 ) over-abstracts fine detail; adapted DiffSketcher ( Xing et al., 2023 ) under-represents local texture. Our SketchForge most faithfully preserves object boundaries and structural detail. A grid comparing six vector-based sketch methods across two content images. The first content image shows a park walkway scene with trees; the second shows a dog resting on a lawn beside a wooden fence. Each subsequent column shows one method's output for both scenes. ControlSketch and SwiftSketch: sparse disconnected lines. CLIPasso: denser but disordered strokes. CLIPascene: bold silhouettes with lost detail. DiffSketcher: lighter, softer strokes. SketchForge: the most detailed, structurally faithful outputs.
Figure 2. SketchForge pipeline. Guided initialization seeds two stroke sets: the structure branch optimizes geometry against Iedge with a patch loss, while the texture branch optimizes all stroke attributes against Itex with an L2 loss. Multiply blending produces the final SVG. A pipeline diagram flowing left to right. On the far left, a colorful painted sailboat scene on turbulent waves under a golden sunset sky, labelled I_cnt, sends one arrow rightward into a grey-bordered box labelled Hybrid Init, and another arrow downward through an Edge Extraction step into a black-and-white line-art rendition of the same scene, labelled I_edge, showing sketched birds, sailboat, and wave contours. The Hybrid Init box contains two thumbnails: a dark Attention Map with a bright orange-purple heat-map glow near the sun, and a dark Contour Map with the sailboat and wave outlines highlighted in magenta. Below Hybrid Init, both I_cnt and I_edge feed into a blue Style Transfer block containing a sequence of colored vertical bar icons, which outputs a grayscale pencil-sketch-like rendition of the sailboat scene labelled I_tex. From the initialization module, the strokes branch into two parallel optimization rows on the right. The upper, blue-tinted row shows I_edge above a Patch Loss and Backpropagation label, feeding into a blue Differentiable Rendering block containing a sequence of small grayscale stroke-render thumbnails connected by an ellipsis, producing V_struct. The lower, pink-tinted row shows I_tex above an L2 Loss and Backpropagation label, feeding into a pink Differentiable Rendering block with its own sequence of small stroke-render thumbnails, producing V_tex. On the far right, V_struct and V_tex are shown as two overlapping diamond-shaped translucent layers, which merge via a downward arrow into the final Output: a detailed grayscale SVG sketch of the sailboat scene with visible bird, wave, and rigging strokes.
Method
Aesthetic ↑
DINO ↑
DreamSim ↓
Edge F1 ↑
CLIPasso ( Vinker et al., 2022 )
4.1690
0.2064
0.7256
0.5262
CLIPascene ( Vinker et al., 2023 )
4.3904
0.2022
0.7024
0.5797
SwiftSketch ( Arar et al., 2025 )
3.9784
0.1566
0.8305
0.3139
ControlSketch ( Arar et al., 2025 )
4.1493
0.1470
0.7960
0.3299
DiffSketcher ( Xing et al., 2023 ) (adapted)
4.8020
0.3760
0.5351
0.8325
SketchForge (Ours)
4.8407
0.4507
0.5552
0.9780
Table 1. Sketch vectorization on 64 content images. DINO and DreamSim use Icnt ; Edge F1 uses Iedge . Best in bold .
Init.
MAE ↓
SSIM ↑
Chamfer ↓
Edge F1 ↑
Random
0.4384
0.1120
0.0006
0.7731
Guided
0.2041
0.8068
0.0003
0.8841
Table 2. Structure initialization ablation against Iedge . Best in bold .
Init.
SSIM ↑
DreamSim ↓
Edge F1 ↑
DINO ↑
LPIPS ↓
Random
0.6718
0.2459
0.7975
0.6378
0.4034
Guided
0.7375
0.1744
0.8067
0.7328
0.3539
Table 3. Texture initialization ablation against Itex ; Edge F1 uses Iedge . Best in bold .
Variant
DINO ↑
DreamSim ↓
Edge F1 ↑
Aesthetic ↑
Texture only
0.7328
0.1744
0.8067
5.0715
Structure only
0.5800
0.2918
0.8841
4.8718
Fusion
0.7372
0.1712
0.8629
5.3507
Table 4. Fusion ablation. DINO/DreamSim use Itex ; Edge F1 uses Iedge . Best in bold .
Outpainting sparse sketches while preserving contour coherence and stroke style remains challenging. We present SketchBloom, a training-free framework that combines pretrained raster outpainting and style-transfer guidance with per-instance vector optimization. Given a sketch, a scene prompt, and an optional reference style, it produces an editable Scalable Vector Graphic (SVG) composed of Bezier strokes. Adaptive latent blending balances content contours and reference texture without model retraining. A reference-derived stroke vocabulary initializes contour- and DINO-attention-guided strokes, which are optimized separately for structure and texture before contour-emphasizing multiply fusion. As in prior optimization-based vector sketching, no task-specific network is trained; our contribution is the synthesis design specialized to sketch outpainting. On LineArtOut-1K, comprising 1,024 input and style combinations, SketchBloom improves perceptual, semantic, and structural fidelity over the evaluated vector baselines when all methods vectorize the same outpainted raster. This protocol evaluates vector synthesis rather than end-to-end outpainting or reference-style superiority. Runtime is approximately six minutes per pair on a single NVIDIA H200 GPU. Project page: https://khangcntn123.github.io/SketchBloom/
Quang-Thinh Nguyen, Duy-Khang Do, Tam V. Nguyen +2
University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Dayton, Ohio, United States
Vector sketches remain one of the most concise and immediate mediums for abstract human expression. However, generating high-quality vector strokes that exhibit human-like drawing styles remains an open challenge due to the severe scarcity of fine-grained, high-quality text-to-sketch paired data. Existing text-conditioned generation methods often rely on unstable, time-consuming optimization or struggle to generalize to unseen categories in a zero-shot manner. To address these limitations, we present SketchFlow, a novel generative framework rooted in Optimal Transport (OT) theory and flow matching. By leveraging pre-trained CLIP models to bypass labor-intensive image-level text annotations, we formulate cross-modal alignment as a continuous mapping problem directly within the CLIP latent space. To bridge the inevitable modality gap between discrete text concepts and continuous sketch features, we first inject noise into discrete category embeddings to construct a continuous Gaussian Mixture Model (GMM) prior. We then utilize an Optimal Transport Conditional Flow Matching (OT-CFM) model to learn a deterministic vector field mapping from this continuous GMM prior to the target sketch feature distribution. Finally, a Hybrid Diffusion Decoder, fusing 1D U-Net and Transformer architectures, is designed to decode these features into fast and high-fidelity stroke trajectories. Extensive experiments demonstrate that SketchFlow substantially outperforms existing baselines in visual quality and adherence to natural human drawing styles. Furthermore, our geometry-preserving framework demonstrates promising local zero-shot synthesis for prompts beyond the QuickDraw training vocabulary, including unseen concept labels and semantic modifiers, while enabling smooth, continuous semantic interpolation between distinct concepts. Source code is available at: https://github.com/doudin404/SketchFlow.
Jin Zhou, Hongliang Yang, Pengfei Xu +1
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, CSSE, Shenzhen University, Shenzhen, China
Sketch-guided image inpainting provides intuitive structural control, yet real sketches often mix reliable global intent with locally crowded, displaced, incomplete, or deliberately unconventional strokes. Existing approaches typically either retain the input sketch as a fixed condition throughout denoising or refine it into a clean structure before RGB synthesis. The former assumes uniformly reliable strokes and can propagate local errors throughout generation; the latter must resolve ambiguous structure before emerging appearance and semantic context become available. We propose SketchSense, a framework that interprets imperfect sketch guidance by synchronously denoising interacting RGB and structure streams. Bidirectional Attention Fusion couples appearance generation with structural recovery, producing a refined structure that exposes the model's evolving sketch interpretation. A phrase-level objective aligns the semantic grounding of the two streams. Sketch-Aware Spatial Regulation further adapts sketch use to local generation states by modulating attention and the fusion process, while an optional signed prior injects preserve-versus-correct intent into feature representations and attention behavior. Experiments on natural and structurally complex imagery show substantial gains over existing methods in both restoration quality and structural fidelity.