cs.CVFeb 17, 2026

VideoSketcher: Sequential Sketch Generation Using Video Model Priors

Authors: Hui Ren, Yuval Alaluf, Omer Bar Tal, Alexander Schwing, Antonio Torralba, Yael Vinker

Organizations: UIUC, USA · Runway, USA · MIT, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Sketching is inherently sequential: strokes are drawn progressively to explore and refine ideas. Yet most generative approaches treat sketches as static images, ignoring the temporal process underlying creative exploration. Modeling this sequential structure remains challenging: prior methods either rely on large-scale human-drawn datasets with limited diversity, or use large language models (LLMs) to produce drawing instructions, often at the cost of visual fidelity. We present VideoSketcher, a method for generating high-quality sketching processes by adapting pretrained text-to-video diffusion models to the sparse, continuous nature of sketch formation. Our key insight is that LLMs and video diffusion models offer complementary strengths: LLMs act as semantic planners that decompose concepts into step-by-step instructions, while video diffusion models serve as powerful "renderers" that translate them into temporally coherent sketch sequences. We introduce a two-stage fine-tuning strategy that decouples temporal structure from visual appearance: stroke ordering is learned from synthetic shape compositions, while style is distilled from as few as seven hand-drawn examples. Despite minimal supervision, our method can generate diverse, high-quality sequential sketches that faithfully follow specified drawing orders. Our framework naturally extends to brush style control and autoregressive generation, supporting artistic applications.

Figures & tables

Figure 1. VideoSketcher enables sequential sketch generation in pixel space via video diffusion priors. Given a text prompt, our method generates a step-by-step drawing process that follows natural stroke order with high visual quality (top). Our approach further supports user-specified brush styles (middle) and real-time human-model co-drawing through an autoregressive framework (bottom). Please see the full videos in our project page: https://videosketcher.github.io/
Figure 2. Complementary strengths of LLMs and video diffusion models. Given a high-level concept to be drawn, the LLM acts as a planner, decomposing it into an ordered sequence of drawing steps. The video model, trained to generate sketch-like outputs, acts as a soft "renderer" — translating the given textual instructions into a sketch-progression video. By bypassing explicit parametric representations, we leverage the rich visual prior of video models: a high-level cue like “wheels” is sufficient for the model to produce faithful visual strokes, without requiring large-scale parametric data.
Figure 3. Two-stage fine-tuning scheme. Left: synthetic sketches composed of simple geometric primitives teach drawing “grammar” and stroke ordering, independently of appearance. Right: a small set of human-drawn sketches of real-world objects adapts the model to a target visual style.
Figure 4. Brush conditioning. Top: Data preparation — each SVG sketch is rendered with diverse brush styles and colors. Bottom: We fine-tune an Image-to-Video model where the first frame, containing a brush exemplar in the corner, conditions the generated stroke appearance.
Figure 5. Qualitative results. Results generated with our fine-tuned video model. Full video results are provided in the supplementary materials.
MethodTop-1Top-5
Naive Prompting (Wan 2.1)0.92 ±\pm 0.030.99 ±\pm 0.01
PaintsUndo (FLUX 2) ( Team, 2024 )1.00 ±\pm 0.001.00 ±\pm 0.00
SketchAgent ( Vinker et al., 2025 )0.48 ±\pm 0.050.71 ±\pm 0.05
Human (QuickDraw ( Jonas et al., 2016 ) )0.52 ±\pm 0.050.70 ±\pm 0.05
Ours0.82 ±\pm 0.040.96 ±\pm 0.02
Table 1. CLIP-based sketch recognition. Average Top-1 and Top-5 accuracy of a CLIP zero-shot classifier evaluated on the last frame of 100 sketches from 50 categories. Results are reported as mean ± std across categories.
Figure 6. CLIP-based recognition over the sketching process. Our method gradually increases semantic recognizability as the sketch progresses, in contrast to baselines that either collapse temporal progression or attain lower recognition. Shaded areas indicate variance across samples.
Figure 7. Qualitative comparison of sketch generation across methods. The concept depicted is “a giraffe”.
 Cat: 1. Body. 2. Head. 3. Ears. 4. Face details. 5. Legs. 6. Tail.Cat: 1. Head. 2. Ears. 3. Body. 4. Legs. 5. Tail. 6. Face details.
Primiti- ves Only  
7 Human Sketches Only  
Full  
Figure 8. Ablation study qualitative comparison. We compare sketching processes from models trained only on geometric primitives, only on real sketches, and the full two-stage model. The full model combines the strengths of both baselines, producing visually appealing sketches that follow the specified order.
Figure 9. Limitations. (a) Multiple strokes may appear together (in red). (b) Model’s prior can override prompt. (c) Concepts outside the video model’s knowledge are incorrectly depicted. (d) Reduced quality for AR outputs.
Figure 10. Qualitative results. Additional results generated with our fine-tuned text-to-video model. Full video results are provided in the supplementary.
Figure 11. Text-specified stroke ordering. Each row shows the same concept generated using a different text prompt that specifies a distinct drawing order.
Figure 12. Diversity of sequential sketch generation. Left: two seeds for the same prompt; Right: the same object in different settings/actions.
Figure 13. Additional results for brush style control. We show concepts drawn with seen (first row) and unseen (second, third rows) brush styles and colors.
Figure 14. Autoregressive results. Additional results from our autoregressive model, which enables interactive generation while maintaining visual quality comparable to diffusion-based results.
Figure 15. Co-Drawing. Left: our interactive demo, where users draw alongside the model in real time. Right: turn-based co-drawing for “a creature.”
Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 16. Six brush styles (left) and 8 colors (right) used for training our brush-conditioned model.
Figure 17. Scientific Concepts. Representative results for scientific concepts across methods. SketchAgent leverages LLM knowledge to capture conceptual structure (e.g., the interference pattern in double-slit, F=ma in Newton’s laws), though with limited visual detail. Wan 2.1 occasionally produces informative diagrams but often includes colored backgrounds and text labels rather than sketch-style outputs. Our method produces visually coherent sketches, inheriting both the strengths and limitations of the base model—performing well on concepts like photosynthesis while struggling with more abstract ones like pendulum motion. Comparison grid of final sketches from Ours, SketchAgent, and Wan~2.1 across four general knowledge categories with four concepts each.
Figure 18. Diagrams. Representative results for diagram concepts. SketchAgent produces structurally correct but visually minimal outputs. Wan 2.1 generates detailed diagrams with text and color, deviating from a sketch aesthetic. Our method captures the visual structure of diagrams (e.g., connected nodes in flowcharts and mind maps, overlapping circles in Venn diagrams) with a cleaner sketch style, though text elements are often garbled or nonsensical. Comparison grid of final sketches from Ours, SketchAgent, and Wan~2.1 across four general knowledge categories with four concepts each.
Figure 19. Classification accuracy across categories. Video-based methods excel on Landmarks (  97%~97\% vs. 63.3%63.3\% for SketchAgent), while SketchAgent dominates Functions due to its LLM backbone. Our method’s accuracy closely tracks Wan 2.1, confirming that both knowledge and limitations transfer from the base model.
Figure 20. Notable Landmarks. Representative results for landmark concepts. This category highlights the strength of video-based approaches: both Wan 2.1 and our method produce detailed, highly recognizable depictions, reflecting the strong prior knowledge of landmarks in video training data. In contrast, SketchAgent’s outputs are overly simplistic and often fail to capture the iconic features (e.g., the Eiffel Tower reduced to basic triangles, the Statue of Liberty barely recognizable). Comparison grid of final sketches from Ours, SketchAgent, and Wan~2.1 across four general knowledge categories with four concepts each.
Figure 21. Functions. Representative results for mathematical functions. This category most clearly demonstrates the advantage of LLM-based approaches: SketchAgent produces precise, mathematically correct curves due to its language model backbone. Both Wan 2.1 and our method struggle—curves are often incorrect or unrecognizable, and text/equations are garbled (e.g., y=log(x)y=log(x) in our output). This reveals a fundamental limitation of video models for concepts requiring symbolic or mathematical knowledge. Comparison grid of final sketches from Ours, SketchAgent, and Wan~2.1 across four general knowledge categories with four concepts each.
Figure 22. Accumulated ratio of newly added pixels as a function of video progress. Values are normalized such that the final frame equals 1. Our method exhibits a smooth and steady accumulation curve, reflecting incremental stroke additions over time and closely mirroring human drawing behavior.
Evaluation SetNum.TotalMulti-StrokeRatio ↓\downarrow
 SamplesFramesFrames 
QuickDraw Concepts20015196297519.58%
Scenes806361235136.96%
Appendix
Table 3. Stroke continuity evaluation. We measure the frequency of frames containing multiple disjoint strokes on QuickDraw object concepts (typically simpler concepts) and scene-level prompts (typically more complex concepts).
Figure 23. Demo of our interactive sketching interface. Users can co-draw with the model on the shared canvas for a concept.
Figure 24. Seven sketching videos derived from the QuickDraw Dataset ( Jonas et al., 2016 ) , illustrating a doodle-like, rapid, drawing style from crowd-sourced participants.
Figure 25. Generated sketches from the model fine-tuned on the QuickDraw samples, reflecting a flat, simple, and imprecise doodle-like style.
Figure 26. Seven sketching videos derived from the DifferSketching Dataset ( Xiao et al., 2022 ) , showcasing more structured and detailed sketches with 3D viewpoints from professional participants.
Figure 27. Generated sketches from the model fine-tuned on samples from the DifferSketching data, capturing a more detailed and structured style with 3D perspective cues.
Figure 28. User study interface. Participants view two anonymous sketch videos side by side and answer the question: “Which video has better visual quality (smoother, less flickering, fewer artifacts)?” with a confidence level. The interface displays 18 pairs per session with progress tracking and navigation controls.
Figure 29. Prompt adherence with incremental details. Given a base concept ( a house ), we progressively add details in a prompt (e.g., with doors , with chimney and doors , with chimney, doors, and windows , and with chimney, doors, windows, and fence ). The generated sketch sequences consistently follow the prompt and incorporate the newly requested elements.
Brush StyleColorScore ↓\downarrowBaseline
bubbles-mediumblack0.118±0.0050.118\pm 0.0050.287±0.0050.287\pm 0.005
indigo-blue0.083±0.0090.083\pm 0.0090.265±0.0040.265\pm 0.004
mocha-brown0.077±0.0050.077\pm 0.0050.265±0.0030.265\pm 0.003
mustard-olive0.058±0.0030.058\pm 0.0030.272±0.0020.272\pm 0.002
pink0.081±0.0070.081\pm 0.0070.284±0.0030.284\pm 0.003
caligraphy-verticalblack0.230±0.0170.230\pm 0.0170.415±0.0220.415\pm 0.022
Appendix
Table 4. Per-brush-style metric breakdown. Gram-matrix distance between generated frames and a manually drawn reference sketch in the target style (Score ↓\downarrow ), compared to the expected distance to mismatched brush-color references (mean distance to the other 24 references). Each per-style entry reports mean ±\pm standard deviation over 30 videos; the Average row reports mean ±\pm standard deviation taken across the 25 style means.
BrushColorScore ↓\downarrowBaselineBrushColorScore ↓\downarrowBaseline
charcoalblue0.088±0.0530.088\pm 0.0530.211±0.0350.211\pm 0.035markerblue0.038±0.0220.038\pm 0.0220.162±0.0070.162\pm 0.007
carrot0.109±0.0620.109\pm 0.0620.235±0.0430.235\pm 0.043carrot0.044±0.0240.044\pm 0.0240.167±0.0060.167\pm 0.006
dark-red0.115±0.0690.115\pm 0.0690.252±0.0520.252\pm 0.052dark-red0.045±0.0250.045\pm 0.0250.163±0.0070.163\pm 0.007
midnight-blue0.102±0.0590.102\pm 0.0590.229±0.0390.229\pm 0.039midnight-blue0.046±0.0260.046\pm 0.0260.166±0.0070.166\pm 0.007
nephritis0.082±0.0480.082\pm 0.0480.207±0.0320.207\pm 0.032nephritis0.040±0.0230.040\pm 0.0230.161±0.0070.161\pm 0.007
turquoise0.087±0.0510.087\pm 0.0510.215±0.0360.215\pm 0.036turquoise0.039±0.0220.039\pm 0.0220.159±0.0070.159\pm 0.007
Appendix
Table 5. Per-brush-style metric breakdown on training-distribution styles. Gram-matrix distance between generated frames and a training-set reference sketch in the target style (Score ↓\downarrow ), compared to the expected distance to mismatched brush-color references (mean distance to the other 47 references). Each per-style entry reports mean ±\pm standard deviation over 7 videos; the Average row reports mean ±\pm standard deviation taken across the 48 style means. The Average serves as an empirical lower-bound reference for the unseen-styles evaluation in Table 4 .
Figure 30. Additional qualitative comparison of sequential sketch generation across methods. The Human drawings are taken from QuickDraw ( Jonas et al., 2016 ) . Full videos, and many more results are provided in the supplementary html.
Figure 31. Diversity of sequential sketch generation. Using four differnt seeds for the same prompt result in four differnt sketches.
Figure 32. Diversity of sequential sketch generation. Using four differnt seeds for the same prompt result in four differnt sketches.
Figure 33. Brush control results for four unseen brush style and color combinations.
Figure 34. Brush control I2V results for “Amsterdam canal.” Results shown for four unseen brush style and color combinations.
Figure 35. Training data exemplars of our simple geometric primitives. Examples of the simple geometric primitives used in the first stage of our fine-tuning pipeline, focused on teaching the model basic drawing “grammar”. Each composition is rendered with multiple drawing orders, as specified by the text prompt.
Figure 36. Training data exemplars of our simple geometric primitives (continued). Additional stacked-shape compositions with varied drawing orders.
Figure 37. Our seven real human sketches used for training. We provide the seven human-drawn sketches used in the second stage of our fine-tuning pipeline, along with their full text prompts.
Figure 38. Additional text-to-video results. Additional qualitative results generated with our text-to-video model.
Figure 39. Autoregressive results (1/3). Additional autoregressive model results on QuickDraw prompts.
Figure 40. Autoregressive results (2/3). Additional autoregressive model results on QuickDraw prompts.
Figure 41. Autoregressive results (3/3). Additional autoregressive model results on QuickDraw prompts.

Explore similar work

CardsList
May 22, 2026

DrawVideo: Generating Long Video from Storyboard Keyframe Sketches

Long video generation requires high-fidelity synthesis, coherent narrative structure, and user control over extended time spans. Existing text-to-video methods often rely on a single long prompt, limiting control over pose, composition, layout, and motion. We propose DrawVideo, a sketch-guided, storyboard-driven framework for controllable long-video generation. DrawVideo decomposes long videos into independently controllable shots, each defined by a black-and-white sketch, an appearance prompt, and a motion prompt. The sketch controls pose and layout, the appearance prompt defines identity, scene, and style, and the motion prompt guides temporal dynamics. DrawVideo follows a hierarchical 'global multi-shot, local single-sketch' strategy: it first generates a structure-aligned reference keyframe, then expands the motion prompt into derivative keyframes representing action states, and finally synthesizes clips between adjacent keyframes to build each shot. We also introduce SketchLongVideo, the first dataset for sketch-guided text-to-long-video generation, constructed from animation videos via shot detection, keyframe extraction, vision-language recognition, prompt decomposition, and sketch conversion. Experiments show that DrawVideo achieves strong structural controllability, appearance consistency, visual stability, and coherent long-video generation.

Chuanzhi Xu, Huiqi Liang, Bang Shi +7

The University of Sydney, NSW 2006, Australia · Charles Sturt University, NSW 2800, Australia

Long Video GenerationVideo GenerationMulti-Shot Video GenerationControllable Video Generation
AskPDF
Aug 12, 2026

Draw This First

We invert the typical formulation of sketch generation: instead of drawing strokes in order, we predict a 2D field that defines the order in which strokes are drawn. We use a pretrained latent flow-matching transformer to supply the image prior to predict an intermediate representation, while training the VAE's decoder to predict the order field, stroke mask, and stroke segmentation. We vectorize the predicted segmentation into polylines and sort them by the field, producing an ordered vector sketch. Our model can predict an ordered vector sketch from a text description or derender an image into ordered vectors; for either, it follows text instructions specifying the order of drawing.

Dazhi Zhong, Rowan Bradbury, Grant Davis

Wand Technologies · Bradbury Group

Image Generation
AskPDF
Aug 21, 2026

SketchFlow: Zero-Shot Vector Sketch Generation via GMM Prior Flow in CLIP Latent Space

Vector sketches remain one of the most concise and immediate mediums for abstract human expression. However, generating high-quality vector strokes that exhibit human-like drawing styles remains an open challenge due to the severe scarcity of fine-grained, high-quality text-to-sketch paired data. Existing text-conditioned generation methods often rely on unstable, time-consuming optimization or struggle to generalize to unseen categories in a zero-shot manner. To address these limitations, we present SketchFlow, a novel generative framework rooted in Optimal Transport (OT) theory and flow matching. By leveraging pre-trained CLIP models to bypass labor-intensive image-level text annotations, we formulate cross-modal alignment as a continuous mapping problem directly within the CLIP latent space. To bridge the inevitable modality gap between discrete text concepts and continuous sketch features, we first inject noise into discrete category embeddings to construct a continuous Gaussian Mixture Model (GMM) prior. We then utilize an Optimal Transport Conditional Flow Matching (OT-CFM) model to learn a deterministic vector field mapping from this continuous GMM prior to the target sketch feature distribution. Finally, a Hybrid Diffusion Decoder, fusing 1D U-Net and Transformer architectures, is designed to decode these features into fast and high-fidelity stroke trajectories. Extensive experiments demonstrate that SketchFlow substantially outperforms existing baselines in visual quality and adherence to natural human drawing styles. Furthermore, our geometry-preserving framework demonstrates promising local zero-shot synthesis for prompts beyond the QuickDraw training vocabulary, including unseen concept labels and semantic modifiers, while enabling smooth, continuous semantic interpolation between distinct concepts. Source code is available at: https://github.com/doudin404/SketchFlow.

Jin Zhou, Hongliang Yang, Pengfei Xu +1

Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, CSSE, Shenzhen University, Shenzhen, China

Flow-Based Generative ModelingImage GenerationZero-Shot Learning
AskPDF