cs.AIMar 19, 2026
SaveTeaching an Agent to Sketch One Part at a Time
Organizations: TTI-Chicago · University of Chicago · MIT CSAIL
Abstract
We develop a method for producing vector sketches one part at a time. To do this, we train a multi-modal language model-based agent using a novel multi-turn process-reward reinforcement learning following supervised fine-tuning. Our approach is enabled by a new dataset we call ControlSketch-Part, containing rich part-level annotations for sketches, obtained using a novel, generic automatic annotation pipeline that segments vector sketches into semantic parts and assigns paths to parts with a structured multi-stage labeling process. Our results indicate that incorporating structured part-level data and providing agent with the visual feedback through the process enables interpretable, controllable, and locally editable text-to-vector sketch generation.
Figures & tables
Figure 1: Progressive vector sketch generation using our VLM agent. Trained on our new dataset via SFT + RL training, our agent generates sketches part-by-part, conditioned on text instructions and the evolving canvas. It produces diverse, structurally plausible sketches and supports localized editing via arbitrary stroke removal and replacement.
Figure 2: An illustration of our automated part annotation pipeline. The same VLM is used to produce part designations and assignments in some stages and to critique these assignments and suggest improvements in other stages. Green check marks indicate outputs retained in the final dataset.
Figure 3: Examples from the ControlSketch-Part dataset. We show part decomposition for 4 sketches with various objects and number of parts. The actual caption and part descriptions are shown for the rightmost sketch. The black text is the overall caption. The color-coded part descriptions and stroke groups demonstrate the part-level semantic annotations.
Figure 4: The visualization of the training pipeline. The task of generating vector sketches based on text prompts is split into multiple turns. Blue arrows: sequential computation; red arrows: loss. Cross-entropy loss and DreamSim reward are used at training signal at SFT and RL stages, respectively. is the policy model, i.e., our VLM.
Figure 5: The Long-CLIP cosine similarity across all tested models. The Ground Truth (GT) value and the Random value are the cosine similarity scores of text to the ground truth sketches from ControlSketch-Part and sketches of randomly sampled paths.
Figure 6: Pairwise preference studies conducted between our final model (SFT + RL) and the baselines. The title for each bar describes the baseline method that we are comparing against. The first column is the ablation between our final method (SFT + RL) versus the SFT only variant, which demonstrates the effectiveness of RL.
Figure 7: Qualitative comparison. Part-by-part generated samples are color-coded to illustrate different parts. One-shot generations are rendered in black. Samples in each group are generated with the same text input. Our model and training process do not rely on the class labels in any way, and we only show these for reference.
Figure 8: Example outputs from Ours (SFT + RL) trained on ControlSketch-Part. Our model and training process do not rely on the class labels in any way, and we only show these for reference.
Figure 9: Additional progressive editing examples. Left: identical part descriptions with different initial canvasses lead to different outputs. Right: changing the description for an early part but keeping the subsequent part descriptions same produces two sketches with significant differences localized to the affected part.
| Method | Long-CLIP |
|---|---|
| Single-turn RL | 0.281 |
| Multi-turn outcome-reward | 0.286 |
| Multi-turn process-reward (ours) | 0.298 |
Table 1: Average Long-CLIP scores across different ablation configurations, using Qwen2.5-VL-3B.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Additional part-by-part results of our model. Part descriptions and caption appear above each sketch’s cumulative frames, with newly added parts color-coded to match corresponding part labels.
Figure A2 : Additional part-by-part results of our model (continued).
Figure A3 : Additional part-by-part results of our model (continued).
Figure A4 : Additional part-by-part results of our model (continued).
Figure A5 : Additional part-by-part results of our model (continued).
Figure A6 : Additional part-by-part results of our model (continued).
Figure A7 : Additional part-by-part results of our model (continued).
Figure A8 : Additional part-by-part results of our model (continued).
Figure A9 : Additional part-by-part results of our model (continued).
Figure A10 : Additional part-by-part results of our model (continued).
Figure A11 : Additional part-by-part results of our model (continued).
Figure A12 : Additional part-by-part results of our model (continued).
Figure A13 : Additional part-by-part results of our model (continued).
Figure A14 : Additional part-by-part results of our model (continued).
Figure A15 : Additional part-by-part results of our model (continued).
Table A1 : Additional Examples of the ControlSketch-Part Dataset.
Table A2 : Additional Examples of the ControlSketch-Part Dataset (continued).
Table A3 : Additional Examples of the ControlSketch-Part Dataset (continued).
Figure A16 : Failure cases.
Figure A17 : Sample outputs on test data. The actual categories are below the sketches. These categories are unseen by the agent during training.
Figure A18 : Adding out-of-domain parts. Each pair shows the original incomplete sketch (left) and the result after the agent attempts to add an out-of-domain part (right). The text below each pair is the part description provided as input to the agent.
Figure A19 : Worst-quality outputs across all in-distribution categories.
Figure A20 : Worst-quality outputs (continued).
Figure A21 : Worst-quality outputs (continued).