Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs

Matthew Perlman; James Beetham; Niels Da Vitoria Lobo; Amrit Singh Bedi; Mubarak Shah

cs.CVAug 26, 2026

Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs

Authors: Matthew Perlman, James Beetham, Niels Da Vitoria Lobo, Amrit Singh Bedi, Mubarak Shah

Organizations: University of Massachusetts Amherst · University of Central Florida

Abstract

Scalable Vector Graphics (SVGs) power much of the modern visual ecosystem, yet state-of-the-art generative models focus almost entirely on rasterized images. We explore whether inference-time methods can unlock SVG generation capabilities in off-the-shelf vision-language models (VLMs). We systematically evaluate a constrained iterative refinement harness that combines visual feedback, structured editing, and constrained decoding to characterize the capabilities and limitations of current VLMs for SVG generation. Across multiple VLMs and generation settings, we find that constrained decoding improves compilation success rates, while iterative refinement reveals a deficit in visual reasoning and self-correction. Our results highlight both the promise and current limitations of using inference-time methods to adapt general-purpose VLMs for SVG generation.

Figures & tables

Figure 1: Our proposed harness. Given a user query, the VLM generates a constrained output specifying an SVG edit. We parse the action, update and render the SVG, and return the code and image to the VLM for the next refinement iteration .

		Text $\rightarrow$ SVG				Image $\rightarrow$ SVG
Decoding	Pipeline	Compile $\uparrow$	CLIP $\uparrow$	Aesthetic. $\uparrow$	#Tokens	Compile $\uparrow$	DINO $\uparrow$	LPIPS $\downarrow$	#Tokens
Unconstrained	Single Turn	0.837	0.255	4.655	1.4k	0.630	0.704	0.396	1.7k
	Unstructured	0.475	0.246	4.560	4.8k	0.113	0.704	0.372	10.7k
	Semi-Structured	0.180	0.205	4.415	7.9k	0.500	0.617	0.418	8.8k
	Structured	0.785	0.226	4.480	3.3k	0.650	0.633	0.413	8.1k
Constrained	Single Turn	0.830	0.220	4.530	0.6k	0.830	0.411	0.579	0.4k

Table 1: Results averaged across Qwen3.5 9B and 27B on MMSVG-Bench.

Figure 2: Images generated by GPT-5.6 using each pipeline and decoding strategy from the Text-to-SVG prompt "A robot painting".

Vector Graphics Generation

See all 26 papers

RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation
Jul 30, 2026Shaobo Liu, Feiqiao Mao, Shuaishuai Zhou +4Image Generation Large Vision-Language Models
Shenzhen University · Shenzhen, China · Peking University +1
IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework
Mar 10, 2026Feiyu Wang, Jiayuan Yang, Zhiyuan Zhao +4Text-to-SVG Generation Vision-Language Models
Fudan University · TeleAI · Northwestern Polytechnical University +1
Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning
Mar 17, 2026Haomin Wang, Qi Wei, Qianli Ma +4Text-to-SVG Generation VLM Reasoning
Shanghai Jiao Tong University · Shanghai AI Laboratory · Nanjing University +1

Jul 30, 2026J/K move · Enter open · S save

RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation

Shaobo Liu, Feiqiao Mao, Shuaishuai Zhou, Yan Zhan+3

Shenzhen University · Shenzhen, China · Peking University+1

Image Generation Large Vision-Language Models RL for VLMs

Open paper AskPDF arXiv

We propose RefineSVG, a single-step closed-loop visual feedback framework that enables multimodal large language models (MLLMs) to perform high-fidelity image-to-SVG generation through self-correction. Existing MLLM-based approaches rely on single-pass open-loop inference, where the model receives visual input only once and must generate thousands of SVG code tokens without intermediate verification. This paradigm inevitably leads to geometric drift, error accumulation, and visual hallucination on complex images. RefineSVG overcomes this limitation by invoking an external rendering engine after an initial SVG generation pass to compare the rendered output against the target image. The comparison yields a multi-dimensional visual residual map (Diff-Map) that is fed back to the model as a ReAct-style correction signal, driving a targeted correction step. To support this render-observe-correct interaction, we further introduce an SVG-oriented semantic vocabulary that compresses token sequences by over 52%. A progressive training pipeline spanning supervised fine-tuning, rejection-sampling cold-start data construction, and end-to-end agentic reinforcement learning aligns the model with closed-loop visual correction. Extensive experiments show that RefineSVG consistently outperforms existing baselines in reconstruction fidelity, structural accuracy, and code efficiency.Code is available at https://github.com/liuxiaobo66/RefineSVG.

Abstract

Vector Graphics Generation

Explore similar work

RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation

IntroSVG: Learning from Rendering Feedback for Text-to-SVG Generation via an Introspective Generator-Critic Framework

Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning