Visual-ERM: Reward Modeling for Visual Equivalence

Ziyu Liu; Shengyuan Ding; Xinyu Fang; Xuanlang Dai; Penghui Yang; Jianze Liang; Jiaqi Wang; Kai Chen

cs.CVMar 13, 2026

Visual-ERM: Reward Modeling for Visual Equivalence

Authors: Ziyu Liu, Shengyuan Ding, Xinyu Fang, Xuanlang Dai, Penghui Yang, Jianze Liang, Jiaqi Wang, Kai Chen,

Organizations: Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University · CUHK

Abstract

Vision-to-code tasks require models to reconstruct structured visual inputs, such as charts, tables, and SVGs, into executable or structured representations with high visual fidelity. While recent Large Vision Language Models (LVLMs) achieve strong results via supervised fine-tuning, reinforcement learning remains challenging due to misaligned reward signals. Existing rewards either rely on textual rules or coarse visual embedding similarity, both of which fail to capture fine-grained visual discrepancies and are vulnerable to reward hacking. We propose Visual Equivalence Reward Model (Visual-ERM), a multimodal generative reward model that provides fine-grained, interpretable, and task-agnostic feedback to evaluate vision-to-code quality directly in the rendered visual space. Integrated into RL, Visual-ERM improves Qwen3-VL-8B-Instruct by +8.4 on chart-to-code and yields consistent gains on table and SVG parsing (+2.7, +4.1 on average), and further strengthens test-time scaling via reflection and revision. We also introduce VisualCritic-RewardBench (VC-RewardBench), a benchmark for judging fine-grained image-to-image discrepancies on structured visual data, where Visual-ERM at 8B decisively outperforms Qwen3-VL-235B-Instruct and approaches leading closed-source models. Our results suggest that fine-grained visual reward supervision is both necessary and sufficient for vision-to-code RL, regardless of task specificity.

Figures & tables

Figure 1: Overview of vision-to-code reward modeling and Visual-ERM . Top: vision-to-code parses a structured visual input (chart, table, or SVG) into code or markup, which is rendered back to an image so that quality can be judged in the visual space. Bottom: we compare reward signals along four properties: visual , fine-grained , interpretable , and task-agnostic . Text-based rewards (e.g., TEDS, edit distance) ignore visual cues; vision-encoder similarities (e.g., DINO [ 34 ] , CLIP [ 29 ] ) capture visual content but remain coarse and opaque. Visual-ERM combines all four properties, providing a reliable supervisor across vision-to-code tasks.

Figure 2: Analysis of existing rewards and downstream impact. Left: a text-based reward (TEDS) and a vision-based reward (DINO) assign near-perfect scores to predictions that Visual-ERM identifies as visually wrong. Right: (a) RL training curves where text- and vision-based rewards plateau via reward hacking while Visual-ERM keeps improving; (b, c) downstream gains on Chart-to-Code and Table-to-Markdown.

Figure 3: (a) Training data and model training. From paired ground-truth and rendered images, we obtain fine-grained discrepancy annotations (category, severity, location, description) and fine-tune a base LVLM into Visual-ERM . (b) Deployment. Visual-ERM produces unified judgments across chart, SVG, and table tasks, yielding scalar rewards for RL and natural-language feedback for test-time scaling.

Figure 4: VC-RewardBench. We construct VC-RewardBench by first leveraging several advanced proprietary models for preliminary annotation, followed by manual consolidation and filtering, resulting in 1,335 high-quality instances.

Model	ChartMimic [ 41 ] (Direct)				ChartMimic [ 41 ] (Customized)				Avg $\uparrow$
Model	Exec_rate $\uparrow$	Low $\uparrow$	High $\uparrow$	Overall $\uparrow$	Exec_rate $\uparrow$	Low $\uparrow$	High $\uparrow$	Overall $\uparrow$	Avg $\uparrow$
Baseline
InternVL3.5-8B [ 38 ]	62.2	41.9	48.9	45.4	79.7	53.0	63.3	58.2	51.8
Qwen3-VL-8B-Instruct [ 3 ]	82.2	62.7	72.7	67.7	86.8	66.3	76.8	71.6	69.6
VinciCoder-8B-SFT [ 47 ]	86.2	68.2	77.5	72.9	86.2	52.3	71.5	61.9	67.4
JanusCoderV-8B [ 36 ]	80.6	65.8	73.2	69.5	80.7	66.7	74.2	70.4	70.0

Table 1: Evaluation Results on Chart-to-Code Tasks. We evaluate on ChartMimic(Direct) and ChartMimic(Customized). ChartMimic(Direct) requires generating Python code that reproduces the input chart, while ChartMimic(Customized) requires generating code for a new chart that matches the given chart’s style and data constraints. Exec_rate denotes the execution success rate; Low and High denote the low-level and high-level scores, respectively.

Model	OmniDocBench [ 26 ]			OmniDocBench-v1.5 [ 26 ]			olmOCRBench [ 28 ]	Avg $\uparrow$
Model	TEDS $\uparrow$	TEDS-S $\uparrow$	Edit-Dist $\downarrow$	TEDS $\uparrow$	TEDS-S $\uparrow$	Edit-Dist $\downarrow$	TA $\uparrow$	Avg $\uparrow$
Baseline
Qwen2.5-VL-7B [ 4 ]	74.4	80.1	58.9	69.3	74.1	58.7	71.1	64.5
InternVL3.5-8B [ 38 ]	71.0	77.6	44.9	62.9	70.4	44.1	74.1	66.7
Qwen3-VL-8B-Instruct [ 3 ]	78.9	83.9	23.2	72.7	77.2	26.9	75.3	76.8
Qwen3-VL-8B-Instruct [ 3 ]

Table 2: Evaluation Results on Table-to-Markdown. TEDS-S denotes TEDS-Structure-Only, and TA represents the table subtask of olmOCRBench. For Edit-Dist, where lower values indicate better performance, we use (

100-\text{Edit-Dist}

) when computing the Average.

Table 7

Proprietary
Model	Chart			Table			SVG			AVG
Model	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$
GPT-4o [ 1 ]	22.8	28.3	48.5	32.9	35.7	49.5	13.0	19.3	50.3	25.0	29.5	56.5
GPT-5.2 [ 35 ]	30.1	32.6	64.8	39.3	40.6	54.6	28.5	32.2	61.1	32.7	35.0	58.9
Gemini-2.5-Pro [ 8 ]	33.7	37.5	61.8	46.4	48.0	49.9	29.3	34.3	63.3	37.8	40.9	59.1
Gemini-3-Flash [ 10 ]	38.5	41.3	62.8	48.1	50.1	45.6	33.3	67.5	64.3	40.6	43.4	53.4

Table 5: Evaluation Results on VC-RewardBench. We evaluate a range of proprietary and open-source models on Visual-ERM-Bench.

F1_{h}

denotes

F1_{hard}

, the strict-match F1 score;

F1_{s}

denotes

F1_{soft}

, the soft-match F1 score; and

S_{c}

denotes the correlation score, measuring the overall agreement between the predicted scores and the ground-truth labels.

Figure 5: Test-Time Scaling Cases. Using Visual-ERM ’s fine-grained feedback, we enable an inference, reflection and refinement loop. Predictions are generated as text and rendered as images for visualization.

Model	ChartMimic [ 41 ] (Direct)				ChartMimic [ 41 ] (Customized)				Avg $\uparrow$
Model	exec_rate $\uparrow$	average $\uparrow$	gpt_score $\uparrow$	overall $\uparrow$	exec_rate $\uparrow$	average $\uparrow$	gpt_score $\uparrow$	overall $\uparrow$	Avg $\uparrow$
Qwen3-VL-8B-Instruct [ 3 ] (+RL on Visual-ERM)	92.5	74.4	84.6	79.5	91.5	71.8	81.1	76.5	78.0
$+$ Reflection ( $w$ Visual-ERM) 2 rounds	92.8	75.2	84.5	79.8	92.0	74.4	82.8	78.6	79.2
$+$ Reflection ( $w$ Visual-ERM) 3 rounds	92.3	74.8	85.8	80.3	94.7	77.1	86.8	82.0	81.1
$+$ Reflection ( $w$ Visual-ERM) 4 rounds	92.0	74.5	85.4	80.0	94.2	76.9	86.1	81.5	80.7

Table 6: Ablation on the number of reflection rounds in test-time scaling. We evaluate test-time scaling with different numbers of reflection and revision rounds d.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Training Data Generation. We construct our Visual-ERM ’s training data as shown in Figure.

Models	ChartQA_TEST	CharXiv_DQ	CharXiv_RQ	DocVQA_VAL	InfoVQA_VAL	AVG
Qwen3-VL-8B-Instruct	82.9	83.8	46.0	95.6	83.1	78.3
$+$ RL on Chart-to-Code	82.6	83.5	47.2	95.6	83.1	78.4
$+$ RL on Table-to-Markdown	81.5	83.8	46.7	95.5	83.1	78.1
$+$ RL on SVG-to-Code	83.2	83.5	46.4	95.8	83.6	78.5

Appendix

Table 7: Results on general VQA benchmarks. We evaluate the policy model trained with Visual-ERM -guided RL on multiple general-purpose benchmarks (with an emphasis on chart- and document-centric VQA) to examine whether its improved vision-to-code performance comes at the cost of general multimodal capability.

Model	Chart			Table			SVG			AVG
Model	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$
Visual-ERM
$+$ Chart-Data-Only	40.1	42.6	58.1	17.2	20.4	10.5	11.3	15.3	50.1	26.3	29.3	23.7
$+$ Table-Data-Only	17.1	20.1	49.3	53.6	54.9	68.8	14.1	19.0	55.2	31.7	34.2	56.6
$+$ SVG-Data-Only	16.4	21.0	56.0	11.4	17.6	37.0	26.4	31.0	57.3	17.5	22.6	44.2
$+$ Mix-Data	39.9	42.8	61.2	56.4	57.6	74.8	28.3	32.6	59.6	42.1	44.7	58.4

Appendix

Table 8: Effect of Multi-Task Data Mixing. We compare reward models trained on mixed multi-task data vs. single-task data.

Baseline
Model	OmniDocBench			OmniDocBench-v1.5			olmOCRBench	Avg $\uparrow$
Model	TEDS $\uparrow$	TEDS-S $\uparrow$	Edit-Dist $\downarrow$	TEDS $\uparrow$	TEDS-S $\uparrow$	Edit-Dist $\downarrow$	TA $\uparrow$	Avg $\uparrow$
Qwen3-VL-8B-Instruct	78.9	83.9	23.2	72.7	77.2	26.9	75.3	76.8
RL with Different Reward Model
+ RM (table-data-only)	79.5	84.4	21.6	74.4	79.4	25.0	79.4	78.6
$\Delta$	+0.6	+0.5	+1.6	+1.7	+2.2	+1.9	4.1	+1.8

Appendix

Table 9: Ablation Study of Multi-Task Data Mixing . We perform RL on the table parsing task using different reward models: one reward model is trained only on table data, while the other reward model is jointly trained on table, chart, and SVG data. TEDS-S denotes TEDS-Structure-Only, and TA represents the table subtask of olmOCRBench. For Edit-Dist, where lower values indicate better performance, we use (

100-\text{Edit-Dist}

) when computing the Average.

Judger	Chart			Table			SVG			AVG
Judger	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$	$F1_{h}$	$F1_{s}$	$S_{c}$
GPT-5-mini	39.9	42.8	61.2	56.4	57.6	74.8	28.3	32.6	59.6	42.1	44.7	58.4
GPT-5.2	40.9	45.4	61.2	55.0	57.1	74.8	24.5	31.6	59.6	40.8	45.1	58.4
Gemini2.5-Pro	40.8	42.1	61.2	54.8	55.7	74.8	27.1	28.7	59.6	41.2	42.4	58.4
Gemini3-Flash	41.5	43.7	61.2	56.5	57.8	74.8	28.5	32.1	59.6	43.0	44.9	58.4

Appendix

Table 10: Ablation on LLM Judges. VC-RewardBench uses LLM-assisted judging to verify answer correctness. To assess the robustness of this evaluation protocol, we compare multiple LLM-as-Judge models and report the resulting consistency across judges.

Model	OmniDocBench			OmniDocBench-v1.5			olmOCRBench	Avg $\uparrow$
Model	TEDS $\uparrow$	TEDS-S $\uparrow$	Edit-Dist $\downarrow$	TEDS $\uparrow$	TEDS-S $\uparrow$	Edit-Dist $\downarrow$	TA $\uparrow$	Avg $\uparrow$
Qwen3-VL-8B-Instruct	78.9	83.9	23.2	72.7	77.2	26.9	75.3	76.8
GRPO (w/o RSR)	80.9	85.6	21.8	75.2	79.9	25.9	79.0	79.0
GRPO (w RSR)	81.4	86.3	20.7	75.4	80.4	24.2	78.1	79.5

Appendix

Table 11: Ablation Study of Reward Design . RSR denotes the Render-Success Reward. TEDS-S denotes TEDS-Structure-Only, and TA represents the table subtask of olmOCRBench. For Edit-Dist, where lower values indicate better performance, we use (

100-\text{Edit-Dist}

) when computing the Average.

Figure 7: Prompt for distillation. The prompt used to distill GPT-5-mini to construct reward-modeling training data for Visual-ERM . We show the Chart-specific prompt here; due to its length, this table includes only the first part.

Figure 8: Prompt for distillation. The prompt used to distill GPT-5-mini to construct reward-modeling training data for Visual-ERM . We show the Chart-specific prompt here; due to its length, this table includes only the second part.

Model	UniSVG(ISVGEN)
Model	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	Score $\uparrow$
Qwen3-VL-8B-Instruct	60.9	60.0	73.3	64.2
$+$ Reflection (Visual-ERM)	60.4	58.1	74.6	65.2
Qwen3-VL-8B-Instruct (+RL on Visual-ERM)	64.3	49.7	77.5	69.4
$+$ Reflection (Visual-ERM)	63.9	49.2	78.1	69.8

Appendix

Table 12: Test-Time Scaling on SVG-to-Code . We evaluate TTS on the SVG-to-Code task. Leveraging Visual-ERM as an evaluator, the model performs multiple rounds of self-reflection and revision, leading to improved parsing performance.

Figure 9: Visual-ERM inference prompt. The prompting template used by Visual-ERM at inference time. This prompt is specialized for the Chart-to-Code setting.

Figure 10: Visual-ERM inference prompt. The prompting template used by Visual-ERM at inference time. This prompt is specialized for the Table-to-Markdown setting.

Figure 11: Visual-ERM inference prompt. The prompting template used by Visual-ERM at inference time. This prompt is specialized for the SVG-to-Code setting.

Figure 12: VC-RewardBench evaluation prompt. For the subjective, free-form error descriptions in VC-RewardBench, we use an LLM-assisted judge to perform error matching and verification; the prompt used in this evaluation protocol is shown here.

Figure 13: Case study. We show representative examples from the Chart-to-Code reward-model training data and the benchmark.

Figure 14: Case study. We show representative examples from the Table-to-Markdown reward-model training data and the benchmark.

Figure 15: Case study. We show representative examples from the SVG-to-Code reward-model training data and the benchmark.

Figure 16: Further Case Analysis. We present additional cases to highlight the limitations of current reward models.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization
Jun 30, 2026Yaozhi Zheng, Yilei Jiang, Manyuan Zhang +5RL for Code Generation Reinforcement Learning
The Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory
Reinforcement Learning from Intermediate Renders for Image-to-Code Generation
Sep 28, 2026Omri Kaduri, Kate Feingold, Phillip Isola +1RL for Code Generation Image-to-Code Generation
Weizmann Institute of Science · MIT
VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?
Aug 11, 2026Mizanur Rahman, Arshia Azimlu, Shadikur Rahman +4VLM Evaluation Data Visualization
‡York University · $Nanyang Technological University · ¶Salesforce AI Research

Jun 30, 2026J/K move · Enter open · S save

UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization

Yaozhi Zheng, Yilei Jiang, Manyuan Zhang, Yuxuan Wan+4

The Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory

RL for Code Generation Reinforcement Learning Reward Shaping Code Generation Multimodal Large Language Models

Open paper AskPDF arXiv

Visual-to-Code generation, which transforms scientific plots, vector graphics, and webpages into executable scripts, demands a level of pixel-precise alignment that standard Multimodal Large Language Models (MLLMs) fail to achieve through Supervised Fine-Tuning (SFT) alone. While Reinforcement Learning (RL) offers a theoretical pathway to bridge this gap, its application is hindered by two fundamental obstacles: (1) Reward Coarseness, where semantic metrics like CLIP scores fail to penalize fine-grained element deviations, and (2) Exploration Stagnation, where the sparse, heterogeneous code search space prevents the policy from bootstrapping valid trajectories. To overcome these limitations, we introduce UniCoder, a unified RL framework that integrates two novel mechanisms. First, we propose Symbolic Attribute Alignment, which employs a lightweight auxiliary LLM to parse generated code into discrete visual attributes (e.g., hex colors, coordinate limits), enabling dense, element-wise reward computation. Second, to escape local optima, we devise Reference-Guided Code Optimization, a strategy that dynamically injects ground-truth trajectories into low-performing rollout groups, transforming blind exploration into guided policy improvement. Extensive experiments on ChartMimic, UniSVG, Design2Code and ScreenBench benchmarks demonstrate that our 8B-parameter model not only surpasses all open-source baselines but also achieves state-of-the-art performance comparable to proprietary models, establishing a new paradigm for generalized visual-to-code synthesis.

Abstract

Reward Modeling

Explore similar work

UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization

Reinforcement Learning from Intermediate Renders for Image-to-Code Generation

VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?