VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents

Hongzhu Yi; Yujia Yang; Yuanxiang Wang; Tong Li; Zhenyu Guan; Tianyu Zong; Jiahuan Chen; Chenxi Bao

cs.CVJan 27, 2026

VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents

Authors: Hongzhu Yi, Yujia Yang, Yuanxiang Wang, Tong Li, Zhenyu Guan, Tianyu Zong, Jiahuan Chen, Chenxi Bao,

Organizations: UCAS · Coserve AI · WashU · SJTU ·

Abstract

Image editing models can modify visual content through natural language, but editing dense documents requires precise local changes amid tightly coupled text, tables, and layout. We introduce VDE Bench, an evaluation resource for studying instruction-guided editing of Chinese and English document images. VDE Bench contains 3,506 instruction-guided image editing samples, including 1,578 text-region edits and 1,928 table-region edits. The benchmark pairs editing instructions and reference images with annotated editing regions and structured OCR outputs. Its evaluation protocol combines region-level and full-page measurements to assess textual correctness, structural fidelity, and visual content preservation. By focusing on the interaction between local edits and their surrounding document context, VDE Bench complements existing visual text editing benchmarks and provides a resource for diagnosing document-specific editing failures.

Figures & tables

Name and Reference	Turns	Human	Text	Mask	Problem
		Verified	Edited		Framing
I2EBench ( Ma et al., 2024 )	$\bullet$ single	✓	✗	✗	$\left\}\vbox{\vrule height=0.0pt,width=0.0pt}\textnormal{\parbox{51.21504pt}{\raggedright{General image editing}\@add@raggedright}}\right.$
EditBench ( Wang et al., 2023a )	$\bullet$ single	✓	✓	✓
EditVal ( Basu et al., 2023 )	$\bullet$ single	✓	✗	✗
EmuEdit ( Sheynin et al., 2024 )	$\bullet$ single	✓	✗	✗
AnyEdit ( Yu et al., 2025a )	$\bullet$ single	✓	✗	✓

Table 1: Comparison of VDE Bench with existing image editing benchmarks. VDE Bench focuses on complex document editing with human-verified references and editing-region annotations. Multi denotes inclusion of multi-turn evaluation; Text Edited includes text rendering or editing;

\times

denotes an unsupported or unreported feature.

Figure 2: Overview of the VDE Bench construction pipeline and a sample annotation. The pipeline consists of four stages: (1) collecting documents across multiple categories and languages; (2) generating editing instructions with Gemini3-Pro, Qwen3-VL, and GPT-5.6-sol and producing edited images with Nano Banana Pro, GPT-Image-2.5-Sunburt, or direct LaTeX rendering; (3) human review and annotation by 18 annotators, including multidimensional scoring, IoU-based region verification, and subjective quality filtering; (4) structured OCR processing with PaddleOCR-VL followed by human verification. VDE Bench contains 3,506 high-quality instruction-based editing samples with rich structural annotations; an example is shown on the right.

Figure 3: GT page layout density in VDE Bench. Each cell shows $z_{s,c}=(d_{s,c}-\mu_{c})/\sigma_{c}$ . For each page in source group $s$ , we sum the bounding-box areas of all elements in category $c$ and divide by the page area; $d_{s,c}$ is the mean of this ratio across pages. For each element category, the mean and standard deviation are computed across the same 25 document categories as in Figure 4 .

Figure 4: Dataset profile of VDE Bench. Top: document-source distribution, with academic literature subdivided by research topic. Sector angles indicate source proportions. Bottom: editing operations, languages, and editing regions. Labels report counts and percentages. Mixed denotes Chinese–English documents.

	OCR metrics $\uparrow$					Image metrics
Model	IoU	TextSim	CDM	BLEU	TEDS	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	FID $\downarrow$
Global ( $n=3{,}506$ )
Step1X	0.830	0.615	0.634	0.304	0.552	20.096	0.859	0.230	0.956	5.763
LongCat	0.876	0.779	0.794	0.410	0.780	18.792	0.861	0.127	0.966	5.018
FireRed	0.856	0.847	0.862	0.485	0.789	20.803	0.883	0.112	0.970	5.273
Qwen	0.901	0.891	0.897	0.541	0.857	24.279	0.936	0.082	0.986	1.387

Table 2: Main evaluation results on VDE Bench. Each model covers the same samples in both settings. Higher is better except LPIPS and FID (

\downarrow

). Blue and pink shading indicate better editing-model scores; bold marks the best editing model per column.

	English ( $n=2{,}602$ )					Chinese ( $n=728$ )
Model	IoU	CDM	BLEU	TEDS	Avg.	IoU	CDM	BLEU	TEDS	Avg.
Step1X	0.767	0.609	0.303	0.550	0.557	0.762	0.475	0.103	0.460	0.450
LongCat	0.801	0.798	0.519	0.772	0.723	0.851	0.786	0.192	0.838	0.667
FireRed	0.771	0.838	0.581	0.756	0.737	0.855	0.830	0.209	0.827	0.680
Qwen	0.824	0.880	0.676	0.848	0.807	0.867	0.846	0.258	0.895	0.716
Instruct	0.584	0.187	0.069	0.152	0.248	0.399	0.167	0.072	0.135	0.193

Table 3: Results by language on VDE Bench. Each score averages the local and global settings and is reported to three decimal places; Avg. is the mean of the four OCR metrics. The 176 mixed-language samples are excluded here but included in the overall and per-operation results. Deeper teal indicates higher scores, amber highlights averages, and bold marks the best result in each column.

Figure 7: Performance by edit type on 3,506 samples. Scores average the separately aggregated global and local results. Modification combines replacement and modification; addition and deletion include text and table regions. Colors distinguish metrics; labels mark the best model for each metric and operation. Avg. is the seven-model mean. Labels are rounded to two decimal places.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Document category	Chinese							English							Mixed							Total
	Text			Table				Text			Table				Text			Table
	M	A	D	M	A	D	S	M	A	D	M	A	D	S	M	A	D	M	A	D	S
Financial reports	0	0	0	0	0	0	0	0	59	57	61	58	58	43	0	0	0	0	0	0	0	336
Biomedical informatics	20	20	20	19	19	19	19	20	20	20	19	19	19	19	0	0	0	0	0	0	0	272
Social NLP & fairness	0	0	0	0	0	0	0	0	33	36	26	46	46	49	0	0	0	0	0	0	0	236
Dialogue & speech	0	0	0	0	0	0	0	0	30	38	30	42	38	41	0	0	0	0	0	0	0	219

Appendix

Table 4: Joint counts for all 3,506 samples by document category, language, region, and operation. Categories follow Figure 4 , with academic literature split into ten subcategories, and are ordered by sample count. M/A/D/S denote modification/addition/deletion/structure; modification combines text replacement and table text modification. Mixed denotes Chinese–English documents; Others refers only to the document-source label.

Figure 8: Layout-element composition on all 3,506 edited GT pages. Rows follow the 25 document categories in Figure 4 , pool detected blocks within each category, and sum to 100%; parentheses give page counts. Cell labels show rounded shares of at least 1%. Newspapers have the highest text-block share; academic subcategories are aggregated separately to show their distinct compositions.

Figure 9: Comprehensive layout statistics on the same 3,506 GT pages. Cells report actual values; colors use column-wise min–max normalization. Text and visual ratios are percentages. Newspapers have the highest area density and block richness, research reports the greatest type complexity, and other academic sources the highest entropy. Visual categories comprise tables, images, charts, and header/footer images.

Figure 10: (a) Agreement among 18 annotators on 50 pilot samples, measured by Krippendorff’s

\alpha

with 95% bootstrap confidence intervals. (b,c) Human versus automated rankings of seven models for box alignment and text correctness, based on human evaluation of 1,000 sampled images. Each point represents one model; ranks range from 1 (best) to 7 (worst). Both comparisons yield Spearman’s

\rho=0.964

		OCR metrics $\uparrow$
Model	$n$	IoU	TextSim	CDM	BLEU	TEDS
All available pairs
Step1X	3,498	0.6424	0.4666	0.5520	0.1998	0.5039
LongCat	3,487	0.6817	0.7388	0.7925	0.4571	0.7089
FireRed	3,492	0.6529	0.7411	0.7812	0.4574	0.6573
Qwen	3,489	0.6781	0.8036	0.8132	0.5481	0.7541

Appendix

Table 5: Local evaluation with Hunyuan-OCR. The upper panel uses all available reference/output pairs for each model; the lower panel uses the same 3,469 samples for all seven models.

n

counts image pairs. Darker blue indicates higher scores within each metric and panel; bold marks the best score. All metrics are higher-is-better. Rank correlations compare the common-subset results with the PaddleOCR-VL main table on 3,506 samples.

		OCR metrics $\uparrow$
Model	$n$	IoU	TextSim	CDM	BLEU	TEDS
All available pairs
Step1X	3,498	0.6708	0.6840	0.7102	0.2995	0.5222
LongCat	3,487	0.7107	0.8195	0.8282	0.4109	0.7067
FireRed	3,492	0.6764	0.8662	0.8678	0.4733	0.6840
Qwen	3,489	0.7265	0.8902	0.8862	0.5056	0.7453

Appendix

Table 6: Global evaluation with Hunyuan-OCR. The upper panel uses all available reference/output pairs for each model; the lower panel uses the same 3,469 samples for all seven models.

n

Metric	Step1X	LongCat	FireRed	Qwen	Instruct	HiDream	JoyAI	Avg.
Text Modify (1099 samples)
IoU	0.832	0.913	0.896	0.905	0.392	0.589	0.719	0.750
CDM	0.605	0.736	0.812	0.850	0.213	0.748	0.751	0.674
BLEU	0.274	0.344	0.397	0.439	0.120	0.322	0.366	0.323
TEDS	0.558	0.749	0.793	0.833	0.148	0.602	0.764	0.635
Text Addition (976 samples)

Appendix

Table 7: Performance by edit type on 3,506 samples. All scores use the global-image setting. Text Modify combines replacement and modification; addition and deletion include text and table regions. Avg. is the mean over the seven editing models. Teal and bold mark the best editing model per row.

Metric	Step1X	LongCat	FireRed	Qwen	Instruct	HiDream	JoyAI	Avg.
Text Modify (1099 samples)
IoU	0.797	0.862	0.899	0.876	0.592	0.645	0.772	0.778
CDM	0.567	0.840	0.886	0.857	0.193	0.675	0.861	0.697
BLEU	0.188	0.363	0.406	0.459	0.049	0.290	0.406	0.309
TEDS	0.584	0.812	0.830	0.877	0.153	0.605	0.824	0.669
Text Addition (976 samples)

Appendix

Table 8: Performance by edit type on 3,506 samples. All scores use the local-image setting. Text Modify combines replacement and modification; addition and deletion include text and table regions. Avg. is the mean over the seven editing models. Teal and bold mark the best editing model per row.

Image Editing Evaluation

Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.

See all 92 papers

GEditBench v2: A Human-Aligned Benchmark for General Image Editing
Mar 30, 2026Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7Image Editing Evaluation Text-Guided Image Editing
Nanyang Technological University · StepFun · Southeast University
Reasoning-Informed Visual Editing
Oct 8, 2026Xue Yang, Peiyuan Zhang, Yilun Zhu +12Image Editing Image Editing Evaluation
Shanghai Jiao Tong University · Southeast University · South China University of Technology +3
UniEditBench: A Unified and Cost-Effective Benchmark for Image and Video Editing via Distilled MLLMs
Apr 17, 2026Lifan Jiang, Tianrun Wu, Yuhang Pei +3Video Editing Benchmarks Image Editing Evaluation
Zhejiang University Hangzhou, China

Mar 30, 2026J/K move · Enter open · S save

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Zhangqi Jiang, Zheng Sun, Xianfang Zeng, Yufeng Yang+6

Nanyang Technological University · StepFun · Southeast University

Image Editing Evaluation Text-Guided Image Editing Vision-Language Model-as-a-Judge

Open paper AskPDF arXiv

Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.

Abstract

Image Editing Evaluation

Explore similar work

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Reasoning-Informed Visual Editing

UniEditBench: A Unified and Cost-Effective Benchmark for Image and Video Editing via Distilled MLLMs