Evaluating Image Editing with LLMs: A Comprehensive Benchmark and Intermediate-Layer Probing Approach

Shiqi Gao; Zitong Xu; Kang Fu; Huiyu Duan; Xiongkuo Min; Jia wang

cs.CVMar 20, 2026

Evaluating Image Editing with LLMs: A Comprehensive Benchmark and Intermediate-Layer Probing Approach

Authors: Shiqi Gao, Zitong Xu, Kang Fu, Huiyu Duan, Xiongkuo Min, Jia wang

Organizations: Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai 200240, China

Abstract

Evaluating text-guided image editing (TIE) methods remains a challenging problem, as reliable assessment should simultaneously consider perceptual quality, alignment with textual instructions, and preservation of original image content. Despite rapid progress in TIE models, existing evaluation benchmarks remain limited in scale and often show weak correlation with human perceptual judgments. In this work, we introduce TIEdit, a benchmark for systematic evaluation of text-guided image editing methods. TIEdit consists of 512 source images paired with editing prompts across eight representative editing tasks, producing 5,120 edited images generated by ten state-of-the-art TIE models. To obtain reliable subjective ratings, 20 experts are recruited to produce 307,200 raw subjective ratings, which accumulates into 15,360 mean opinion scores (MOSs) across three evaluation dimensions: perceptual quality, editing alignment, and content preservation. Beyond the benchmark itself, we further propose EditProbe, an LLM-based evaluator that estimates editing quality via intermediate-layer probing of hidden representations. Instead of relying solely on final model outputs, EditProbe extracts informative representations from intermediate layers of multimodal large language models to better capture semantic and perceptual relationships between source images, editing instructions, and edited results. Experimental results demonstrate that widely used automatic evaluation metrics show limited correlation with human judgments on editing tasks, while EditProbe achieves substantially stronger alignment with human perception. Together, TIEdit and EditProbe provide a foundation for more reliable and perceptually aligned evaluation of text-guided image editing methods.

Figures & tables

Fig. 1: Overview of the proposed TIEdit benchmark and the EditProbe evaluation method. (a) We collect 512 source images and design editing prompts covering eight representative editing tasks. Ten text-guided image editing models are applied to generate 5,120 edited images. (b) 307,200 human ratings are collected from 20 expert annotators across three evaluation dimensions and processed to obtain 15,360 MOSs. (c) We analyze the benchmark from three perspectives: task-wise, model-wise, and dimension-wise. (d) We propose EditProbe to predict editing quality through intermediate-layer probing of a multimodal LLM.

Task	Description	Scope
Object Addition	Insert a new object into the scene.	Local
Object Removal	Remove an object and restore the background.	Local
Object Replacement	Replace an object with another one.	Local
Appearance Modification	Change object color, material, or texture.	Local
Action Modification	Modify the pose or action of a subject.	Local
Emotion Modification	Change facial expression or affective cues.	Local

TABLE I: Summary of the eight editing tasks defined in the TIEdit benchmark.

Fig. 2: Distribution of MOSs across the three evaluation dimensions in the TIEdit benchmark. The histogram and kernel density curves illustrate the global score distributions for perceptual quality, editing alignment, and content preservation. Perceptual quality and content preservation generally concentrate at higher score ranges, while editing alignment exhibits a broader distribution, indicating greater difficulty in accurately following editing instructions.

Fig. 3: MOS distributions across different editing tasks in the TIEdit benchmark. Each boxplot summarizes the subjective scores for perceptual quality, editing alignment, and content preservation within a specific editing task. Tasks involving complex semantic transformations, such as action modification and emotion modification, tend to exhibit lower editing alignment scores, indicating increased editing difficulty.

Fig. 4: MOS distributions across different text-guided image editing models. The boxplots illustrate the performance variations of ten representative editing models across the three evaluation dimensions. Clear performance differences can be observed among models, reflecting the diversity of editing capabilities across different editing paradigms.

Fig. 5: Dimension-wise MOS comparison across editing models and tasks. Each heatmap shows the average MOS values for perceptual quality, editing alignment, and content preservation across different model–task pairs. The results reveal that editing alignment varies significantly across models and tasks, whereas perceptual quality and content preservation remain relatively stable.

Fig. 6: Overview of EditProbe. The pipeline consists of two stages. In Stage 1, layer-wise analysis is performed on the pre-trained MLLM to identify the most informative transformer layer for editing evaluation. In Stage 2, the selected layer is used for intermediate-layer probing, and a lightweight regression head is trained to predict MOS scores.

Fig. 7: Layer-wise analysis of hidden state properties in the pre-trained multimodal large language model (InternVL3) on the TIEdit benchmark. KL divergence, layer discriminability ratio (LDR), information entropy, and saliency score exhibit consistent trends, with their values peaking in the intermediate transformer layers, indicating that these layers capture the most informative representations for text-guided image editing evaluation.

Methods/Metrics	Perceptual Quality ( $MOS_{q}$ )			Editing Alignment ( $MOS_{e}$ )			Content Preservation ( $MOS_{p}$ )			Overall Score ( $MOS_{o}$ )
Methods/Metrics	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
$\spadesuit$ FSIM [ 42 ]	0.3906	0.3514	0.2620	0.2061	0.1419	0.0776	0.0134	0.0509	0.1152	0.1794	0.2437	0.3079
$\spadesuit$ FSIMc [ 42 ]	0.3896	0.3495	0.2617	0.2056	0.1417	0.0777	0.0137	0.0503	0.1142	0.1782	0.2422	0.3062
$\spadesuit$ IFC [ 43 ]	0.1624	0.3137	0.1163	0.1514	0.1283	0.1053	0.0822	0.0592	0.0361	0.0131	0.0100	0.0330
$\spadesuit$ IW-MSE [ 72 ]	0.3976	0.2298	0.2666	0.0138	0.0517	0.1172	0.1827	0.2482	0.3137	0.3792	0.4446	0.5101
$\spadesuit$ IW-PSNR [ 72 ]	0.3976	0.3751	0.2666	0.2155	0.1500	0.0845	0.0190	0.0465	0.1120	0.1775	0.2430	0.3085

TABLE II: Performance comparison of automatic evaluation methods on the TIEdit benchmark from four perspectives: perceptual quality, editing alignment, content preservation, and overall score. SRCC, PLCC, KRCC are reported to measure the correlation between metric predictions and human MOSs, where higher values indicate better agreement with human perceptual judgments.

\spadesuit

denotes traditional full-reference (FR) IQA metrics,

\heartsuit

denotes traditional no-reference (NR) IQA metrics,

\bigstar

denotes vision–language and preference-based metrics,

\diamondsuit

denotes deep learning-based IQA models, and

\blacktriangle

denotes LLM-based evaluators. The best results in each column are highlighted in bold , and the second-best results are underlined . The last row reports the relative improvement of EditProbe over the second-best result in each column.

Methods	Perceptual Quality ( $MOS_{q}$ )			Editing Alignment ( $MOS_{e}$ )			Content Preservation ( $MOS_{p}$ )			Overall Score ( $MOS_{o}$ )
Methods	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
w/o MLP	0.6594	0.6644	0.4764	0.7624	0.7464	0.5588	0.7205	0.7522	0.5374	0.7296	0.7335	0.5407
LoRA (only LLM)	0.6775	0.6876	0.4932	0.7814	0.7712	0.5778	0.7456	0.7673	0.5630	0.7567	0.7567	0.5671
LoRA (only Vision)	0.6899	0.6994	0.5039	0.7926	0.7833	0.5908	0.7656	0.7826	0.5822	0.7708	0.7737	0.5833
Final Layer	0.6953	0.7056	0.5087	0.7967	0.7876	0.5939	0.7710	0.7888	0.5876	0.7771	0.7794	0.5897
EditProbe	0.6959	0.7064	0.5095	0.7995	0.7910	0.5974	0.7715	0.7892	0.5887	0.7795	0.7807	0.5913

TABLE III: Ablation study of different design choices in EditProbe on the TIEdit benchmark from four perspectives: perceptual quality, editing alignment, content preservation, and overall score. SRCC, PLCC, and KRCC measure the correlation between metric predictions and human MOSs, where higher values indicate better agreement. The best results in each column are highlighted in bold , and the second-best results are underlined .

Explore similar work

Mar 16, 2026

EditHF-1M: A Million-Scale Rich Human Preference Feedback for Image Editing

Recent text-guided image editing (TIE) models have achieved remarkable progress, while many edited images still suffer from issues such as artifacts, unexpected editings, unaesthetic contents. Although some benchmarks and methods have been proposed for evaluating edited images, scalable evaluation models are still lacking, which limits the development of human feedback reward models for image editing. To address the challenges, we first introduce EditHF-1M, a million-scale image editing dataset with over 29M human preference pairs and 148K human mean opinion ratings, both evaluated from three dimensions, i.e., visual quality, instruction alignment, and attribute preservation. Based on EditHF-1M, we propose EditHF, a multimodal large language model (MLLM) based evaluation model, to provide human-aligned feedback from image editing. Finally, we introduce EditHF-Reward, which utilizes EditHF as the reward signal to optimize the text-guided image editing models through reinforcement learning. Extensive experiments show that EditHF achieves superior alignment with human preferences and demonstrates strong generalization on other datasets. Furthermore, we fine-tune the Qwen-Image-Edit using EditHF-Reward, achieving significant performance improvements, which demonstrates the ability of EditHF to serve as a reward model to scale-up the image editing. Both the dataset and code will be released in our GitHub repository: https://github.com/IntMeGroup/EditHF.

Zitong Xu, Huiyu Duan, Zhongpeng Ji +9

Shanghai Jiao Tong University · Vivo Mobile Communication Co., Ltd · University of Electronic and Science Technology of China +2

Image Editing Evaluation Text-Guided Image Editing Human Preference Modeling Vision-Language Model-as-a-Judge RL for Image Editing

Ask PDF

May 8, 2026

ReasonEdit: Towards Interpretable Image Editing Evaluation via Reinforcement Learning

Recent text-guided image editing (TIE) models have achieved remarkable progress, however, many edited results still suffer from artifacts, unintended modifications, and suboptimal aesthetics. Although several benchmarks and evaluation methods have been proposed, most existing approaches rely on scalar scores and lack interpretability. This limitation largely stems from the absence of high-quality interpretation datasets for TIE and effective reward models to train interpretable evaluators. To address these challenges, we introduce ReasonEdit-22K, the first dataset that combines 22K edited images with 113K Chain-of-Thought (CoT) samples, along with 1.3M human judgments assessing these interpretations in terms of logicality, accuracy, and usefulness. Building upon this dataset, we propose RE-Reward, a multimodal large language model (MLLM)-based reward model designed to provide human-aligned feedback for evaluating interpretable reasoning in image editing. Furthermore, we develop ReasonEdit, which is trained using reward signals derived from RE-Reward and the Group Relative Policy Optimization (GRPO) algorithm to learn an interpretable evaluation model. Extensive experiments demonstrate that ReasonEdit achieves superior alignment with human preferences and exhibits strong generalization across public benchmarks. In addition, it is capable of generating high-quality interpretable evaluation text, enabling more transparent and trustworthy assessment for image editing. The code is available at https://github.com/IntMeGroup/ReasonEdit.

Honghua Chen, Zitong Xu, Huiyu Duan +3

University of Electronic Science and Technology of China · 2Shanghai Jiao Tong University

Image Editing Evaluation Text-Guided Image Editing Multimodal Reward Modeling

Ask PDF

Mar 30, 2026

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.

Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7

Nanyang Technological University · StepFun · Southeast University

Image Editing Evaluation Text-Guided Image Editing Vision-Language Model-as-a-Judge

Ask PDF