Position: Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered

Jinfan Hu; Fanghua Yu; Zhiyuan You; Xiang Yin; Hongyu An; Xinqi Lin; Chao Dong; Jinjin Gu

cs.CVFeb 28, 2026

Position: Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered

Authors: Jinfan Hu, Fanghua Yu, Zhiyuan You, Xiang Yin, Hongyu An, Xinqi Lin, Chao Dong, Jinjin Gu

Organizations: Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of the Chinese Academy of Sciences · The University of Hong Kong · Shenzhen Loop Area Institute · The Chinese University of Hong Kong · Fudan University · INSAIT, Sofia University “St. Kliment Ohridski” · Shenzhen University of Advanced Technology · Shanghai AI Laboratory

Abstract

This position paper argues that the evaluation of modern visual processing systems should no longer be driven primarily by single-metric image quality assessment benchmarks, particularly in the era of generative and perception-oriented methods. Image restoration exemplifies this divergence: while objective IQA metrics enable reproducible, scalable evaluation, they have increasingly drifted apart from human perception and user preferences. We contend that this mismatch risks constraining innovation and misguiding research progress across visual processing tasks. Rather than rejecting metrics altogether, this paper calls for a rebalancing of evaluation paradigms, advocating a more human-centered, context-aware, and fine-grained approach to assessing the visual models' outcomes.

Figures & tables

Figure 1: Trend of full-reference IQA metrics (PSNR, SSIM, LPIPS). The percentages shown below each category represent the corresponding winning rates among those categories. “Best Category Mean” represents the mean value of the optimal model performance of each category.

Figure 2: Metrics such as PSNR, SSIM, and LPIPS often fail to accurately reflect perceptual image quality. Higher values indicate better performance for PSNR and SSIM, while lower values are preferred for LPIPS. The best result for each metric across different methods is highlighted in red. Zoom in for a better observation.

Figure 3: Trend of NR-IQA metrics (MUSIQ, MANIQA, CLIP-IQA). The percentages shown below each category represent the corresponding winning rates among those categories. “Best Category Mean” represents the mean value of the optimal model performance of each category.

Figure 4: Simple image manipulations can artificially boost NR IQA metrics, highlighting their vulnerability to manipulation.

Figure 5: User preference for model performance varies across different semantic scenarios.

Figure 6: Overall IQA scores fail to provide fine-grained evaluation; over-sharpening and over-generation can artificially inflate MUSIQ, MANIQA, and CLIP-IQA scores.

Model	Datasets (amount of images)	Training Resolution	Architecture
MUSIQ	PaQ-2-PiQ (120k), SPAQ (11k), KonIQ (10k), AVA (250k)	Multi-scale (Orig, 224, 384)	Transformer (27M)
MANIQA	PIPAL (23k), LIVE (1.2k), CSIQ (0.9k), TID2013 (3k), KADID (10k)	$224\times 224$ (Crop)	ViT-B/8 + TAB (135M)
CLIP-IQA	CLIP-400M (Pre-train), KonIQ-10k (8k fine-tune)	Native (ARP)	CLIP-RN50 ( 38M)
Q-ALIGN	KonIQ, SPAQ, KADID, AVA (236k), LSVQ (28k video)	Fixed $448\times 448$	LLaMA-2-7B (7B)
DeQA	KonIQ, SPAQ, KADID, PIPAL	Fixed $448\times 448$	LLaMA-2-7B (7B)
SUPIR	20M HQ images, 70k FFHQ-raw, 100k Negative samples	$512\times 512$	SDXL + Adapter (3B+)

Table 2: Comparison of Recent IQA and Restoration Models: Data, Resolution, and Architecture Scale

Figure 7: Current IQA metrics tend to favor globally increased high-frequency content, failing to recognize context-dependent blur such as depth-of-field and motion blur. In some cases, no-reference metrics even assign higher scores than GT images.

Explore similar work

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Jun 26, 2026Yana Wei, Hongbo Peng, Yanlin Lai +14Rubric-Based Evaluation Automated Evaluation
Johns Hopkins University · StepFun · Tsinghua University +1
APEX: Assumption-free Projection-based Embedding eXamination Metric for Generative Distribution Evaluation
May 8, 2026Caterina Gallegati, Monica Bianchini, Franco Scarselli +2Image Quality Assessment Image Generation Evaluation
University of Siena, Italy · AI for Good (AIGO), Istituto Italiano di Tecnologia, Italy · University of Verona, Italy
An Evolutionary Agentic Approach for Open-ended Image Quality Perception
Sep 19, 2026Zhenchen Tang, Bo Peng, Zichuan Wang +4Image Quality Assessment Vision-Language Model-as-a-Judge
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China +1

Jun 26, 2026J/K move · Enter open · S save

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

Yana Wei, Hongbo Peng, Yanlin Lai, Liang Zhao+13

Johns Hopkins University · StepFun · Tsinghua University+1

Rubric-Based Evaluation Automated Evaluation Multimodal Model Evaluation Fine-Grained Visual Perception

Open paper AskPDF arXiv

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 10,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.

Abstract

Explore similar work

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

APEX: Assumption-free Projection-based Embedding eXamination Metric for Generative Distribution Evaluation

An Evolutionary Agentic Approach for Open-ended Image Quality Perception