SIQA: Toward Reliable Scientific Image Quality Assessment

Wenzhe Li; Liang Chen; Junying Wang; Yijing Guo; Ye Shen; Farong Wen; Chunyi Li; Zicheng Zhang

cs.CVMar 5, 2026

SIQA: Toward Reliable Scientific Image Quality Assessment

Authors: Wenzhe Li, Liang Chen, Junying Wang, Yijing Guo, Ye Shen, Farong Wen, Chunyi Li, Zicheng Zhang, +1 more

Organizations: TongJi University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University

Abstract

Scientific images fundamentally differ from natural and AI-generated images in that they encode structured domain knowledge rather than merely depict visual scenes. Assessing their quality therefore requires evaluating not only perceptual fidelity but also scientific correctness and logical completeness. However, existing image quality assessment (IQA) paradigms primarily focus on perceptual distortions or image-text alignment, implicitly assuming that depicted content is factually valid. This assumption breaks down in scientific contexts, where visually plausible figures may still contain conceptual errors or incomplete reasoning. To address this gap, we introduce Scientific Image Quality Assessment (SIQA), a framework that models scientific image quality along two complementary dimensions: Knowledge (Scientific Validity and Scientific Completeness) and Perception (Cognitive Clarity and Disciplinary Conformity). To operationalize this formulation, we design two evaluation protocols: SIQA-U (Understanding), which measures semantic comprehension of scientific content through multiple-choice tasks, and SIQA-S (Scoring), which evaluates alignment with expert quality judgments. We further construct the SIQA Challenge, consisting of an expert-annotated benchmark and a large-scale training set. Experiments across representative multimodal large language models (MLLMs) reveal a consistent discrepancy between scoring alignment and scientific understanding. While models can achieve strong agreement with expert ratings under SIQA-S, their performance on SIQA-U remains substantially lower. Fine-tuning improves both metrics, yet gains in scoring consistently outpace improvements in understanding. These results suggest that rating consistency alone may not reliably reflect scientific comprehension, underscoring the necessity of multidimensional evaluation for scientific image quality assessment.

Figures & tables

Figure 1: Comparison of quality assessment methods across image domains. Traditional IQA focuses on perceptual fidelity (e.g., PSNR, SSIM), while AIGC evaluation emphasizes image-text alignment (e.g., FID, CLIP Score). In contrast, scientific images require joint evaluation of perceptual clarity and semantic correctness, but no dedicated framework currently exists.

Dataset	Domain	Understand	Questions Type	Score	Ratings Type
LIVE Challenge [ 10 ]	NSI	✗	—	✓	MOS
SPAQ [ 8 ]	NSI	✗	—	✓	MOS
CGIQA-6k [ 42 ]	CGI	✗	—	✓	MOS
AGIQA-3k [ 16 ]	AIGC	✗	—	✓	MOS
GenExam [ 35 ]	AIGC	✓	Knowledge QA	✗	—
ChemVLM [ 17 ]	Chemistry	✓	Scientific Reasoning	✗	—

Table 1: Comparison of datasets across image domains, understanding tasks, and quality scoring. Perc. is perception while Know. is knowledge.

Figure 2: Quantitative and qualitative evaluation of SIQA on multiple models. Left and Middle: Case studies illustrating SIQA-U and SIQA-S protocols across question types. Right: Radar map showing performance gains from fine-tuning.

Figure 3: Examples from the SIQA Challenge, illustrating its design and annotation scheme. (a) Sample questions from SIQA-U across four dimensions. (b) Example of different levels in SIQA-S. (c) Comparison of Perception and Knowledge scores across image-quality levels, illustrating their partial correlation and structural distinction.

Figure 4: The construction pipeline of the SIQA Challenge, detailing image sourcing, MCQ generation, and annotation strategies. By combining MLLM-driven generation with rigorous human expert validation, the pipeline ensures high-quality data construction, yielding both a large-scale Training Set for SFT and a strictly isolated Benchmark for robust performance evaluation.

Figure 5: Statistical properties of the SIQA benchmark. (a) Question length distribution in SIQA-U, (b) Word cloud visualizing domain-specific terminology in SIQA-U, (c) Score distributions for Perceptual Quality and Knowledge-based Quality in SIQA-S, (d) Joint distribution of Perceptual Quality and Knowledge-based Quality scores in SIQA-S, revealing their correlation structure.

Closed-Source
Model	Comp.	Val.	Conf.	Clar.	Yes/No	What	How	Overall
O3 [ 28 ]	0.424	0.434	0.465	0.434	0.417	0.648	0.255	0.435
GPT-5 [ 27 ]	0.441	0.455	0.479	0.455	0.448	0.677	0.243	0.450
GPT-4o [ 14 ]	0.427	0.421	0.477	0.421	0.426	0.612	0.283	0.436
GPT-3.5-turbo [ 26 ]	0.399	0.460	0.459	0.460	0.444	0.595	0.271	0.432
Gemini-2.5-Pro [ 7 ]	0.448	0.482	0.479	0.482	0.462	0.700	0.256	0.467

Table 2: Performance of MLLMs on the SIQA-U benchmark. Results are reported as accuracy across four scientific quality dimensions: Scientific Completeness (Comp.), Scientific Validity (Val.), Disciplinary Conformity (Conf.), and Cognitive Clarity (Clar.), and three question types (Yes/No, What, How). Bold : best; Underline : second-best.

Figure 6: Answer distributions of different MLLMs on SIQA-U.

Closed-Source MLLM (zero-shot)
Dim	Perception		Knowledge		Overall
Model	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC
GPT-4o [ 14 ]	0.694	0.699	0.836	0.850	0.765	0.775
GPT-3.5-turbo [ 26 ]	0.480	0.425	0.699	0.665	0.589	0.545
Doubao-1.5-Vision-Pro [ 5 ]	0.596	0.669	0.799	0.845	0.698	0.757
Open-Source MLLM (zero-shot)

Table 3: Performance of various models on the SIQA-S benchmark. Correlation metrics (SRCC&PLCC) are reported for Perception and Knowledge and the average as SIQA-S. Bold : best; Underline : second-best.

Panel of Figure 7 — Figure 7: Evaluation of SIQA across MLLMs. Points are color-coded by model family. Results show a positive correlation between SIQA-U and SIQA-S.

Qwen2.5-VL [ 2 ]
Dimension	Perception				Knowledge				Overall
Model	SIQA-U		SIQA-S		SIQA-U		SIQA-S		SIQA-U		SIQA-S
Model	Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post
Qwen2.5-VL-32B	0.469	0.545	0.616	0.899	0.479	0.542	0.807	0.928	0.474	0.543	0.712	0.913
Qwen2.5-VL-7B	0.442	0.543	0.605	0.896	0.446	0.557	0.771	0.927	0.444	0.551	0.688	0.911
Qwen2.5-VL-3B	0.441	0.524	0.514	0.891	0.467	0.560	0.511	0.924	0.455	0.543	0.513	0.907

Table 4: Effect of model architecture and scale on fine-tuning to SIQA-Judger . We report SIQA-U accuracy (ACC) and SIQA-S correlation metrics (average of SRCC and PLCC) before (Pre) and after (Post) fine-tuning on the SIQA trainset.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Data Source	Whole	Select	Bench	Platform	URL
BMMR [ 40 ]	88,991	5,000	590	Hugging Face	https://huggingface.co/datasets/guanyu615/BMMR
ChenVLM [ 17 ]	1,689	1,689	280	Hugging Face	https://huggingface.co/datasets/Duke-de-Artois/ChemVLM_test_data
EuroSAT [ 12 ]	27,000	2,000	245	Github	https://github.com/phelber/EuroSAT
GenExam [ 35 ]	1,000	1,000	113	Hugging Face	https://huggingface.co/datasets/OpenGVLab/GenExam
GeoTrust [ 9 ]	3,040	3,040	218	Hugging Face	https://huggingface.co/datasets/InternScience/GeoTrust
PEACE [ 13 ]	124	124	16	Hugging Face	https://huggingface.co/datasets/microsoft/PEACE

Appendix

Table 5: Summary of data sources in the SIQA benchmark.

Panel of Figure 8 — Figure 8: Prompt for SIQA Dimensions

Panel of Figure 9 — Figure 9: Prompt for Question Types

Panel of Figure 10 — Figure 10: Prompt for Generating Image Descriptions

Panel of Figure 11 — Figure 11: Prompt for Generating SIQA-U MCQ Pairs

Model	Yes-or-No	How	What
qwen-vl-max-no-thinking	48.4	38.2	56.4
gpt-4o	44.6	37.0	57.4
doubao-1.5-thinking	46.9	39.9	56.1
gemmi-thinking	48.1	49.4	63.6

Appendix

Table 6: Accuracy (%) of four MLLMs on the refined SIQA-U benchmark.

Figure 12: The SIQA-S annotation interface, featuring two-dimensional scoring, a knowledge explanation panel, and efficient navigation controls.

Panel of Figure 13 — Figure 13: SFT conversion logic for SIQA-U (MCQ Format).

Panel of Figure 14 — Figure 14: SFT conversion logic for SIQA-S (Disentangled Rating Format).

Method	Architecture	Training on SIQA	Output Strategy
NIQE	Hand-crafted	No (Zero-Shot)	1 Score (Mapped to 2 dims)
Other MLLMs	MLLM	No (Zero-Shot)	Dynamic (MCQ or 2 Scores)
Q-Align	MLLM	No (Zero-Shot)	2 Scores (Queried Twice)
CLIP-IQA	CLIP-based	No (Zero-Shot)	2 Scores (Queried Twice)
CLIP-IQA+	CLIP-based	Yes (Fine-tuned)	2 Scores (Joint Head)
HyperIQA	CNN	Yes (Trained)	2 Scores (Joint Head)

Appendix

Table 7: Summary of baseline experimental settings.

Explore similar work

Aug 6, 2026

SciQNet: Two-Stage Multimodal Adaptation for Scientific Image Quality Assessment

Scientific images are essential for communicating experimental observations, quantitative evidence and conceptual knowledge. Unlike natural images, their quality depends on both visual clarity and scientific informativeness, making assessment challenging. In this work, we present SciQNet, a two-stage multimodal adaptation framework for scientific image quality assessment. The first stage performs domain-adaptive pretraining on scientific document images and the second stage conducts task-specific fine-tuning with joint scoring and understanding supervision. For scoring-oriented supervision, we combine instruction tuning with a Huber loss derived from rating-word logits, while understanding-oriented supervision is formulated as multiple-choice visual question answering. Experiments show that using a 40% stratified subset of the domain-adaptive data gives the best performance among the evaluated pretraining fractions, suggesting that pretraining-data relevance may be as important as pretraining-data scale. The final model achieves an SIQA-S score of 92.21, an SIQA-U score of 47.38 and a combined score of 69.80. This work presents our solution to the ICME 2026 Scientific Image Quality Assessment Challenge, which ranked 2nd in the scoring track.

Yin-Loon Khor, Yi-Jie Wong, Jing Jie Tan +1

Universiti Malaya · Universiti Tunku Abdul Rahman · National University of Singapore

VLM Adaptation Multimodal Pretraining Image Quality Assessment

Ask PDF

Jul 29, 2026

SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) methods are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visual-surface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness. To address this, we propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers top computer-science conferences from 2020 to 2025; 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into gold-standard annotations. Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context. To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation framework SFQ-Agent to achieve auditable and refined scoring through the collection and fusion of modal evidence. Multiple mainstream large models were evaluated on the test subset eval1200, and SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest consistency rate (93.4%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes.

Zihan Deng, Chuanzhi Xu, Huiqi Liang +3

The University of Hong Kong · The University of Sydney · University of Electronic Science and Technology of China

Automated Evaluation Image Quality Assessment Multimodal Model Evaluation

Ask PDF

Jun 2, 2026

Towards Characterizing Scientific Image Utility and Upgradability

Scientific images function as critical evidence in research communication, yet their integrity faces unprecedented threats from AI-generated content that introduces subtle but consequential errors. Existing evaluation paradigms prove inadequate: perceptual quality metrics poorly correlate with scientific validity, while language models lack domain-specific verification capabilities. To address this gap, we propose the Scientific Image Utility and Upgradability Assessment (\textbf{SIU

^2

A}) framework, which introduces two complementary dimensions for scientific image evaluation. Utility encompasses error detection (identifying scientific inaccuracies) and correction feasibility (assessing whether errors can be reliably repaired). Upgradability measures the quality of correction. We categorize scientific image corruption into four fundamental types: Detail Distortion, Incompleteness, False Content, and Entity Confusion. Based on this taxonomy, we construct SIU

^2

A-Benchmark, a dataset with expert annotations for error identification and repair. The framework implements a two-stage evaluation protocol: the Utility stage evaluates error detection capability and repair instruction generation, while the Upgradability stage assesses whether corrections faithfully restore scientific validity without compromising existing accurate information. Experiments reveal that current multimodal systems exhibit significant limitations in both scientific error assessment and faithful correction, exposing a fundamental gap between visual perception and scientific usability.

WenZhe Li, Qihang Yan, Liang Chen +6

Shanghai Research Institute for Intelligent Autonomous Systems, TongJi University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University

Image Quality Assessment Multimodal Model Evaluation

Ask PDF