LLaVA-Assessor: Building the Foundation LMM For Visual Quality Assessment

Ziheng Jia; Zicheng Zhang; Jiaying Qian; Guangtao Zhai; Xiongkuo Min

cs.CVAug 11, 2026

LLaVA-Assessor: Building the Foundation LMM For Visual Quality Assessment

Authors: Ziheng Jia, Zicheng Zhang, Jiaying Qian, Guangtao Zhai, Xiongkuo Min

Organizations: Shanghai Artificial Intelligence Laboratory, Shanghai 200032, China

Abstract

Aligning with the human visual system~(HVS) in perceiving and evaluating the quality of visual signals is a central objective of machine-vision-based visual quality assessment systems. With the rapid progress of large multi-modal models~(LMMs), visual question answering provides a promising paradigm for building unified foundation models for visual quality assessment under multi-modal and multi-task scenarios. Inspired by the classical perception-decision" process in HVS-based quality evaluation, we formulate visual quality assessment for LMM-based machine vision as two complementary tasks: quality interpretation'' and ``quality scoring". Centered on these objectives, we propose LLaVA-Assessor, a unified data construction and model training system. To support multi-modal inputs, we design an adaptive model architecture that enables efficient processing of both images and videos. For data construction, we develop rigorous human annotation protocols and a novel machine-synthesis-dominated data expansion pipeline to build a large-scale and high-quality datasets. Furthermore, we introduce a simple yet effective prompt disentanglement strategy to alleviate training-objective confusion in multi-task learning, thereby enabling stable and coherent joint training. The resulting all-in-one LMM LLaVA-Assessor-GIGA achieves superior performance on $11$ image/video quality scoring test sets and 4 visual quality interpretation benchmarks. Extensive results demonstrate the effectiveness of integrating structured data construction, adaptive model design, and multi-task joint training for automated visual quality assessment. Our work provides compelling insights for developing foundation LMMs for automatic visual quality assessment. Project page at https://github.com/jzhws/LLaVA-Assessor.

Figures & tables

Figure 1: The motivation guideline of our work. The “ perception-decision ” mechanism in the HVS system outlines the key tasks for machine-vision-based models, namely “quality interpreting” and “quality scoring”. Building upon this, a visual quality assessment foundation LMM should focus on the model’s versatility and efficiency . A unified model, when deployed once, should be capable of handling multi-modal, multi-task scenarios, achieving performance that aligns with human perception and is comparable to or surpasses that of proprietary models on benchmark tests for the corresponding tasks.

Figure 2: An overview of the technical map. We begin by conducting rigorous human annotation and training the intermediate LLaVA-Assessor-SFT using the human-annotated data for subsequent machine-dominated data expansion. Next, we employ techniques such as general-LMM-as-annotator/judger and human-in-the-loop scrutinizing , while also integrating high-quality image quality assessment training data . By utilizing the prompt disentanglement strategy, we achieve collective training, resulting in the LLaVA-Assessor-GIGA , a powerful unified visual quality assessment foundation LMM.

Figure 3: The model structure featured on the “modal-selective” input strategy for different modalities. The vision token, motion token, and text token are concatenated in an interleaved manner that aligns with their semantic sequence. Specifically, the “motion tokens” are applicable only for video inputs.

Model Structure	Name	More Information
Vision Tower	SigLIP-SO400m	Parameter size= $397.75$ M, Tokens per keyframe= $196$
Vision Projector	2-layers MLP+GeLU	Parameter size = $16.98$ M
Motion Extractor	SlowFast-R50	Use the fast-path feature, token numbers are the same as the frame number
Motion Projector	2-layers MLP+GeLU	Parameter size = $13.77$ M, the same with the structure of vision projector
LLM init.	Qwen-2 (7B)	Decoder-only model, parameter size = $7660.56$ M

TABLE I: Details of the model structure configurations of the LLava-Assessor-GIGA .

Type	Categories	# Instruction Pairs	Source Dataset	# Video/Image
Videos (human)	Scoring	28,056	LSVQ	28,056
Videos (human)	Interpreting	115,124	LSVQ	15,500
Videos (machine)	Technical	230,000	Panda-70m	55,128
	In-context	127,000	Panda-70m	22,927
	Aesthetic	45,000	Panda-70m	7,728
Image	Scoring	13,000	Q-ALIGN-DB	13,000

TABLE II: Statistical summary of the training MIDB for our LLaVA-Assessor-GIGA .

Figure 4: The human-annotated data construction pipeline . The quality scoring data (left part of the figure) primarily derives from existing video quality assessment datasets and is converted based on quality level. The quality interpretation data (on the right part) is annotated via human experts based on three aspects: “quality factor”, “temporal description”, and “degree description”. Additionally, in-context descriptions or extended conversations are provided. These annotations are then rewritten by GPT into multiple types of question-answer pairs suitable for SFT training.

Quality Factors	Standard Annotation Examples
Sharpness	Sharp, relatively clear, relatively fuzzy, very blurry
Focus	In-focus, out-of-focus
Noise	Noiseless, a small amount of noise, severe noise densely distributed
Motion Blur	Clear-motion, blur-motion
Flicker/Camera Shake	Stable, small amplitude shake, shaky
Exposure (subjective to light)	Well-exposed, underexposure/overexposure

TABLE III: The representative quality factors examples for human annotation . The quality factors in bold are those that must be considered in most videos. The quality factors in italic font are the ones that can only be noticed in a small number of videos. The remaining factors should be considered depending on the content and quality level of the specific video.

Figure 5: The machine-dominated annotation process . This process includes three branches: the technical branch , which uses a domain-specific model fine-tuned on the previous human-annotated data as the primary annotator, and incorporates human-in-the-loop and LMM-as-judger strategies for rejection sampling to construct data; the in-context branch , which uses LMMs in conjunction with synthesized local distortion data to build the dataset; and the aesthetic branch , which selects appropriate aesthetic evaluation tasks for LMM annotation, utilizing carefully designed prompts for LMM annotation.

Figure 6: The illustration of the prompt disentanglement strategy (we use VQA tasks as an example). The core lies in minimizing the similarity of guiding prompts across different tasks during SFT to avoid training target confusion.

Hyper-Parameters	Value
Image Input Resulution	Original
Keyframes Sampling Rate	$1fps$
Keyframes (for videos) Resolution	$384\times 384$
Frames (for motion extraction) Resolution	$192\times 192$
Batch Size	8
LR Max	1e-5 / 1e-6

TABLE IV: Details of the hyperparameters for model training. The red / blue colors represent the hyper-parameters used in the LLaVA-Assessor-SFT and LLaVA-Assessor-GIGA training, respectively. Other entries without color differentiation indicate that the hyperparameter remains consistent across both rounds of training.

Figure 7: The training loss curve along with the training steps. It can be observed that the training loss drops quickly and then stabilizes.

Datasets	LSVQ(1080p) [ 70 ] $\spadesuit$		LSVQ(test) [ 70 ] $\spadesuit$		LIVE-VQC [ 45 ] $\diamondsuit$		KoNViD-1K [ 24 ] $\diamondsuit$		YT-UGC [ 57 ] $\diamondsuit$		YT-Gaming [ 74 ] $\diamondsuit$		Avg.
Models	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	Avg.
Simple-VQA (ACM.MM 2022) [ 47 ]	0.760	0.805	0.870	0.868	0.755	0.793	0.862	0.859	0.826	0.821	0.686	0.746	0.804
BVQA (TCSVT 2022) [ 32 ]	0.747	0.785	0.870	0.861	0.795	0.814	0.795	0.817	0.845	0.847	0.677	0.669	0.794
FAST-VQA (TPAMI 2023) [ 60 ]	0.765	0.793	0.880	0.871	0.830	0.822	0.869	0.870	0.725	0.742	0.631	0.677	0.790
Dover (ICCV 2023) [ 61 ]	0.797	0.821	0.893	0.892	0.835	0.857	0.885	0.879	0.801	0.814	0.647	0.728	0.821
Modular-VQA (CVPR 2024) [ 59 ]	0.810	0.834	0.897	0.895	0.803	0.839	0.876	0.887	0.786	0.803	0.696	0.703	0.819

TABLE V: Performance on video quality scoring tasks. Per column: highest in red , second in blue , third underlined . [

\spadesuit

: Intra-dataset testing,

\diamondsuit

: Inter-dataset testing, Textit : Out of domain test datasets]

Datasets	KonIQ [ 25 ] $\spadesuit$		SPAQ [ 17 ] $\spadesuit$		LIVE-C [ 21 ] $\diamondsuit$		AGIQA-3K [ 33 ] $\diamondsuit$		KADID-10K [ 34 ] $\diamondsuit$		Avg.
Models	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	Avg.
NIMA (TIP 2018) [ 50 ]	0.859	0.896	0.856	0.838	0.771	0.814	0.654	0.715	0.535	0.532	0.747
DBCNN (TCSVT 2020) [ 77 ]	0.875	0.884	0.806	0.812	0.755	0.773	0.641	0.730	0.484	0.497	0.726
HyperIQA (CVPR 2020) [ 46 ]	0.906	0.917	0.788	0.791	0.749	0.772	0.640	0.702	0.468	0.506	0.724
MUSIQ (ICCV 2021) [ 30 ]	0.929	0.924	0.863	0.868	0.830	0.789	0.630	0.722	0.556	0.575	0.769
CLIP-IQA+ (AAAI 2023) [ 54 ]	0.895	0.909	0.864	0.866	0.805	0.832	0.685	0.736	0.654	0.653	0.790

TABLE VI: Performance on image quality scoring task. Per column: highest in red , second in blue , third underlined . [

\spadesuit

: Intra-dataset testing,

\diamondsuit

: Inter-dataset testing, Textit : Out of domain test datasets]

Sub-categories	Question Types			Quality Concerns				Overall $\uparrow$
Models	Binary $\uparrow$	Multi. $\uparrow$	Open $\uparrow$	Tech. $\uparrow$	Aes. $\uparrow$	Temp. $\uparrow$	AIGC $\uparrow$	Overall $\uparrow$
Performance of Open-sourced LMMs
mPLUG-Owl3- 7B [ 69 ]	56.90%	57.14%	42.88%	53.40%	61.85%	50.34%	45.34%	52.06%
InternVL2.5- 8B [ 10 ]	48.15%	39.37%	31.49%	39.06%	46.68%	42.52%	31.37%	39.50%
InternVL2.5- 38B	50.84%	46.69%	37.18%	44.57%	54.50%	48.13%	40.68%	44.72%
InternVL2.5- 78B	48.82%	46.34%	38.13%	43.52%	57.35%	47.28%	36.65%	44.28%

TABLE VII: Evaluation results on the Q-bench-video-test (single) . Per column: highest in red , second in blue , third underlined .

Performance of Open-sourced LMMs
Sub-categories	Question Types			Quality Concerns				Overall $\uparrow$
Models	Binary $\uparrow$	What $\uparrow$	How $\uparrow$	Technical $\uparrow$	Other $\uparrow$	In-context
Models	Binary $\uparrow$	What $\uparrow$	How $\uparrow$	Technical $\uparrow$	Other $\uparrow$	Technical $\uparrow$	Other $\uparrow$
mPLUG-Owl3- 7B	78.72%	79.77%	67.45%	73.44%	71.74%	71.19%	84.89%	74.21%
InternVL2.5- 8B	78.84%	79.67%	66.13%	69.23%	75.98%	69.18%	85.15%	73.80%
InternVL2.5- 38B	79.37%	82.64%	68.93%	73.70%	78.52%	72.26%	86.31%	76.98%

TABLE VIII: Evaluation results on the LLVisionQA-test . Per column: highest in red , second in blue , third underlined . [

\spadesuit

: Domain-specific LMM]

Performance of Open-sourced LMMs
Sub-categories	Question Types			Low-level Concerns		Pairwise Concerns		Overall $\uparrow$
Models	Binary $\uparrow$	What $\uparrow$	How $\uparrow$	Distortion $\uparrow$	Other $\uparrow$	Compare $\uparrow$	Joint $\uparrow$	Overall $\uparrow$
mPLUG-Owl3- 7B	54.21%	43.38%	45.32%	49.57%	45.67%	48.32%	48.88%	48.44%
InternVL2.5- 8B	51.94%	29.78%	53.84%	42.01%	55.71%	46.26%	49.09%	47.08%
InternVL3- 8B	70.36%	28.13%	35.98%	44.08%	57.43%	47.02%	51.11%	47.94%
LLaVA-OneVision- 7B	60.09%	45.42%	50.86%	53.09%	58.82%	54.52%	55.55%	52.75%

TABLE IX: Results on LLVisionQA-pair-test . Per column: highest in red , second in blue , third underlined .

Performance of Open-sourced LMMs
Sub-categories	Video Pairs
Models	Joint $\uparrow$	Compare	Compare	Overall $\uparrow$
Models	Joint $\uparrow$	-fine $\uparrow$	-coarse $\uparrow$	Overall $\uparrow$
InternVL2.5- 8B	48.85%	51.10%	49.20%	49.79%
InternVL3- 8B	55.73%	40.40%	51.84%	49.72%
LLaVA-OneVision- 7B	51.56%	48.43%	49.89%	53.48%

TABLE X: Results on the Q-bench-video-test (pair) . Per column: highest in red , second in blue , third underlined .

LLaVA-	Image	Image	Video	Video	Scoring		Interpreting
Assessor	Scoring	Interpreting	Scoring	Interpreting	Image	Video	Image	Video
-	$\times$	✓	$\times$	✓	0.735	0.629	78.28%	62.89%
-	✓	$\times$	✓	$\times$	0.851	0.818	/	/
-	$\times$	$\times$	✓	✓	0.778	0.825	69.62%	59.25%
-	✓	✓	$\times$	$\times$	0.853	0.653	74.56%	43.76%
-	$\times$	$\times$	$\times$	✓	0.632	0.758	60.12%	59.11%

TABLE XI: Ablation study results. PD denotes the prompt disentanglement strategy. Scoring performance is reported as the average of SRCC and PLCC on corresponding task datasets, while Interpreting performance is evaluated on the LLVisionQA-test and Q-bench-video-test (single) for image and video, respectively. Per column: highest in red .

Figure 8: The overview visualization of the performance across a series of LMMs and our LLaVA-Assessor-GIGA .

Figure 9: Visualization of the data-scaling effects, which shows the performance trends for the four tasks with the equal-proportional training data scaling-up. The image quality scoring task is evaluated on the SPAQ test set, while the video quality scoring task is assessed on KoNViD-1K . The performance for image and video quality interpreting is measured on overall performance on the LLVisionQA-test and Q-bench-video-test (single) , respectively.

Explore similar work

DPC-VQA: Decoupling Quality Perception and Residual Calibration for Video Quality Assessment
Apr 14, 2026Xinyue Li, Shubo Xu, Zhichao Zhang +3Video Quality Assessment VLM Calibration
Shanghai Jiao Tong University, Shanghai, China · Baidu Inc., Beijing, China · Xinjiang University, Urumqi, China
IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment
Jul 14, 2026Jinjian Wu, Jiaqi Tang, Wei Wei +5Tool Use in VLMs VLM Reasoning
School of Computer Science, Northwestern Polytechnical University, Xi’an, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
VersusQ: Pairwise Margin Reasoning for Generalizable Video Quality Assessment
May 20, 2026Shibei Meng, Binxin Yang, Yuan Liu +4Pairwise Comparison Learning to Rank
The Chinese University of Hong Kong · Beijing Normal University · WeChat Vision, Tencent Inc. +2

Apr 14, 2026J/K move · Enter open · S save

DPC-VQA: Decoupling Quality Perception and Residual Calibration for Video Quality Assessment

Xinyue Li, Shubo Xu, Zhichao Zhang, Zhaolin Cai+2

Shanghai Jiao Tong University, Shanghai, China · Baidu Inc., Beijing, China · Xinjiang University, Urumqi, China

Video Quality Assessment VLM Calibration Video Generation Evaluation

Open paper AskPDF arXiv

Recent multimodal large language models (MLLMs) have shown promising performance on video quality assessment (VQA) tasks. However, adapting them to new scenarios remains expensive due to large-scale retraining and costly mean opinion score (MOS) annotations. In this paper, we argue that a pretrained MLLM already provides a useful perceptual prior for VQA, and that the main challenge is to efficiently calibrate this prior to the target MOS space. Based on this insight, we propose DPC-VQA, a decoupling perception and calibration framework for video quality assessment. Specifically, DPC-VQA uses a frozen MLLM to provide a base quality estimate and perceptual prior, and employs a lightweight calibration branch to predict a residual correction for target-scenario adaptation. This design avoids costly end-to-end retraining while maintaining reliable performance with lower training and data costs. Extensive experiments on both user-generated content (UGC) and AI-generated content (AIGC) benchmarks show that DPC-VQA achieves competitive performance against representative baselines, while using less than 2% of the trainable parameters of conventional MLLM-based VQA methods and remaining effective with only 20% of MOS labels. The code will be released upon publication.

Abstract

Explore similar work

DPC-VQA: Decoupling Quality Perception and Residual Calibration for Video Quality Assessment

IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment

VersusQ: Pairwise Margin Reasoning for Generalizable Video Quality Assessment