Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality

Elias Rotondo; Lin Duan; Yanming Xiu; Sangjun Eom; Conrad Li; Maria Gorlatova

cs.CVSep 30, 2026

Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality

Authors: Elias Rotondo, Lin Duan, Yanming Xiu, Sangjun Eom, Conrad Li, Maria Gorlatova

Organizations: Duke University

Abstract

Advancements in augmented reality (AR) continue to foster innovative solutions, facilitating novel methodologies within educational systems, healthcare delivery, and risk-mitigation protocols. However, optimizing for end-user immersion and comfort remains challenging, as AR head-mounted displays contend with constrained scene geometry, spatial jitter, and temporal instability. User studies are the standard AR evaluation method for visual quality, but their cost, diminishing scalability, and inflexibility pose bottlenecks during iterative application design. To address this problem, we present an automated framework for AR content evaluation and refinement, built on vision-language models (VLMs), to evaluate and predict the visual fidelity of AR scenes as perceived by users. First, we introduce RateAR, a benchmark of AR images and videos collected across diverse scenes and environmental conditions, with good-to-excellent reliability (ICC(2,5) >= .90) across perceptual factors, including object placement, scale, and shadow consistency. Subsequently, we evaluate eleven commercial VLMs on the crafted benchmark. Results support that VLM-based quality predictions strongly correlate with human subjective judgments, achieving Spearman's rank-order correlations of up to 0.8695. An ablation study further suggests that, compared to other prompting strategies, our contextual prompting yields better alignment with human ratings while balancing introduced complexity cues. Building on these findings, we construct an automated AR content adjustment system and conduct a 21-participant user study. More than 90% of participants found that the system improved placement and size coherence of virtual content.

Figures & tables

Figure 1. Augmented reality content adjustments guided by vision-language models: (a) a virtual car is scaled to fill a parking spot; (b) a digital car is downsized to fit a toy city’s scale; and (c) a coffee cup asset is lowered to rest on the table’s surface. Examples of virtual content being adjusted automatically using vision language models to meet human expectations in augmented reality: (a) a virtual car is scaled to fill a parking spot; (b) a digital car is downsized to fit a toy city's scale; and (c) a coffee cup asset is lowered to rest on the table's surface.

Figure 2. AR deployment challenges from the DiverseAR+ dataset ( Duan et al., 2025a ) : (a) conflicting real/virtual shadow directions; (b) an unnaturally small AR table; and (c) a spatially misaligned digital glove on the user’s hand. AR deployment challenges from the DiverseAR+ dataset demonstrating conflicts between asset shadow, size, and placement renderings in real-world settings.

Figure 3. RateAR dataset examples illustrating varying quality across three features: (a) Placement plausibility: virtual desk floating a few feet above (poor), inches above (fair), or level with the ground (good); (b) Size appropriateness: digital car appearing far too small (poor), slightly small (fair), or realistically sized (good); and (c) Shadow realism: AR noodle bowl shadow opposing the real lighting (poor), slightly misaligned (fair), or fully aligned (good). Representative examples from the RateAR dataset illustrating various quality levels across three key visual features: placement plausibility, size appropriateness, and shadow realism.

Figure 4. Autonomous AR content adjustment framework: The user initially sees a floating virtual microwave (top). Our system evaluates the scene and automatically lowers it to the floor (bottom). Autonomous AR content adjustment framework: The AR device captures an image or video of the scene and sends it to the edge server. The edge server encodes it in base64 format along with an adjustment-aware prompt, then sends the task to a cloud VLM. After inference, the edge server extracts the VLM's suggestions to identify actionable adjustment values within the rendering framework. These changes are sent to the AR device to manipulate the AR content's visual appearance before rerendering.

Methods	Placement			Size			Shadow
Methods	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
HyperIQA	0.5093	0.5084	0.3727	0.6767	0.6418	0.5150	0.5025	0.5470	0.3585
CNNIQA	0.3595	0.3666	0.2573	0.4512	0.4737	0.3216	0.3154	0.3734	0.2219
CLIPIQA+	0.3792	0.3614	0.2749	0.6258	0.5721	0.4388	0.4038	0.4768	0.2814
LIQE	0.4564	0.4518	0.3293	0.7079	0.6838	0.5283	0.4950	0.5413	0.3525
LPIPS-VGG+	0.6970	0.6504	0.4955	0.5672	0.4934	0.3913	0.3441	0.3359	0.2416

Table 1. Correlation metrics between human scores and model predictions on RateAR images. Bold indicates column-wise best.

Methods	Placement			Size			Shadow
Methods	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
DOVER	0.4444	0.4372	0.3206	0.5084	0.6189	0.3549	-0.0334	0.0256	-0.0212
COVER	0.3800	0.4231	0.2700	0.4614	0.5931	0.3110	0.0450	-0.0385	0.0368
ModularBVQA	0.4348	0.4391	0.3126	0.6348	0.7756	0.4450	0.2190	0.1438	0.1451
ReLaX-VQA	0.3385	0.3456	0.2137	0.4755	0.5796	0.3461	-0.0371	-0.1027	-0.0212
E-VQA	0.0830	0.0937	0.0563	0.2790	0.3412	0.1890	0.0162	0.0860	0.0112

Table 2. Correlation metrics between human scores and model predictions on RateAR videos. Bold indicates column-wise best.

Figure 5. User study scenarios: (a) Parking lot, for scaling a virtual car relative to parking spaces; (b) Toy city, for rendering the car among miniature buildings; and (c) Office table, for desktop item placement and sizing. User study scenarios utilized throughout framework experimentation: parking lot, miniature toy city, and office desk.

Prompt Strategy	VLM	Placement			Size			Shadow
Prompt Strategy	VLM	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
	GPT-o3	0.7119	0.7049	0.5776	0.6298	0.6509	0.5080	0.4407	0.4118	0.3555
Primitive	GPT-5	0.7394	0.7246	0.6061	0.4940	0.5364	0.3776	0.4686	0.4638	0.3723
	GPT-5.4	0.6487	0.6354	0.5300	0.0292	0.1008	0.0219	0.6153	0.5984	0.5008
	GPT-o3	0.7416	0.7154	0.6078	0.7378	0.7562	0.5812	0.5866	0.5440	0.4677
Contextual	GPT-5	0.8336	0.7979	0.6845	0.4968	0.5832	0.3873	0.5025	0.4927	0.3961

Table 3. Correlation metrics between human scores and GPT-family flagship model predictions on RateAR images. Bold indicates column-wise best.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	Placement			Size			Shadow
Methods	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
HyperIQA	[0.40, 0.60]	[0.40, 0.60]	[0.25, 0.48]	[0.60, 0.74]	[0.56, 0.71]	[0.41, 0.61]	[0.40, 0.60]	[0.45, 0.63]	[0.24, 0.47]
CNNIQA	[0.24, 0.47]	[0.25, 0.48]	[0.13, 0.38]	[0.34, 0.55]	[0.36, 0.57]	[0.20, 0.44]	[0.19, 0.43]	[0.25, 0.48]	[0.09, 0.34]
CLIPIQA+	[0.26, 0.49]	[0.24, 0.47]	[0.15, 0.39]	[0.54, 0.70]	[0.48, 0.65]	[0.33, 0.54]	[0.29, 0.51]	[0.37, 0.57]	[0.16, 0.40]
LIQE	[0.35, 0.55]	[0.34, 0.55]	[0.21, 0.44]	[0.64, 0.77]	[0.61, 0.75]	[0.43, 0.62]	[0.39, 0.59]	[0.44, 0.63]	[0.23, 0.46]
LPIPS-VGG+	[0.62, 0.76]	[0.57, 0.72]	[0.39, 0.59]	[0.47, 0.65]	[0.39, 0.59]	[0.27, 0.50]	[0.22, 0.46]	[0.21, 0.45]	[0.11, 0.36]

Appendix

Table A1. Correlation metric 95% confidence intervals between human scores and model predictions on RateAR images, corresponding to the coefficients in Table 1 . Confidence interval format: [lower_bound, upper_bound].

Methods	Placement			Size			Shadow
Methods	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
HyperIQA	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001
CNNIQA	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001
CLIPIQA+	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001
LIQE	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001
LPIPS-VGG+	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001

Appendix

Table A2. Correlation coefficient statistical significance p-values between human scores and model predictions on RateAR images, corresponding to the coefficients in Table 1 . P-Value format:

p_{adj}\ (p_{raw})

; for highly significant results (i.e.,

p_{adj}<.001

), only the adjusted p-value is reported.

Methods	Placement			Size			Shadow
Methods	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
DOVER	[0.22, 0.62]	[0.21, 0.62]	[0.08, 0.53]	[0.30, 0.67]	[0.44, 0.75]	[0.12, 0.56]	[-0.26, 0.23]	[-0.19, 0.31]	[-0.26, 0.24]
COVER	[0.14, 0.58]	[0.19, 0.61]	[0.02, 0.49]	[0.24, 0.64]	[0.40, 0.73]	[0.07, 0.52]	[-0.21, 0.29]	[-0.29, 0.21]	[-0.21, 0.28]
ModularBVQA	[0.21, 0.62]	[0.21, 0.62]	[0.07, 0.52]	[0.46, 0.76]	[0.65, 0.86]	[0.22, 0.63]	[-0.03, 0.44]	[-0.11, 0.38]	[-0.11, 0.38]
ReLaX-VQA	[0.10, 0.54]	[0.10, 0.55]	[-0.04, 0.44]	[0.26, 0.65]	[0.39, 0.72]	[0.11, 0.55]	[-0.28, 0.21]	[-0.34, 0.15]	[-0.27, 0.23]
E-VQA	[-0.17, 0.33]	[-0.16, 0.34]	[-0.20, 0.30]	[0.03, 0.49]	[0.10, 0.54]	[-0.06, 0.42]	[-0.23, 0.26]	[-0.17, 0.33]	[-0.24, 0.26]

Appendix

Table A3. Correlation metric 95% confidence intervals between human scores and model predictions on RateAR videos, corresponding to the coefficients in Table 2 . Confidence interval format: [lower_bound, upper_bound].

Methods	Placement			Size			Shadow
Methods	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
DOVER	.005 (<.001)	.006 (<.001)	.009 (<.001)	<.001	<.001	.001 (<.001)	>.999 (.797)	>.999 (.635)	>.999 (.815)
COVER	.039 (.002)	.010 (<.001)	.059 (.003)	.003 (<.001)	<.001	.009 (<.001)	>.999 (.728)	>.999 (.767)	>.999 (.685)
ModularBVQA	.007 (<.001)	.006 (<.001)	.012 (<.001)	<.001	<.001	<.001	>.999 (.087)	>.999 (.265)	>.999 (.110)
ReLaX-VQA	.121 (.007)	.101 (.006)	.352 (.021)	.002 (<.001)	<.001	.002 (<.001)	>.999 (.775)	>.999 (.427)	>.999 (.815)
E-VQA	>.999 (.521)	>.999 (.469)	>.999 (.542)	.477 (.028)	.113 (.007)	.604 (.036)	>.999 (.901)	>.999 (.507)	>.999 (.902)

Appendix

Table A4. Correlation coefficient statistical significance p-values between human scores and model predictions on RateAR videos, corresponding to the coefficients in Table 2 . P-Value format:

p_{adj}\ (p_{raw})

; for highly significant results (i.e.,

p_{adj}<.001

), only the adjusted p-value is reported.

Figure A1. Pre–post changes in human-rated quality for placement, via (a) image-based and (b) video-based pipelines, and size, via (c) image-based and (d) video-based pipelines. Scores indicate improvement (positive), no change (zero), or degradation (negative) after single-iteration VLM adjustment. Pre–post changes in human-rated quality for placement and size. General trends reflect left-skewed patterns toward zero, but more users report feature improvements than not.

Trial	p-Value
Placement, Image-based	0.0012
Placement, Video-based	0.0012
Size, Image-based	0.0101
Size, Video-based	0.0044

Appendix

Table A5. Paired Wilcoxon signed-rank test p-values for statistical testing reported in Section 5.4.1

Prompt Strategy	VLM	Placement			Size			Shadow
Prompt Strategy	VLM	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
	GPT-o3	[0.64, 0.77]	[0.63, 0.77]	[0.48, 0.66]	[0.54, 0.70]	[0.57, 0.72]	[0.40, 0.60]	[0.33, 0.54]	[0.30, 0.52]	[0.23, 0.47]
Primitive	GPT-5	[0.67, 0.79]	[0.66, 0.78]	[0.52, 0.68]	[0.39, 0.59]	[0.44, 0.62]	[0.26, 0.49]	[0.36, 0.57]	[0.35, 0.56]	[0.25, 0.48]
	GPT-5.4	[0.57, 0.72]	[0.55, 0.71]	[0.43, 0.62]	[-0.10, 0.16]	[-0.03, 0.23]	[-0.11, 0.15]	[0.53, 0.69]	[0.51, 0.68]	[0.39, 0.59]
	GPT-o3	[0.68, 0.80]	[0.64, 0.77]	[0.52, 0.68]	[0.67, 0.79]	[0.69, 0.81]	[0.49, 0.66]	[0.49, 0.67]	[0.44, 0.63]	[0.36, 0.56]
Contextual	GPT-5	[0.79, 0.87]	[0.74, 0.84]	[0.61, 0.75]	[0.39, 0.59]	[0.49, 0.66]	[0.27, 0.49]	[0.40, 0.60]	[0.39, 0.59]	[0.28, 0.50]

Appendix

Table A6. Correlation metrics between human scores and GPT-family flagship model predictions on RateAR images, corresponding to the coefficients in Table 3 . Confidence interval format: [lower_bound, upper_bound].

Prompt Strategy	VLM	Placement			Size			Shadow
Prompt Strategy	VLM	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC	SRCC	PLCC	KRCC
	GPT-o3	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001
Primitive	GPT-5	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001
	GPT-5.4	<.001	<.001	<.001	>.999 (.666)	>.999 (.135)	>.999 (.677)	<.001	<.001	<.001
	GPT-o3	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001
Contextual	GPT-5	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001

Appendix

Table A7. Correlation coefficient statistical significance p-values between human scores and GPT-family flagship model predictions on RateAR images, corresponding to the coefficients in Table 3 . P-Value format:

p_{adj}\ (p_{raw})

; for highly significant results (i.e.,

p_{adj}<.001

), only the adjusted p-value is reported.

Explore similar work

Feb 1, 2026cs.HC

Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raters

Digital video is central to communication, education, and entertainment, but without audio description (AD), blind and low-vision users are excluded. While crowdsourced platforms and vision-language models (VLMs) expand AD production, quality is rarely checked systematically. Existing evaluations rely on NLP metrics and short-clip guidelines, leaving open the question of how to assess long-form AD quality at scale. To address this, we developed a methodological workflow using Item Response Theory to evaluate VLM and human rater proficiency against expert-established ground truth. Evaluations were based on a six-dimensional framework, grounded in professional guidelines and shaped by insights from our accessibility experts and blind consultants. Findings suggest that top-performing VLMs can approximate ground-truth ratings at levels comparable to human raters. However, qualitative analysis reveals that VLM reasoning is less reliable and actionable than that of human respondents. These insights underscore the potential of hybrid evaluation systems that leverage VLMs alongside human oversight, offering a path toward scalable AD quality control.

Lana Do, Gio Jung, Juvenal Francisco Barajas +5

Northeastern University · United States · San Francisco State University +2

Video Quality Multimodal Evaluation Rater Large Language Model Judges Crowdsourced Framework Agentic Workflow Design

Ask PDF

Jun 23, 2025cs.IR

Can Argus Judge Them All? Comparing VLMs Across Domains

Vision-Language Models (VLMs) are increasingly used in industry VLM applications such as retrieval systems, content generation platforms, and decision-support workflows, where model selection is commonly guided by benchmark rankings. These rankings are largely determined by retrieval, captioning, and reasoning downstream tasks; however, models with similar task performance often show substantially different behavior across datasets. This creates a Capability-Reliability Gap between benchmark performance and observed model stability. We present ARGUS-EVAL, a capability-reliability-oriented evaluation framework for VLMs that characterizes model behavior through Benchmark Capability P(M), Cross-Dataset Consistency CDC(M), Robustness Retention RR(M), and Efficiency E(M). We evaluate CLIP, BLIP, LXMERT, Gemma-3-4B, and Qwen-2.5VL-3B-Instruct across retrieval, captioning, and reasoning downstream tasks. The results reveal notable differences between capability-oriented and reliability-oriented rankings. Qwen-2.5VL-3BInstruct achieves the strongest overall capability (R@1 = 82.7%, BLEU-4 = 47.2%, CIDEr = 141.6, CDC = 0.91), whereas CLIP records the lowest latency (31 ms) and memory footprint (0.9 GB).

Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali +5

Bharati Vidyapeeth, New Delhi, India · Macquarie University, Sydney, Australia · DSEU-Okhla, New Delhi, India +3

Evaluation Benchmarks

Ask PDF

Apr 23, 2026cs.CV

Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T2I) generation tasks. Despite this growing reliance, the reliability of these Evaluator VLMs remains under explored. In this work, we systematically evaluate the reliability of Evaluator VLMs across both I2T and T2I tasks. We introduce targeted perturbations that degrade output quality along key error dimensions, including object hallucinations, spatial reasoning, factual grounding, and visual fidelity. These perturbations test whether Evaluator VLMs can reliably account for these quality degrading errors in their evaluations. Using a comprehensive benchmark of over 4000 perturbed instances spanning 40 perturbation dimensions, we evaluate 4 prominent VLMs using single-answer scoring, pairwise comparison, and reference-guided paradigms. Our findings reveal that current VLM evaluators exhibit substantial blind spots: they often fail to detect perturbed outputs - in some cases exceeding 50%, struggle particularly with fine-grained compositional and spatial errors, and are often insensitive to hallucinated content that contradicts the input image. Pairwise comparison proves more reliable, though failure rates persist. These results highlight the unreliable nature of current Evaluator VLMs and urge caution in their deployment for benchmarking and development decisions. Code and data have been made publicly available.

Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1

♠Nilekani Centre at AI4Bharat ♢Indian Institute of Technology Madras · ♠Nilekani Centre at AI4Bharat · ♣BITS Pilani, Hyderabad

Large Language Model Evaluation Blind Spots Text-To-Image Destination-Only Scoring Visual Fidelity

Ask PDF