Can Vision Language Models Judge Action Quality? An Empirical Evaluation

Miguel Monte e Freitas; Rui Henriques; Ricardo Rei; Pedro Henrique Martins

cs.CVApr 9, 2026

Can Vision Language Models Judge Action Quality? An Empirical Evaluation

Authors: Miguel Monte e Freitas, Rui Henriques, Ricardo Rei, Pedro Henrique Martins

Organizations: Sword Health · Instituto Superior T´ecnico, Universidade de Lisboa · INESC-ID

Abstract

Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging. Although Vision Language Models (VLMs) hold considerable promise for AQA, their actual performance in this domain remains largely uncharacterised. We present a comprehensive evaluation of state-of-the-art VLMs across activity domains (e.g. fitness, figure skating, diving), tasks, representations, and prompting strategies. Baseline results reveal that Gemini 3.1 Pro, Qwen3-VL and InternVL3.5 models perform only marginally above random chance, and although strategies such as incorporation of skeleton information, grounding instructions, reasoning structures and in-context learning lead to isolated gains, none is consistently effective. Analysis of prediction distributions uncovers two systematic biases: a tendency to predict correct execution regardless of visual evidence, and a sensitivity to superficial linguistic framing. Reformulating tasks contrastively to mitigate these biases yields minimal improvement, suggesting that the models' limitations go beyond these biases, pointing to a fundamental difficulty with fine-grained movement quality assessment. Our findings establish a rigorous baseline for future VLM-based AQA research and provide an actionable outline for failure modes requiring mitigation prior to reliable real-world deployment.

Figures & tables

	Classification			Regression
	LLM-FMS	EgoExo-Fitness	Fitness-AQA	FineFS		MTL-AQA
Model	Bal. Acc. $\uparrow$	Bal. Acc. $\uparrow$	Bal. Acc. $\uparrow$	$\rho\uparrow$	$\text{R-}\ell_{2}\downarrow$	$\rho\uparrow$	$\text{R-}\ell_{2}\downarrow$
Random guess	0.4310	0.5000	0.5000	0.0000	0.9312	0.0000	0.2181
Qwen3-VL-Instruct	0.4725	0.5733	0.5557	0.1998	0.2590	0.2319	0.2026
Qwen3-VL-Thinking	0.4605	0.5650	0.5224	0.2806	0.2673	0.1367	0.2665
InternVL3.5	0.4488	0.5429	0.5296	0.2042	0.2540	0.2636	0.1847

Table 1: Baseline results. Classification tasks (LLM-FMS, EgoExo-Fitness, Fitness-AQA) are evaluated with balanced accuracy. Regression tasks (FineFS, MTL-AQA) are evaluated with Spearman correlation (

\rho

) and R-

\ell_{2}

. Best result per column shown in bold . Random guess represents uniform sampling across labels for classification, and predicting the ground truth mean for regression.

Dataset	Model	Original	Cropped	Sk. overlays	Sk. only
LLM-FMS	Qwen3-VL-Instruct	0.4725	0.4808	0.4600	0.4374
	Qwen3-VL-Thinking	0.4605	0. 4693	0.4678	0.4416
	InternVL3.5	0.4488	0.4762	0.4473	0.4443
	InternVL3.5 †	0.4697	0.4591	0.4576	0.4628
	Gemini 3.1 Pro	0.6029	0.6257	0.6298	0.5925
EgoExo- Fitness	Qwen3-VL-Instruct	0.5733	0.5725	0.5472	0.5403

Table 2: Balanced accuracy (%) across preprocessing methods for classification tasks, with best result per row shown in bold .

		Original		Sk. overlays		Sk. only
Dataset	Model	$\rho\uparrow$	$\text{R-}\ell_{2}\downarrow$	$\rho\uparrow$	$\text{R-}\ell_{2}\downarrow$	$\rho\uparrow$	$\text{R-}\ell_{2}\downarrow$
FineFS	Qwen3-VL-Instruct	0.1998	0.2590	0.2041	0.2690	0.0507	0.3004
	Qwen3-VL-Thinking	0.2806	0.2673	0.3071	0.2909	0.0457	0.2412
	InternVL3.5	0.2042	0.2540	0.2687	0.2817	-0.0020	0.2186
	InternVL3.5 †	0.2797	0.2372	0.3110	0.2633	0.0751	0.1702
	Gemini 3.1 Pro	0.2651	0.3079	0.2435	0.3037	0.0385	0.3626

Table 3: Spearman correlation (

\rho

) and

\text{R-}\ell_{2}

across preprocessing methods for the regression tasks, with best result per row for each metric shown in bold . Cropping was not applied to these modalities, as it could omit important contextual factors such as jump height or splash.

Dataset	Model	Base Prompt	Visual Grounding	Two- Step	Structured Reasoning	Positive Guidelines	Negative Guidelines	ICL
Classification		Bal. Acc. $\uparrow$	Bal. Acc. $\uparrow$	Bal. Acc. $\uparrow$	Bal. Acc. $\uparrow$	Bal. Acc. $\uparrow$	Bal. Acc. $\uparrow$	Bal. Acc. $\uparrow$
LLM-FMS	Qwen3-VL-Instruct	0.4725	0.4631	0.4634	0.4397	0.4659	0.4404	0.5038
	Qwen3-VL-Thinking	0.4605	0.4685	0.4248	0.4397	0.4572	0.4670	0.4655
	InternVL3.5	0.4488	0.4554	0.4446	0.4640	0.4518	0.4376	0.4937
	InternVL3.5 †	0.4697	0.4508	0.4572	0.4582	0.4549	0.4319	0.4829
	Gemini 3.1 Pro	0.6029	0.6046	0.5913	0.5949	0.5788	0.6055	0.6260

Table 4: Results across prompt engineering strategies. Classification tasks (LLM-FMS, EgoExo-Fitness, Fitness-AQA) are evaluated with balanced accuracy (%). Regression tasks (FineFS, MTL-AQA) are evaluated with Spearman correlation (

\rho

) and R-

\ell_{2}

. Two best results per row for each metric are shown in bold .

Figure 1: Prompting effects on model predictions. For classification tasks (top) , each subplot shows the percentage of predictions matching the correct execution form. For regression tasks (bottom) , each subplot shows the mean predicted score normalised to 0–1. In both cases, results are shown across all models and three prompting strategies: base prompt (blue), positive guidelines (green), and negative guidelines (orange) , with dashed lines indicating ground truth values.

Model	LLM-FMS	EgoExo- Fitness	Fitness- AQA	FineFS	MTL-AQA
Random guess	50.00	50.00	50.00	50.00	50.00
Qwen3-VL-Instruct	68.97	54.61	50.60	56.40	60.06
Qwen3-VL-Thinking	60.87	54.24	51.80	57.40	55.52
InternVL3.5	62.55	– ∗	47.17	– ∗	52.41
InternVL3.5 †	59.98	– ∗	49.74	– ∗	53.26
Gemini 3.1 Pro	80.12	51.20	52.49	58.91	61.76

Table 5: Accuracy (%) on contrastive tasks, with best result per column shown in bold .

Figure 2: Accuracy (%) on contrastive tasks for MTL-AQA and FineFS as a function of the score gap between samples. Larger gaps generally correspond to easier comparisons, as the quality difference between samples becomes more pronounced.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure A.1: LLM-FMS Preprocessing Examples.

Panel of Figure 9 — Figure A.2: EgoExo-Fitness Preprocessing Examples.

Panel of Figure 10 — Figure A.3: Fitness-AQA Preprocessing Examples.

Explore similar work

Sep 16, 2026

Can Vision-Language Models Judge Olympic Diving? From Reasoning to Scores in Zero-Shot Action Quality Assessment

Automated action quality assessment (AQA) in Olympic sports remains a challenging task due to the complexity of human motion and the subjectivity inherent in expert judging. This work evaluates the capability of open-source Vision-Language Models (VLMs) to perform zero-shot action quality assessment on Olympic diving videos using the AQA-7 benchmark dataset. In this regard, a regression-based framework is pro-posed to leverage both the semantic reasoning and phase-level sub-scores generated by the VLMs, combining TF-IDF vectorization, dimensionality reduction, and ensemble learning to predict final competition scores. Experimental results show that standalone VLMs achieve moderate Spearman correlations below 0.32, while the proposed ensemble regression framework substantially improves performance in the reported evaluation, reaching a Spearman correlation of 0.67 with a four-model configuration. Textual reasoning features con-sistently outperformed raw numerical sub-scores, highlighting the richness of VLM-generated explanations for action quality analysis. These findings suggest that VLMs hold strong potential as assistive tools for explainable and semi-automated sports performance evaluation. The code is publicly available on GitHub https://github.com/hvelesaca/olympic diving judge vlm

Henry O. Velesaca, David Freire-Obregon, Luigi Miranda +1

ESPOL Polytechnic University, Campus Gustavo Galindo, Guayaquil, Ecuador · Software Engineering Department, Research Center for Information and Communication Technologies (CITIC-UGR), University of Granada, 18071, Granada, Spain · SIANI, Universidad de Las Palmas de Gran Canaria, Spain +1

VLM Evaluation Action Quality Assessment

Ask PDF

Mar 25, 2026

Evaluating Vision-Language Models for Image Quality Assessment using Psychophysical Data

Psychophysical experiments remain the most reliable approach for perceptual image quality assessment (IQA), yet their cost and limited scalability motivate automated alternatives. This paper investigates whether Vision-Language Models (VLMs) can assist in assessing perceived image appearance and quality. We introduce a psychophysics-inspired framework to probe VLM perceptual sensitivity through controlled pairwise image comparisons of contrast, colorfulness, and overall preference. Six VLMs (four proprietary and two open-weight models) are compared against psychophysical data. Results reveal strong attribute-dependent variability: Claude exhibits the highest internal consistency, whereas GPT achieves the strongest agreement for overall preference. Claude and Qwen show the strongest alignment for colorfulness, while Qwen performs best for contrast. However, no model consistently matches human perception across all attributes. High self-consistency does not necessarily imply perceptual validity, and VLM--human agreement generally improves when perceptual differences among renderings are more pronounced.

Imran Mehmood, Imad Ali Shah, Ming Ronnier Luo +1

School of Engineering, University of Galway, Ireland · Lero the Research Ireland Centre for Software, Ireland · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou, China.

Image Quality Assessment VLM Evaluation Vision-Language Alignment

Ask PDF

Mar 19, 2026

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes (

n{=}1{,}968

). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, X-VLA mean-pool, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas

Bryce Grant, Xijia Zhao, Peng Wang

Case Western Reserve University

Vision-Language-Action Models Robotics

Ask PDF