AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty

Yan Ma; Lizhuo Zhang

cs.CVAug 17, 2026

AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty

Authors: Yan Ma, Lizhuo Zhang

Organizations: School of Foreign Studies, Changsha University of Science and Technology, Changsha, China and School of Education, Hunan Agricultural University, Changsha, China · School of Information and Intelligence, Hunan Agricultural University, Changsha, China

Abstract

Multimodal large language models (MLLMs) are widely used for automated annotation, yet their per-class accuracy varies widely (e.g., 12%-98% across the 13 classes of three classroom sub-datasets) and is expensive to measure: evaluating one 27B MLLM on 5,416 validation images takes roughly 14 hours, whereas a frozen-CLIP pass over the same images completes in about 3 minutes. A low-cost signal for ranking classes by expected MLLM annotation difficulty a priori remains underexplored. Building on the AnchorProxy construct (per-class zero-shot CLIP accuracy) introduced in the companion study, this paper systematically evaluates its full-frame formulation, termed AnchorScore here, as an a priori diagnostic that flags the classes MLLMs are least likely to annotate reliably. On classroom behavior data (SCB5, 13 classes, 6 MLLMs), AnchorScore correlates with per-class MLLM accuracy (Spearman rho = 0.769, p = 0.002, n = 13). None of the alternative difficulty predictors (DINOv2, ResNet-50, SigLIP, or MLLM self-verbalized uncertainty) showed a significant class-level correlation at n = 13. A cross-model consensus control suggests AnchorScore primarily captures a shared class-difficulty factor rather than a CLIP-specific signal. An independent replication on Stanford40 Actions yields a nearly identical effect (rho = 0.817, p < 0.001); the association is strongest on activity-recognition data and attenuates on medical and satellite imagery. Three practical applications follow: a deployable hybrid CLIP/MLLM routing strategy (predicted-class routing: up to +23 pp over CLIP-only at roughly 44% fewer MLLM calls), prompt disambiguation on hard classes (exploratory), and review-priority prediction for human verification. AnchorScore does not estimate exact MLLM accuracy; it provides a low-cost ranking signal that directs expensive MLLM evaluation to the classes where it is most informative.

Figures & tables

Figure 1: AnchorScore at a glance. (a) Cost asymmetry: evaluating one 27B MLLM on the 5,416 SCB5 validation images takes roughly 14 hours, whereas a single frozen-CLIP forward pass yields per-class scores in about 3 minutes (

\approx

270

\times

fewer FLOPs, no additional training). (b) Illustrative classes ranked by AnchorScore: per-class CLIP zero-shot accuracy (left bars) and six-MLLM mean accuracy (right bars) co-move from “teacher” (high AnchorScore) to “answer” (very low AnchorScore); values are the canonical per-class estimates of § 4.3 .

Model	HuggingFace ID	Architecture
Qwen series
Qwen3.5-27B †	Qwen/Qwen3.5-27B	Dense (27B)
Qwen3.6-27B †	Qwen/Qwen3.6-27B	Dense (27B)
Qwen3.5-35B-A3B †	Qwen/Qwen3.5-35B-A3B	MoE (35B, 3B active)
Qwen3.6-35B-A3B †	Qwen/Qwen3.6-35B-A3B	MoE (35B, 3B active)
Google Gemma series

Table 1: MLLMs evaluated in this study. HuggingFace model identifiers are provided for reproducibility. The core SCB5 analysis uses the six Qwen and Gemma-4 models (marked † ). Coverage of the additional models varies by experiment, as described in § 4.2 .

Figure 2: AnchorScore vs. MLLM accuracy across 78 data points (48 TeacherBehavior, 18 HandriseReadWrite, 12 BowTurnHead). Each point is one MLLM–class pair, jittered horizontally for visibility; colors and marker shapes indicate sub-datasets, and large filled markers show class means with error bars (

\pm 1

SD across MLLMs). The dashed line and gray band show the OLS fit and its 95% confidence band on class means; the primary inference is the class-level correlation (

\rho=0.769

p=0.002

n=13

Predictor	Spearman $\rho$	$p$	Requires
AnchorScore (CLIP zero-shot acc.)	0.769	0.002	CLIP only
CLIP inverse entropy †	0.604	$0.029^{\lx@sectionsign}$	CLIP only
CLIP confidence / margin †	0.434	0.14	CLIP only
SigLIP zero-shot accuracy	0.201	0.51	SigLIP model
DINOv2 kNN accuracy	+0.050	0.87	DINOv2 + labels
ResNet-50 accuracy (supervised on SCB5)	+0.081	0.79	Labels + GPU training

Table 2: Baseline comparison at the class level (

n=13

). AnchorScore substantially outperforms all alternative predictors. CLIP inverse entropy reaches nominal significance (

p=0.029

) but does not survive Benjamini–Hochberg correction (threshold

p<0.006

for 8 predictors,

q=0.05

). Bold = best predictor.

Figure 3: AnchorScore across diverse domains spanning satellite imagery (EuroSAT), medical pathology (PathMNIST, BloodMNIST, TissueMNIST), and classroom scenes (SCB5, included as an in-domain reference, separated by a dashed divider). Bars show overall accuracy weighted by class size; error bars show

\pm 1

SEM of per-class accuracies within each dataset. All AnchorScores use CLIP ViT-L/14 (LAION-2B).

Dataset	Cls	Chance%	Anchor%	Qwen2-VL%	Qwen2.5-VL%	LLaVA-1.5%	LLaVA-NeXT%	Mean%
EuroSAT	10	10.0	38.7	36.8	32.4	48.2	43.6	40.3
TissueMNIST	8	12.5	15.6	12.3	12.5	10.5	12.5	11.9
PathMNIST	9	11.1	15.5	11.8	18.9	15.3	11.1	14.3
BloodMNIST	8	12.5	5.9	15.5	16.0	14.8	15.3	15.4

Table 3: Cross-domain validation: AnchorScore predicts MLLM accuracy beyond classroom domains, though the signal is domain-dependent—the between-domain ordering is partially preserved (EuroSAT highest in every column), while within-dataset correlations are weak on medical data. Four MLLMs evaluated on 50 images per class (zero-shot). Chance rate =

1/C

per dataset. Pooled per-class Spearman

\rho=0.462

(

p=0.006

n=34

). Bold = best per column.

Analysis	$\rho$ / Value	95% CI / $p$	Key finding
A. Sensitivity analyses
Leave-one-class-out	0.706–0.909	all $p<0.02$	No single class dominates
Leave-one-MLLM-out ( $n=6$ )	0.725–0.808	all $p\leq 0.005$	Consistent across MLLMs †
TeacherBehavior-only ( $k=8$ )	0.595	$p=0.12$	Directional; power-limited
$+$ LLaVA (third family)	0.819	$p<0.001$	Stronger with broader coverage
Mixed-effects (MLLM RE)	$\beta=0.849$	—	Mixed-effects sensitivity check; ICC $\approx$ 0

Table 4: Comprehensive robustness analysis. All sensitivity checks confirm the positive AnchorScore–MLLM association: the signal survives class removal (LOCO), model exclusion (LOMO), backbone substitution (Panel B; the class-level comparison is underpowered and should not be read as a rigorous capacity gradient), input-representation change (Panel C), and dataset variation (LODO). Bold = best per panel.

Figure 4: Forest plot summarizing the point-estimate robustness checks of the AnchorScore–MLLM association (the range-based sensitivity analyses, e.g., leave-one-class-out and leave-one-MLLM-out, appear in Table 4 ). Point estimates with 95% Fisher

z

-transform CIs (error bars); pooled estimates shown as diamonds. These CIs are methodologically consistent with, but may differ slightly from, the bootstrap CIs in the corresponding tables.

Figure 5: The AnchorScore pipeline. (A) AnchorScore computation: per-class validation images with domain prompts pass through frozen CLIP ViT-L/14 zero-shot inference, yielding a per-class AnchorScore. (B) Downstream workflows: (i) hybrid routing thresholds the AnchorScore of CLIP’s predicted class (

\tau

), accepting high-score predictions from CLIP and sending low-score images to the MLLM (gains of

+

1 to

+

23 pp at 44–77% cost savings across the deployable datasets; Table 5 Panel A); (ii) prompt disambiguation derives enhanced MLLM prompts from CLIP’s confusion matrix (combined

+

25.0 pp with direct visual-distinction descriptions; § 5.2 ).

Panel A: deployable predicted-class routing
Dataset	CLIP	Deploy acc	Fixed range	Cost	$\tau^{*}$
		(mean MLLM)	(gain, pp)	saved
TeacherBehavior (8 cls)	32.4%	55.7%	$+$ 21.8 to $+$ 24.6	44%	45
HandriseReadWrite (3 cls)	61.0%	61.0%	$+$ 0.0 (CLIP-only)	100%	0
BowTurnHead (2 cls)	60.4%	82.0%	$+$ 19.5 to $+$ 23.4	61%	50
Stanford40 (40 cls)	91.9%	93.3%	$+$ 1.0 to $+$ 1.6	77%	95

Table 5: Hybrid CLIP/MLLM routing. Panel A (deployable): each image is routed by CLIP’s predicted class; accuracy is an expected value over per-image CLIP predictions and per-class MLLM rates, validated by a realized Qwen3.5-27B run on TeacherBehavior (54.1% realized vs. 55.9% expected; § 5.1 ). “Fixed range” = per-MLLM gains with a single committed MLLM. Panel B (class-aware bound): images are routed by their true class, bounding only policies that decide by true class; useful for class-known settings. Cost saved: Panel A = fraction of images not sent to the MLLM (CLIP cost treated as negligible); Panel B = savings under a 100:1 MLLM:CLIP per-image cost ratio.

Figure 6: Pareto visualization of hybrid routing: overall accuracy (

y

-axis) vs. MLLM cost saved (

x

-axis); better operating points lie toward the upper-left. Open circles with dashed horizontal lines are the all-CLIP baselines (no MLLM calls, 99% cost saving): TeacherBehavior 32.4%, HandriseReadWrite 61.0%, BowTurnHead 60.4%, Stanford40 91.9%. Filled squares are the class-aware bound (routing decided on the true class; a theoretical reference, not deployable). Filled triangles are the deployable predicted-class operating points: TeacherBehavior 55.7% at 43% saving (

+

23.3 pp), BowTurnHead 82.0% at 60% (

+

21.6 pp), Stanford40 93.3% at 76% (

+

1.4 pp); a triangle above its baseline means accuracy gained while cost is saved. HandriseReadWrite’s deployable optimum is CLIP-only, so no triangle is drawn. The Stanford40 curve (connected circles) shows the full bound trade-off as the threshold

\tau

varies from 0 to 100; only Stanford40 is drawn as a curve because its 40-class bound is smooth, whereas the SCB5 bounds (8/3/2 classes) are step functions that would clutter the figure. The

x

-axis uses the 100:1 MLLM:CLIP per-image cost-ratio convention (all-CLIP

=

99%); the deployable markers differ from Table 5 Panel A’s image-fraction savings by

\leq

0.8 pp.

Class	Anchor	Std	Neg	$\Delta$	Std	Neg	$\Delta$
		Qwen3.5-27B			Qwen3.6-27B
blackboard	3.2%	5.0	10.0	$+$ 5.0 † [0, 15]	35.0	5.0	$-$ 30.0 ∗ [ $-$ 50, $-$ 10]
answer	5.6%	10.0	40.0	$+$ 30.0 ∗ [10, 50]	10.0	15.0	$+$ 5.0 † [0, 15]
stand	9.6%	25.0	70.0	$+$ 45.0 ∗ [25, 65]	15.0	20.0	$+$ 5.0 † [ $-$ 10, 20]
read	23.0%	65.0	80.0	$+$ 15.0 † [0, 30]	65.0	80.0	$+$ 15.0 † [0, 30]
BowHead	49.1%	80.0	85.0	$+$ 5.0 † [ $-$ 10, 20]	75.0	90.0	$+$ 15.0 † [0, 30]

Table 6: Prompt disambiguation with negation-style hints (“this image may show {cls}, not {confuser}.”). Gains are limited (combined

+

11.0 pp) and can backfire (blackboard on Qwen3.6-27B:

-

30.0 pp).

Class	Anchor	Std	Dir	$\Delta$	Std	Dir	$\Delta$
		Qwen3.5-27B			Qwen3.6-27B
blackboard	3.2%	5.0	55.0	$+$ 50.0 ∗ [30, 70]	35.0	65.0	$+$ 30.0 † [ $-$ 10, 65]
answer	5.6%	10.0	80.0	$+$ 70.0 ∗ [50, 90]	10.0	65.0	$+$ 55.0 ∗ [35, 75]
stand	9.6%	25.0	20.0	$-$ 5.0 † [ $-$ 25, 15]	15.0	20.0	$+$ 5.0 † [0, 15]
read	23.0%	65.0	75.0	$+$ 10.0 † [0, 25]	65.0	75.0	$+$ 10.0 † [0, 25]
BowHead	49.1%	80.0	90.0	$+$ 10.0 † [ $-$ 10, 30]	75.0	90.0	$+$ 15.0 † [0, 30]

Table 7: Prompt disambiguation with direct visual-distinction descriptions. Substantially more effective than negation (combined

+

25.0 pp vs.

+

11.0 pp), with 3/10 class–model pairs significantly improved and no significant decreases.

Class	Anchor	Gen	Wr	$\Delta$	Gen	Wr	$\Delta$
		Qwen3.5-27B			Qwen3.6-27B
blackboard	3.2%	0.0	25.0	$+$ 30.0 † [0, 55]	5.0	10.0	$+$ 55.0 ∗ [35, 75]
answer	5.6%	0.0	20.0	$+$ 60.0 ∗ [40, 80]	0.0	10.0	$+$ 55.0 ∗ [35, 75]
stand	9.6%	20.0	45.0	$-$ 25.0 † [ $-$ 50, 0]	10.0	20.0	$+$ 0.0 † [0, 0]
read	23.0%	70.0	75.0	$+$ 0.0 † [ $-$ 15, 15]	70.0	75.0	$+$ 0.0 † [0, 0]
BowHead	49.1%	80.0	—	—	85.0	—	—

Table 8: Confuser controls for the direct condition, on the identical 20 images per class. Gen: generic note naming no pair; Wr: direct template with the second-most-confused CLIP confuser.

\Delta

= direct

-

Wr; direct accuracies are in Table 7 .

Dataset	$k$	AUC	Precision@ $k/3$	Recall@ $k/3$
TeacherBehavior	8	0.938 [0.667, 1.000]	1.000	0.750
Stanford40	40	0.842 [0.690, 0.958]	0.857	0.632
HandriseReadWrite †	3	1.000	1.000	1.000
BowTurnHead †	2	1.000	1.000	1.000

Table 9: Review priority ranking: AnchorScore identifies hard classes (below-median MLLM accuracy) with AUC

>

0.84 on both TeacherBehavior and Stanford40, supporting review-priority ranking. Bootstrap 95% CI (5000 replicates) for TeacherBehavior and Stanford40; HandriseReadWrite (

k=3

) and BowTurnHead (

k=2

) have trivially perfect AUC at such small

k

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Class	Anchor%	MLLM avg%	MLLM std%	$\Delta$ (pp)
TeacherBehavior
blackboard	3.2	44.6	15.6	$+$ 41.4
answer	5.6	12.4	7.6	$+$ 6.8
stand	9.6	23.1	6.7	$+$ 13.5
blackboard-writing	26.1	97.5	1.9	$+$ 71.4
guide	29.3	42.5	22.6	$+$ 13.2

Appendix

Table A1: Per-class AnchorScore and average MLLM accuracy across all 13 SCB5 classes (TeacherBehavior, HandriseReadWrite, BowTurnHead; 6 MLLMs), grouped by sub-dataset and sorted by ascending AnchorScore. MLLM std is the sample standard deviation of the six per-model accuracies.

Class	Anchor%	MLLM%	$\Delta$	Class	Anchor%	MLLM%	$\Delta$
cutting vegetables	47.6	81.3	$+$ 33.7	reading	97.6	96.0	$-$ 1.6
texting message	52.9	90.3	$+$ 37.5	cooking	97.6	99.3	$+$ 1.8
writing on a book	58.9	89.7	$+$ 30.7	watching TV	97.8	98.7	$+$ 0.9
pouring liquid	61.0	96.3	$+$ 35.3	holding an umbrella	98.0	99.7	$+$ 1.7
applauding	69.4	91.3	$+$ 22.0	throwing frisbee	98.0	98.0	$+$ 0.0
waving hands	78.1	99.3	$+$ 21.2	riding a bike	98.3	100.0	$+$ 1.7

Appendix

Table A2: Per-class AnchorScore and average MLLM accuracy on Stanford40 (40 classes, 6 MLLMs), sorted by ascending AnchorScore. Left panel: classes 1–20; right panel: classes 21–40. MLLM std omitted for space.

Class	Anchor%	MLLM avg%	MLLM std%	$\Delta$
EuroSAT (10 classes; within-dataset $\rho=0.286$ )
HerbaceousVegetation	0.3	1.0	1.2	$+$ 0.7
AnnualCrop	0.5	36.5	41.5	$+$ 36.0
Forest	2.5	56.0	24.7	$+$ 53.5
Pasture	13.7	36.5	41.8	$+$ 22.9
SeaLake	23.7	41.0	36.2	$+$ 17.3

Appendix

Table A3: Per-class AnchorScore and mean MLLM accuracy (4 MLLMs) across the four cross-domain datasets, sorted by ascending AnchorScore within each dataset. MLLM std is the sample standard deviation of the four per-model accuracies.

Prompt variant	$T$	$\rho$	$p$	Domain context?
3-template average (original)	3	0.769	0.002	Yes
“a photo of {cls} in a classroom”	1	0.736	0.004	Yes
Single: “a person {cls} in classroom”	1	0.670	0.012	Yes
Alternative 3-template phrasing	3	0.599	0.031	Yes
“a photo of {cls}.”	1	0.238	0.435	No
“a photo of a person {cls}.”	1	0.149	0.627	No

Appendix

Table A4: Prompt robustness: class-level Spearman

\rho

between AnchorScore and MLLM accuracy under six prompt variants on SCB5 (

n=13

, 6-MLLM subset).

Figure A1: Calibration plot: AnchorScore vs. mean MLLM accuracy across 47 classes (5 bins); perfect calibration would equalize bar heights per bin. The lower panel quantifies the gap (MLLM

-

CLIP in pp); ECE

=

0.152, consistent with AnchorScore supporting ranking rather than absolute accuracy estimation.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

Multimodal Large Language Models as Image Classifiers
Mar 6, 2026Nikita Kisel, Illia Volkov, Klara Janouskova +1Multimodal Large Language Models Multimodal Classification
Visual Recognition Group, Czech Technical University in Prague
Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance
Oct 16, 2026Tong Lin, Grant Schoenebeck, Sabina TomkinsLanguage Model Performance Prediction LLM-Assisted Annotation
School of Information, University of Michigan
Are Multimodal Large Language Models Good Annotators for Image Tagging?
Feb 24, 2026Ming-Kun Xie, Jia-Hao Xiao, Zhiqiang Kou +3LLM-Assisted Annotation Multimodal Large Language Models
RIKEN Center for Advanced Intelligence Project, Japan · Southeast University, China · China University of Mining and Technology, China +1

Mar 6, 2026J/K move · Enter open · S save

Multimodal Large Language Models as Image Classifiers

Nikita Kisel, Illia Volkov, Klara Janouskova, Jiri Matas

Visual Recognition Group, Czech Technical University in Prague

Multimodal Large Language Models Multimodal Classification Multimodal Model Evaluation Image Classification

Open paper AskPDF arXiv

Multimodal Large Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and Vision-Language Models (VLMs) report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLM underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLM predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale dataset curation. This work is part of the Aiming for Perfect ImageNet-1k project, see https://klarajanouskova.github.io/ImageNet/.

Abstract

Multimodal Model Evaluation

Explore similar work

Multimodal Large Language Models as Image Classifiers

Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance

Are Multimodal Large Language Models Good Annotators for Image Tagging?