AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty
Authors: Yan Ma, Lizhuo Zhang
Organizations: School of Foreign Studies, Changsha University of Science and Technology, Changsha, China and School of Education, Hunan Agricultural University, Changsha, China · School of Information and Intelligence, Hunan Agricultural University, Changsha, China
Multimodal large language models (MLLMs) are widely used for automated annotation, yet their per-class accuracy varies widely (e.g., 12%-98% across the 13 classes of three classroom sub-datasets) and is expensive to measure: evaluating one 27B MLLM on 5,416 validation images takes roughly 14 hours, whereas a frozen-CLIP pass over the same images completes in about 3 minutes. A low-cost signal for ranking classes by expected MLLM annotation difficulty a priori remains underexplored. Building on the AnchorProxy construct (per-class zero-shot CLIP accuracy) introduced in the companion study, this paper systematically evaluates its full-frame formulation, termed AnchorScore here, as an a priori diagnostic that flags the classes MLLMs are least likely to annotate reliably. On classroom behavior data (SCB5, 13 classes, 6 MLLMs), AnchorScore correlates with per-class MLLM accuracy (Spearman rho = 0.769, p = 0.002, n = 13). None of the alternative difficulty predictors (DINOv2, ResNet-50, SigLIP, or MLLM self-verbalized uncertainty) showed a significant class-level correlation at n = 13. A cross-model consensus control suggests AnchorScore primarily captures a shared class-difficulty factor rather than a CLIP-specific signal. An independent replication on Stanford40 Actions yields a nearly identical effect (rho = 0.817, p < 0.001); the association is strongest on activity-recognition data and attenuates on medical and satellite imagery. Three practical applications follow: a deployable hybrid CLIP/MLLM routing strategy (predicted-class routing: up to +23 pp over CLIP-only at roughly 44% fewer MLLM calls), prompt disambiguation on hard classes (exploratory), and review-priority prediction for human verification. AnchorScore does not estimate exact MLLM accuracy; it provides a low-cost ranking signal that directs expensive MLLM evaluation to the classes where it is most informative.
Figures & tables
Figure 1: AnchorScore at a glance. (a) Cost asymmetry: evaluating one 27B MLLM on the 5,416 SCB5 validation images takes roughly 14 hours, whereas a single frozen-CLIP forward pass yields per-class scores in about 3 minutes ( ≈ 270 × fewer FLOPs, no additional training). (b) Illustrative classes ranked by AnchorScore: per-class CLIP zero-shot accuracy (left bars) and six-MLLM mean accuracy (right bars) co-move from “teacher” (high AnchorScore) to “answer” (very low AnchorScore); values are the canonical per-class estimates of § 4.3 .
Model
HuggingFace ID
Architecture
Qwen series
Qwen3.5-27B †
Qwen/Qwen3.5-27B
Dense (27B)
Qwen3.6-27B †
Qwen/Qwen3.6-27B
Dense (27B)
Qwen3.5-35B-A3B †
Qwen/Qwen3.5-35B-A3B
MoE (35B, 3B active)
Qwen3.6-35B-A3B †
Qwen/Qwen3.6-35B-A3B
MoE (35B, 3B active)
Google Gemma series
Table 1: MLLMs evaluated in this study. HuggingFace model identifiers are provided for reproducibility. The core SCB5 analysis uses the six Qwen and Gemma-4 models (marked † ). Coverage of the additional models varies by experiment, as described in § 4.2 .
Figure 2: AnchorScore vs. MLLM accuracy across 78 data points (48 TeacherBehavior, 18 HandriseReadWrite, 12 BowTurnHead). Each point is one MLLM–class pair, jittered horizontally for visibility; colors and marker shapes indicate sub-datasets, and large filled markers show class means with error bars ( ±1 SD across MLLMs). The dashed line and gray band show the OLS fit and its 95% confidence band on class means; the primary inference is the class-level correlation ( ρ=0.769 , p=0.002 , n=13 ).
Predictor
Spearman ρ
p
Requires
AnchorScore (CLIP zero-shot acc.)
0.769
0.002
CLIP only
CLIP inverse entropy †
0.604
0.029\lx@sectionsign
CLIP only
CLIP confidence / margin †
0.434
0.14
CLIP only
SigLIP zero-shot accuracy
0.201
0.51
SigLIP model
DINOv2 kNN accuracy
+0.050
0.87
DINOv2 + labels
ResNet-50 accuracy (supervised on SCB5)
+0.081
0.79
Labels + GPU training
Table 2: Baseline comparison at the class level ( n=13 ). AnchorScore substantially outperforms all alternative predictors. CLIP inverse entropy reaches nominal significance ( p=0.029 ) but does not survive Benjamini–Hochberg correction (threshold p<0.006 for 8 predictors, q=0.05 ). Bold = best predictor.
Figure 3: AnchorScore across diverse domains spanning satellite imagery (EuroSAT), medical pathology (PathMNIST, BloodMNIST, TissueMNIST), and classroom scenes (SCB5, included as an in-domain reference, separated by a dashed divider). Bars show overall accuracy weighted by class size; error bars show ±1 SEM of per-class accuracies within each dataset. All AnchorScores use CLIP ViT-L/14 (LAION-2B).
Dataset
Cls
Chance%
Anchor%
Qwen2-VL%
Qwen2.5-VL%
LLaVA-1.5%
LLaVA-NeXT%
Mean%
EuroSAT
10
10.0
38.7
36.8
32.4
48.2
43.6
40.3
TissueMNIST
8
12.5
15.6
12.3
12.5
10.5
12.5
11.9
PathMNIST
9
11.1
15.5
11.8
18.9
15.3
11.1
14.3
BloodMNIST
8
12.5
5.9
15.5
16.0
14.8
15.3
15.4
Table 3: Cross-domain validation: AnchorScore predicts MLLM accuracy beyond classroom domains, though the signal is domain-dependent—the between-domain ordering is partially preserved (EuroSAT highest in every column), while within-dataset correlations are weak on medical data. Four MLLMs evaluated on 50 images per class (zero-shot). Chance rate = 1/C per dataset. Pooled per-class Spearman ρ=0.462 ( p=0.006 , n=34 ). Bold = best per column.
Analysis
ρ / Value
95% CI / p
Key finding
A. Sensitivity analyses
Leave-one-class-out
0.706–0.909
all p<0.02
No single class dominates
Leave-one-MLLM-out ( n=6 )
0.725–0.808
all p≤0.005
Consistent across MLLMs †
TeacherBehavior-only ( k=8 )
0.595
p=0.12
Directional; power-limited
+ LLaVA (third family)
0.819
p<0.001
Stronger with broader coverage
Mixed-effects (MLLM RE)
β=0.849
—
Mixed-effects sensitivity check; ICC ≈ 0
Table 4: Comprehensive robustness analysis. All sensitivity checks confirm the positive AnchorScore–MLLM association: the signal survives class removal (LOCO), model exclusion (LOMO), backbone substitution (Panel B; the class-level comparison is underpowered and should not be read as a rigorous capacity gradient), input-representation change (Panel C), and dataset variation (LODO). Bold = best per panel.
Figure 4: Forest plot summarizing the point-estimate robustness checks of the AnchorScore–MLLM association (the range-based sensitivity analyses, e.g., leave-one-class-out and leave-one-MLLM-out, appear in Table 4 ). Point estimates with 95% Fisher z -transform CIs (error bars); pooled estimates shown as diamonds. These CIs are methodologically consistent with, but may differ slightly from, the bootstrap CIs in the corresponding tables.
Figure 5: The AnchorScore pipeline. (A) AnchorScore computation: per-class validation images with domain prompts pass through frozen CLIP ViT-L/14 zero-shot inference, yielding a per-class AnchorScore. (B) Downstream workflows: (i) hybrid routing thresholds the AnchorScore of CLIP’s predicted class ( τ ), accepting high-score predictions from CLIP and sending low-score images to the MLLM (gains of + 1 to + 23 pp at 44–77% cost savings across the deployable datasets; Table 5 Panel A); (ii) prompt disambiguation derives enhanced MLLM prompts from CLIP’s confusion matrix (combined + 25.0 pp with direct visual-distinction descriptions; § 5.2 ).
Panel A: deployable predicted-class routing
Dataset
CLIP
Deploy acc
Fixed range
Cost
τ∗
(mean MLLM)
(gain, pp)
saved
TeacherBehavior (8 cls)
32.4%
55.7%
+ 21.8 to + 24.6
44%
45
HandriseReadWrite (3 cls)
61.0%
61.0%
+ 0.0 (CLIP-only)
100%
0
BowTurnHead (2 cls)
60.4%
82.0%
+ 19.5 to + 23.4
61%
50
Stanford40 (40 cls)
91.9%
93.3%
+ 1.0 to + 1.6
77%
95
Table 5: Hybrid CLIP/MLLM routing. Panel A (deployable): each image is routed by CLIP’s predicted class; accuracy is an expected value over per-image CLIP predictions and per-class MLLM rates, validated by a realized Qwen3.5-27B run on TeacherBehavior (54.1% realized vs. 55.9% expected; § 5.1 ). “Fixed range” = per-MLLM gains with a single committed MLLM. Panel B (class-aware bound): images are routed by their true class, bounding only policies that decide by true class; useful for class-known settings. Cost saved: Panel A = fraction of images not sent to the MLLM (CLIP cost treated as negligible); Panel B = savings under a 100:1 MLLM:CLIP per-image cost ratio.
Figure 6: Pareto visualization of hybrid routing: overall accuracy ( y -axis) vs. MLLM cost saved ( x -axis); better operating points lie toward the upper-left. Open circles with dashed horizontal lines are the all-CLIP baselines (no MLLM calls, 99% cost saving): TeacherBehavior 32.4%, HandriseReadWrite 61.0%, BowTurnHead 60.4%, Stanford40 91.9%. Filled squares are the class-aware bound (routing decided on the true class; a theoretical reference, not deployable). Filled triangles are the deployable predicted-class operating points: TeacherBehavior 55.7% at 43% saving ( + 23.3 pp), BowTurnHead 82.0% at 60% ( + 21.6 pp), Stanford40 93.3% at 76% ( + 1.4 pp); a triangle above its baseline means accuracy gained while cost is saved. HandriseReadWrite’s deployable optimum is CLIP-only, so no triangle is drawn. The Stanford40 curve (connected circles) shows the full bound trade-off as the threshold τ varies from 0 to 100; only Stanford40 is drawn as a curve because its 40-class bound is smooth, whereas the SCB5 bounds (8/3/2 classes) are step functions that would clutter the figure. The x -axis uses the 100:1 MLLM:CLIP per-image cost-ratio convention (all-CLIP = 99%); the deployable markers differ from Table 5 Panel A’s image-fraction savings by ≤ 0.8 pp.
Qwen3.5-27B
Qwen3.6-27B
Class
Anchor
Std
Neg
Δ
Std
Neg
Δ
blackboard
3.2%
5.0
10.0
+ 5.0 † [0, 15]
35.0
5.0
− 30.0 ∗ [ − 50, − 10]
answer
5.6%
10.0
40.0
+ 30.0 ∗ [10, 50]
10.0
15.0
+ 5.0 † [0, 15]
stand
9.6%
25.0
70.0
+ 45.0 ∗ [25, 65]
15.0
20.0
+ 5.0 † [ − 10, 20]
read
23.0%
65.0
80.0
+ 15.0 † [0, 30]
65.0
80.0
+ 15.0 † [0, 30]
BowHead
49.1%
80.0
85.0
+ 5.0 † [ − 10, 20]
75.0
90.0
+ 15.0 † [0, 30]
Table 6: Prompt disambiguation with negation-style hints (“this image may show {cls}, not {confuser}.”). Gains are limited (combined + 11.0 pp) and can backfire (blackboard on Qwen3.6-27B: − 30.0 pp).
Qwen3.5-27B
Qwen3.6-27B
Class
Anchor
Std
Dir
Δ
Std
Dir
Δ
blackboard
3.2%
5.0
55.0
+ 50.0 ∗ [30, 70]
35.0
65.0
+ 30.0 † [ − 10, 65]
answer
5.6%
10.0
80.0
+ 70.0 ∗ [50, 90]
10.0
65.0
+ 55.0 ∗ [35, 75]
stand
9.6%
25.0
20.0
− 5.0 † [ − 25, 15]
15.0
20.0
+ 5.0 † [0, 15]
read
23.0%
65.0
75.0
+ 10.0 † [0, 25]
65.0
75.0
+ 10.0 † [0, 25]
BowHead
49.1%
80.0
90.0
+ 10.0 † [ − 10, 30]
75.0
90.0
+ 15.0 † [0, 30]
Table 7: Prompt disambiguation with direct visual-distinction descriptions. Substantially more effective than negation (combined + 25.0 pp vs. + 11.0 pp), with 3/10 class–model pairs significantly improved and no significant decreases.
Qwen3.5-27B
Qwen3.6-27B
Class
Anchor
Gen
Wr
Δ
Gen
Wr
Δ
blackboard
3.2%
0.0
25.0
+ 30.0 † [0, 55]
5.0
10.0
+ 55.0 ∗ [35, 75]
answer
5.6%
0.0
20.0
+ 60.0 ∗ [40, 80]
0.0
10.0
+ 55.0 ∗ [35, 75]
stand
9.6%
20.0
45.0
− 25.0 † [ − 50, 0]
10.0
20.0
+ 0.0 † [0, 0]
read
23.0%
70.0
75.0
+ 0.0 † [ − 15, 15]
70.0
75.0
+ 0.0 † [0, 0]
BowHead
49.1%
80.0
—
—
85.0
—
—
Table 8: Confuser controls for the direct condition, on the identical 20 images per class. Gen: generic note naming no pair; Wr: direct template with the second-most-confused CLIP confuser. Δ = direct − Wr; direct accuracies are in Table 7 .
Dataset
k
AUC
Precision@ k/3
Recall@ k/3
TeacherBehavior
8
0.938 [0.667, 1.000]
1.000
0.750
Stanford40
40
0.842 [0.690, 0.958]
0.857
0.632
HandriseReadWrite †
3
1.000
1.000
1.000
BowTurnHead †
2
1.000
1.000
1.000
Table 9: Review priority ranking: AnchorScore identifies hard classes (below-median MLLM accuracy) with AUC > 0.84 on both TeacherBehavior and Stanford40, supporting review-priority ranking. Bootstrap 95% CI (5000 replicates) for TeacherBehavior and Stanford40; HandriseReadWrite ( k=3 ) and BowTurnHead ( k=2 ) have trivially perfect AUC at such small k .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Class
Anchor%
MLLM avg%
MLLM std%
Δ (pp)
TeacherBehavior
blackboard
3.2
44.6
15.6
+ 41.4
answer
5.6
12.4
7.6
+ 6.8
stand
9.6
23.1
6.7
+ 13.5
blackboard-writing
26.1
97.5
1.9
+ 71.4
guide
29.3
42.5
22.6
+ 13.2
Appendix
Table A1: Per-class AnchorScore and average MLLM accuracy across all 13 SCB5 classes (TeacherBehavior, HandriseReadWrite, BowTurnHead; 6 MLLMs), grouped by sub-dataset and sorted by ascending AnchorScore. MLLM std is the sample standard deviation of the six per-model accuracies.
Class
Anchor%
MLLM%
Δ
Class
Anchor%
MLLM%
Δ
cutting vegetables
47.6
81.3
+ 33.7
reading
97.6
96.0
− 1.6
texting message
52.9
90.3
+ 37.5
cooking
97.6
99.3
+ 1.8
writing on a book
58.9
89.7
+ 30.7
watching TV
97.8
98.7
+ 0.9
pouring liquid
61.0
96.3
+ 35.3
holding an umbrella
98.0
99.7
+ 1.7
applauding
69.4
91.3
+ 22.0
throwing frisbee
98.0
98.0
+ 0.0
waving hands
78.1
99.3
+ 21.2
riding a bike
98.3
100.0
+ 1.7
Appendix
Table A2: Per-class AnchorScore and average MLLM accuracy on Stanford40 (40 classes, 6 MLLMs), sorted by ascending AnchorScore. Left panel: classes 1–20; right panel: classes 21–40. MLLM std omitted for space.
Class
Anchor%
MLLM avg%
MLLM std%
Δ
EuroSAT (10 classes; within-dataset ρ=0.286 )
HerbaceousVegetation
0.3
1.0
1.2
+ 0.7
AnnualCrop
0.5
36.5
41.5
+ 36.0
Forest
2.5
56.0
24.7
+ 53.5
Pasture
13.7
36.5
41.8
+ 22.9
SeaLake
23.7
41.0
36.2
+ 17.3
Appendix
Table A3: Per-class AnchorScore and mean MLLM accuracy (4 MLLMs) across the four cross-domain datasets, sorted by ascending AnchorScore within each dataset. MLLM std is the sample standard deviation of the four per-model accuracies.
Prompt variant
T
ρ
p
Domain context?
3-template average (original)
3
0.769
0.002
Yes
“a photo of {cls} in a classroom”
1
0.736
0.004
Yes
Single: “a person {cls} in classroom”
1
0.670
0.012
Yes
Alternative 3-template phrasing
3
0.599
0.031
Yes
“a photo of {cls}.”
1
0.238
0.435
No
“a photo of a person {cls}.”
1
0.149
0.627
No
Appendix
Table A4: Prompt robustness: class-level Spearman ρ between AnchorScore and MLLM accuracy under six prompt variants on SCB5 ( n=13 , 6-MLLM subset).
Figure A1: Calibration plot: AnchorScore vs. mean MLLM accuracy across 47 classes (5 bins); perfect calibration would equalize bar heights per bin. The lower panel quantifies the gap (MLLM − CLIP in pp); ECE = 0.152, consistent with AnchorScore supporting ranking rather than absolute accuracy estimation.
Multimodal Large Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and Vision-Language Models (VLMs) report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLM underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLM predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale dataset curation. This work is part of the Aiming for Perfect ImageNet-1k project, see https://klarajanouskova.github.io/ImageNet/.
Nikita Kisel, Illia Volkov, Klara Janouskova +1
Visual Recognition Group, Czech Technical University in Prague
Large language models (LLMs) are increasingly used in annotation tasks with varying success. While several studies have demonstrated the opportunities and shortcomings of LLMs for annotation, a critical question remains unanswered: is it possible to predict how well they will perform on a new dataset? To answer this and related questions, we construct two task families which induce 461 text classification datasets. We find that it is possible to predict LLM performance - to a certain degree - and that contrary to other work which highlights the usefulness of LLM verbalizations of self-confidence, internal log-probabilities are the most predictive signal.
Image tagging, a fundamental vision task, traditionally relies on human-annotated datasets to train multi-label classifiers, which incurs significant labor and costs. While Multimodal Large Language Models (MLLMs) offer promising potential to automate annotation, their capability to replace human annotators remains underexplored. This paper aims to analyze the gap between MLLM-generated and human annotations and to propose an effective solution that enables MLLM-based annotation to replace manual labeling. Our analysis of MLLM annotations reveals that, under a conservative estimate, MLLMs can reduce annotation cost to as low as one-thousandth of the human cost, mainly accounting for GPU usage, which is nearly negligible compared to manual efforts. Their annotation quality reaches about 50% to 80% of human performance, while achieving over 90% performance on downstream training tasks.Motivated by these findings, we propose TagLLM, a novel framework for image tagging, which aims to narrow the gap between MLLM-generated and human annotations. TagLLM comprises two components: Candidates generation, which employs structured group-wise prompting to efficiently produce a compact candidate set that covers as many true labels as possible while reducing subsequent annotation workload; and label disambiguation, which interactively calibrates the semantic concept of categories in the prompts and effectively refines the candidate labels. Extensive experiments show that TagLLM substantially narrows the gap between MLLM-generated and human annotations, especially in downstream training performance, where it closes about 60% to 80% of the difference.
Ming-Kun Xie, Jia-Hao Xiao, Zhiqiang Kou +3
RIKEN Center for Advanced Intelligence Project, Japan · Southeast University, China · China University of Mining and Technology, China +1