How Finely Can Vision-Language Models See Biology? Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning

S M Rayeed; Qian Ma; Charles V. Stewart; Yao Ma

cs.CVNov 3, 2026

How Finely Can Vision-Language Models See Biology? Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning

Authors: S M Rayeed, Qian Ma, Charles V. Stewart, Yao Ma

Organizations: Rensselaer Polytechnic Institute

Abstract

In biodiversity monitoring, a single image raises several questions: which taxon, how many, which trait, and whether the individual has been seen before. Vision-language models (VLMs) could answer all of these from the image, but their ability is largely untested in biology, as existing benchmarks focus mostly on species identification within a few taxonomic groups. We present Bio-VQA, the largest biological visual question answering (VQA) benchmark and the only one to span four aspects of biological visual understanding: hierarchical taxonomic reasoning, fine-grained attribute grounding, quantitative spatial reasoning, and individual re-identification. Bio-VQA contains 1.86M questions over 217K images from 11 public datasets, and extends right-or-wrong scoring to graded scoring, so a near-miss counts for more than a distant one. Evaluating 18 open-weight models from 3.75B to 38.4B parameters, we find that general visual capabilities do not transfer to biological organisms: the best model counts animals exactly on 41.7% of images with 2 to 10 individuals, less than half its accuracy on everyday objects at the same counts. Taxonomic accuracy falls to one sixth, from 0.72 at phylum to 0.12 at scientific name, and scaling helps mostly at coarse ranks. The gap to a domain specialist widens accordingly, from 0.731 against BioCLIP-2's 0.874 at class (1.2x) to 0.269 against 0.544 at scientific name (2x). On trait size measurement, 15 of the 18 models do no better than an image-blind baseline that predicts one fixed number for every specimen, and on re-identification, 17 of the 18 models fall below chance under adversarial conditions. These failures persist across a tenfold range of model size. We hope Bio-VQA becomes a standard test of how well VLM capabilities transfer to the biological domain.

Figures & tables

Figure 1: Bio-VQA is a benchmark on biological domain, consisting of four components: (a) hierarchical taxonomic reasoning, (b) fine-grained attribute grounding, (c) quantitative spatial reasoning (3 of 8 tasks shown: Ag counting , Ag OCR , Ag relative position ), (d) re-identification.

	Benchmark	QA	Img.	Org. grp.	Question type
	Benchmark	QA	Img.	Org. grp.	Tax	Att	Spa	Ind
General	MMBench [ 28 ]	3.0K	3.0K	—	✗	$\circ$	$\circ$	✗
	BLINK [ 12 ]	3.8K	7.4K	—	✗	$\circ$	✓	✗
	CV-Bench [ 50 ]	2.6K	2.6K	—	✗	$\circ$	✓	✗
	AVA-Bench [ 31 ]	218K	218K	—	✗	✓	✓	✗
Biology	J-EDI QA [ 65 ]	0.1K	0.1K	1	✓	✗	✗	✗

Table 1: Comparison with prior benchmarks. Columns report the number of question-answer pairs (QA), images (Img.) and organismal groups (Org. grp.); and whether they cover question types: taxonomic (Tax), attribute (Att), spatial (Spa) and individual identity (Ind). ✓ means the question type is a primary task family,

\circ

that it appears incidentally, and ✗ that it is absent.

Figure 2: C1 asks one image for its identity at eight ranks, from phylum down to common name, and repeats common name as four-way multiple choice. Species is the epithet and scientific name is the binomial, so we score the two separately.

Figure 3: C2 poses four categories of question, with one example item each. We ask naming both open-ended and as multiple choice on the same image. Anatomical localization is visually prompted, so the question refers to a rendered marker rather than to a named part.

Figure 4: C3’s eight spatial tasks, with one example item each, its answer format and its ground truth. The first panel asks counting, OCR and relative position of the same beetle plate. The relative distance and counting tasks are multiple-choice, rest are open-ended.

	Component	Tasks	Data	Images	Questions
C1	Hierarchical taxonomic reasoning	9	1	100,000	890,000
C2	Fine-grained attribute grounding	4	1	23,576	824,104
C3	Quantitative spatial reasoning	8	8 †	78,139	132,504
C4	Individual re-identification	3	5	15,000	15,000
Total		24	11 †	216,715	1,861,608

Table 2: Bio-VQA statistics: We draw C1 from TreeOfLife-200M [ 18 ] ; C2 from CUB-200-2011 [ 55 ] ; C3 from WAID [ 36 ] , Bees [ 4 ] , 2018-NEON-Beetles [ 22 ] , DePondFi [ 54 ] , GZCD [ 37 ] ; and C4 from WhaleSharkID [ 59 ] , BelugaID [ 60 ] , HyenaID [ 61 ] , LeopardID [ 62 ] , GZCD [ 37 ] . † C3 uses WhaleSharkID, HyenaID, LeopardID, GZCD on two tasks: so the total count becomes 11.

#	Model	C1 (Exact match)								C2 ( cags )				C3								C4
#	Model	phy	cls	ord	fam	gen	spc	sci	com	sal	mrp	loc	nam	cnt	rcnt	adst	rdst	lcl	rpos	view	ocr	$d^{\prime}$	acc
	BioCLIP-2 (tol) ∗	.925	.874	.821	.775	.691	.545	.544	.500	—	—	—	—	—	—	—	—	—	—	—	—	—	—
	MegaDescriptor ∗∗	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	.733	.683
1	Qwen3-VL-32B	.831	.615	.712	.612	.444	.066	.244	.078	.584	.644	.475	.731	.789	.555	.622	.443	.866	.550	.427	.900	.696	.636
2	Qwen3-VL-30B-A3B	.864	.731	.757	.612	.417	.072	.221	.107	.558	.631	.453	.658	.795	.483	.528	.423	.861	.355	.391	.979	.518	.602
3	InternVL3-38B	.911	.683	.638	.484	.355	.054	.189	.044	.562	.645	.413	.669	.799	.567	.367	.363	.744	.205	.373	.105	.463	.581

Table 3: Bio-VQA results for 18 models across 24 tasks. Colors mark the best , second best , second worst and worst model in each column, over the 18 models only. C1 is exact match at eight ranks. C2 is cags for relative salience (sal), morphological attributes (mrp), anatomical localization (loc) and naming (nam). C3 reports one metric per task: cas for counting (cnt); balanced accuracy for relative counting (rcnt), relative distance (rdst) and relative position (rpos); the share of measurements within 20% of the true value for absolute distance (adst); mean IoU for localization (lcl) with unparsed replies scored as zero; agreement over gold labels for viewpoint (view); and exact match for OCR. C4 is discriminability (

d^{\prime}

) and accuracy under random pairing. Floor is the score an image-blind rule reaches, and the most frequent taxon at each rank for C1. cags and

d^{\prime}

are chance-corrected, so floors are zero. ∗ Domain specialist, taxonomy, with the full TreeOfLife-200M label set (tol). ∗∗ Domain specialist, re-identification, on Grevy’s zebra only.

Spearman $\rho$	C1	C2	C3	C4
C1 ( $\alpha$ .981)	—	.689	.787	.433 †
C2 ( $\alpha$ .901)		—	.927	.514 †
C3 ( $\alpha$ .925)			—	.610
C4 ( $\alpha$ .889)				—

Table 4: Correlation between components over the 18 models scored on all four.

\alpha

is each component’s internal consistency.

\dagger

not significant after Holm correction. Confidence intervals are wide at

n=18

(median width .66), enough to separate correlated from uncorrelated pairs but not to rank one pair above another.

Appendix figures & tables107 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Filter rank	Value
Amphibians	class	Amphibia
Angiosperms	class	Magnoliopsida , Liliopsida
Arachnids	class	Arachnida
Birds	class	Aves
Bryophytes	phylum	Bryophyta
Fish	class	Actinopterygii , Teleostei ,

Appendix

Table 5: 10 organismal groups in C1 and the backbone filter that defines each. Six of them do not appear in prior biological VQAs.

Question type	Scored rank	Example answer
phylum	phylum	Chordata
class	class	Amphibia
order	order	Anura
family	family	Hylidae
genus	genus	Dryophytes
species epithet	species	cinereus

Appendix

Table 6: The nine C1 question types. We ask the common name twice on the same image, once open-ended and once as four-way multiple choice to measure the recognition–recall gap.

Category	Family	Questions	Format
Morphological	attr_bare_value	$280\,433$	MCQ
Rel. salience	attr_group_value	$321\,073$	MCQ
Anatomical	marker_localization	$11\,788$	MCQ
	visibility_partset	$117\,880$	MCQ
	visibility_side	$35\,364$	MCQ
Naming	category_mcq	$11\,191$	MCQ

Appendix

Table 7: C2 categories and question families. About 94% of items are multiple choice questions (MCQ) with two to five distractors. The other 6% are open-ended (OE) naming.

Family	Question	Options	Answer
attr_bare_value	What is the bill color of this bird?	white / yellow / black / pink / brown	black
attr_group_value	Which of the following is the most prominently visible physical trait of the bird in this image?	bill color: black / underparts color: white / primary color: green / breast color: red / eye color: white	breast color: red
marker_localization	What is the body part of this bird marked with the yellow dot?	beak / breast / left leg / left eye / left wing	beak
visibility_partset	Which of the following parts of the bird are visible in this image?	breast / beak / left wing / None / All three	beak
visibility_side	Which of the eyes of the bird are visible in this image?	Left / Right / Both / None	Both
category_mcq	What type of bird is this?	Waxwing / Nutcracker / Waterthrush / Grebe / Gull	Gull

Appendix

Table 8: One example per C2 family, all asked against the same photograph of a California gull. The four naming families ask the same bird at four levels of specificity, and only one of the four gives the model options.

#	Task	Items	Dataset
1	Counting	$53\,498$	all ten subsets
2	Viewpoint	$13\,000$	wildlife datasets
3	Absolute distance	$14\,108$	beetles
4	Relative distance	$3976$	beetles
5	Relative counting	$42\,412$	all ten subsets
6	Localization	$3988$	wildlife datasets

Appendix

Table 9: C3 spatial tasks with the dataset each runs on. A task runs only where an existing annotation provides the ground truth, which is why five of them run on beetles.

Task	Question	Answer form	Answer
Counting	How many beetles are there in this image?	integer	11
Relative counting	Which of the 4 images contains the most number of beetles?	one of four panels	Image B
Absolute distance	What is the measurement of the elytra width, marked with the red line, with respect to the scalebar, marked with the blue line, in cm?	number in cm	0.497
Relative distance	Which beetle has the largest elytral length?	one of four specimens	Beetle D
Localization	Provide the bounding box coordinate of the zebra in this image.	four integers	[854, 819, 3009, 2322]
Relative position	Where is Beetle (blue) located with respect to Beetle (red)?	one of eight directions	Top-right

Appendix

Table 10: One example per C3 task. Five of the eight take an open-ended answer, so a model cannot recover the answer form from a list of options. The answer form column is what the parser expects, and Appendix K gives the full template for each.

dataset	Species	Identity signal
WhaleSharkID	whale shark	spot pattern
BelugaID	beluga whale	scarring, dorsal ridge
HyenaID	spotted hyena	spot pattern
LeopardID	leopard	rosette pattern
GZCD	Grevy’s zebra	stripe pattern

Appendix

Table 11: The five C4 datasets and the marking that identifies an individual in each. Four carry a pattern that is unique to the animal and stable across sightings. Beluga is the exception.

Dataset	Used by	Content
TreeOfLife-200M	C1	100K subset, ten groups
CUB-200-2011	C2	$11\,788$ annotated birds
WAID	C3	aerial wildlife
Boxes on Bees	C3	dense hive frames
2018-NEON-beetles	C3	trays with scalebars
DePondFi	C3	pond fish

Appendix

Table 12: The eleven public datasets behind Bio-VQA . Four of them serve two components.

Model	Checkpoint	Params	LLM backbone	Vision path
Qwen2.5-VL-3B	Qwen/Qwen2.5-VL-3B-Instruct	3.75B	Qwen2.5-3B	Native-resolution ViT (0.67B), window attention
Qwen2.5-VL-7B	Qwen/Qwen2.5-VL-7B-Instruct	8.29B	Qwen2.5-7B	Native-resolution ViT (0.67B), window attention
Qwen3-VL-8B	Qwen/Qwen3-VL-8B-Instruct	8.8B	Qwen3-8B	Qwen3-VL ViT, DeepStack fusion
Qwen3-VL-30B-A3B	Qwen/Qwen3-VL-30B-A3B-Instruct	30.5B / 3B active	Qwen3-30B-A3B	Qwen3-VL ViT, DeepStack fusion
Qwen3-VL-32B	Qwen/Qwen3-VL-32B-Instruct	32B	Qwen3-32B	Qwen3-VL ViT, DeepStack fusion
InternVL3-8B	OpenGVLab/InternVL3-8B-Instruct	8B	Qwen2.5-7B	InternViT-300M-448px-V2.5

Appendix

Table 13: The 18 open-weight models, from 3.75B to 38.4B parameters. Two are mixture-of-expert variants that activate 3B parameters per token.

Component and task	Answer form	Parse	Mismatch	Parsing Criteria
C1, eight rank types	open text	$1.000$	—	We match answers with ground truth after normalization ( The phylum is Chordata. normalizes to chordata )
C1, common name MCQ	bare options	$1.000$	—	We match the answer against the four option strings
C2, six MCQ subtasks	lettered options	$.960$	$.0117$	A mismatch in letter and text. Original options: (A) Red , (B) Blue . Answer: (B) Red .
C2, four open subtasks	open text	$1.000$	—	We accept any string and score it after normalization
C3, counting	open integer	$.987$	—	We reject an answer that holds no integer, such as There are many beetles
C3, viewpoint	open text	$1.000$	—	10 candidates given in prompt. We reject an answer that holds none.

Appendix

Table 14: Parse quality: We report the model with lowest parsing score in each component and task. Mismatch needs a letter and an option text that can disagree, so only C2 reports it. Parsing Criteria column shows what the parser accepts or rejects for that answer form. † In C4,

269\,903

answers are Yes or No . The other

97

contains Yes or No but in a sentence ( e.g . Yes, they are the same ).

Space	Candidates	Size	Role
kingdom	species in the named kingdom	$10^{4}$ – $10^{5}$	fair
tol	all of TreeOfLife-200M	${\sim}9{\times}10^{5}$	ablation
group	the group’s gold species	$10^{2}$	ceiling

Appendix

Table 15: The three candidate sets for BioCLIP-2. We report tol in the main paper, which gives BioCLIP-2 no kingdom cue; kingdom differs from it by

+0.060

at phylum and by at most

0.005

at every finer rank.

Truth $y$	Answer $\hat{y}$	$\mathcal{S}(\hat{y})$	$\mathrm{EM}_{\mathrm{soft}}$
Squamata	Reptilia	the four reptile classes	1
Squamata	Crocodylia	Crocodylia	0
Puma concolor	Felis concolor	Puma concolor	1

Appendix

Table 20

Answer	What it is	$\mathrm{EM}$	$\mathrm{EM}_{\mathrm{soft}}$	sev	wup	lin
D. cinereus	the right species	1	1	0	$1.00$	$1.00$
Hyla cinerea	its pre-2016 name	0	1	0	$1.00$	$1.00$
D. gratiosus	barking treefrog	0	0	1	$0.83$	$0.75$
A. americanus	a toad, other family	0	0	3	$0.50$	$0.35$
Mus musculus	a mouse, other class	0	0	5	$0.17$	$0.05$

Appendix

Table 21

Answer	$s$	cags	Reading
blue	$1.000$	$1.000$	correct
purple	$0.498$	$0.350$	better than a guess
green	$0.134$	$-0.122$	worse than a guess
yellow	$0.053$	$-0.227$	much worse than a guess

Appendix

Table 22

Pair	exponential	linear
blue, purple	0.498	0.791
blue, green	0.134	0.396
blue, yellow	0.053	0.120
brown, rufous	0.760	0.918
yellow, orange	0.507	0.796
black, white	0.036	0.000

Appendix

Table 23

Pair	part similarity
nape, crown	0.922
beak, forehead	0.857
left wing, right wing	0.883
crown, belly	0.430
beak, tail	0.000

Appendix

Table 24

Prediction	Ground truth	Score	Reason
song sparrow	fox sparrow	$0.700$	$F_{1}=0.5$ , same head
american crow	fox sparrow	$0.000$	no shared token, other head
M. georgiana	M. melodia	$0.575$	genus agrees, epithet fallback
C. brachyrhynchos	M. melodia	$0.054$	both components fallback

Appendix

Table 25

true	pred	cas	$\mathrm{EM}$
10	10	1.000	1
10	12	0.800	0
10	15	0.500	0
10	20	0.000	0
10	90	0.000	0
100	95	0.950	0

Appendix

Table 26

Control	Strategy it catches	Score under it
per-type constant floor	one value for length, another for width	$\mathrm{acc}@20=0.410$
$\rho_{\text{within}}$	any predictor constant within a type	$\rho_{\text{within}}=0$
compression $\kappa$	one remembered size for both types	$\kappa\approx 1$

Appendix

Table 27

$d^{\prime}$	balanced acc.	reading
0.0	50%	no discrimination
0.5	60%	weak
1.0	69%	moderate
2.0	84%	strong
3.0	93%	near ceiling

Appendix

Table 28

Figure 5: How

d^{\prime}

and criterion

c

separate discrimination from bias. A negative

d^{\prime}

indicates the adversarial condition (lower panel), where the decision variable separates the two classes but points the wrong way.

#	Model	Img	Ques	$\mathrm{EM}$	$\mathrm{EM}_{\mathrm{soft}}$	wup	lin
—	BioCLIP-2 kingdom	100K	890K	.743	.743	.854	.836
—	BioCLIP-2 tol	100K	890K	.737	.737	.836	.824
1	Qwen3-VL-30B-A3B	100K	890K	.503	.518	.721	.702
2	Qwen3-VL-32B	100K	890K	.488	.505	.702	.682
3	Qwen3-VL-8B	100K	890K	.483	.497	.707	.685
4	InternVL3-38B	100K	890K	.455	.470	.680	.661

Appendix

Table 18: C1 pooled leaderboard, sorted by

\mathrm{EM}_{\mathrm{soft}}

. Every configuration answers the same

890\,000

questions over the same

100\,000

images, so the rows are directly comparable. The strongest generalist recovers roughly 68% of the specialist pooled accuracy.

	20 models		18 VLMs
Rank	$\mathrm{EM}$	step	$\mathrm{EM}$	step	gap
phylum	.748	—	.725	—	.023
class	.578	$\downarrow$ .170	.545	$\downarrow$ .180	.033
order	.568	$\downarrow$ .010	.541	$\downarrow$ .004	.027
family	.421	$\downarrow$ .147	.382	$\downarrow$ .159	.039
genus	.292	$\downarrow$ .129	.247	$\downarrow$ .135	.045

Appendix

Table 19: Exact match falls at every step down the hierarchy. The step column gives the change from the row above, in red as the scores fall. The left block pools all 20 configurations, and the right block pools the 18 VLMs. The gap column is the difference between the two, and it widens from

.023

at phylum to

.051

at species, so the specialist contributes most exactly where the generalists collapse.

Figure 6: Every model’s accuracy falls at every step down the tree. The dashed floor line shows that apparent competence at the phylum level largely comes from guessing common categories, not from actually reading the image.

Model	open-ended	4-way MCQ	gap
Qwen3-VL-32B	.078	.809	$+.732$
Qwen3-VL-8B	.065	.775	$+.710$
InternVL3-38B	.044	.752	$+.708$
Qwen2.5-VL-7B	.062	.753	$+.692$
Qwen3-VL-30B-A3B	.107	.772	$+.665$
18-model mean	.039	.645	$+.606$

Appendix

Table 20: We show models the identical image and ask for the identical organism. They recognize the correct name among four options far more often than they produce it unprompted. For the 18 VLMs, the ratio of mean accuracies is

16.4\times

Figure 7: Every model recognizes the right answer from a list vastly better than it produces the name unprompted.

Model	phy	cls	ord	fam	gen	spc	sci	com	mcq	pooled
BioCLIP-2 kingdom	1.00 †	1.11	1.83	2.40	2.78	2.88	2.89	2.29	2.72	2.523
BioCLIP-2 tol	1.00 †	1.61	2.22	2.65	2.94	2.96	2.99	2.37	2.72	2.638
Qwen3-VL-30B-A3B	0.53	0.72	1.60	1.84	2.06	3.47	2.48	2.71	3.04	2.472
Qwen3-VL-32B	0.38	0.77	1.41	1.78	2.09	3.52	2.48	2.89	3.02	2.459
Qwen3-VL-8B	0.61	0.69	1.42	1.80	2.04	3.65	2.46	2.85	3.04	2.480
InternVL3-38B	0.65	0.62	1.49	1.90	2.29	3.71	2.82	3.22	3.10	2.614

Appendix

Table 21: Mistake severity by rank, with the unresolved share beside it (lower is better). Colors mark the best , second best , second worst and worst over the 18 generalists only. Severity is conditional on being wrong and on resolving, so it is not comparable across models with very different accuracy. † BioCLIP-2 carries the worst phylum severity in the table because it is wrong at phylum only

1.5\%

of the time, and that residual is the hard part.

Group	$\mathrm{EM}$	severity	unresolved
Bryophytes	.246	3.396	.154
Fish	.274	2.966	.109
Gymnosperms	.321	2.424	.176
Reptiles	.323	2.463	.167
Angiosperms	.401	2.423	.101
Amphibians	.404	2.493	.122

Appendix

Table 22: Exact match, severity and the unresolved share by organismal group, pooled over all 20 configurations and sorted by exact match. Unresolved is the percentage of answers that name no node in the tree, so they carry no severity. The three columns do not track each other. Gymnosperms carry the lowest severity and the second highest unresolved share, so their few measurable misses are close ones. Bryophytes carry the worst severity and the field’s lowest accuracy together.

Figure 8: Accuracy and severity measure different things. A model can be wrong often and still miss by a close relative.

Rank	$\mathrm{EM}$	$\mathrm{EM}_{\mathrm{soft}}$	gap	Group	$\mathrm{EM}$	$\mathrm{EM}_{\mathrm{soft}}$	gap
class	.578	.658	$\uparrow$ .080	Reptiles	.323	.422	$\uparrow$ .100
phylum	.748	.789	$\uparrow$ .041	Angiosperms	.401	.432	$\uparrow$ .031
scientific name	.165	.172	$\uparrow$ .007	Gymnosperms	.321	.339	$\uparrow$ .018
family	.421	.425	$\uparrow$ .004	Bryophytes	.246	.250	$\uparrow$ .004
genus	.292	.295	$\uparrow$ .003	Fish	.274	.277	$\uparrow$ .003
order	.568	.570	$\uparrow$ .002	Amphibians	.404	.407	$\uparrow$ .002

Appendix

Table 23: Taxonomic revision lag, pooled over all 20 models. The gap is

\mathrm{EM}_{\mathrm{soft}}-\mathrm{EM}

, so it is always a gain and we mark it with a teal arrow. Both panels are sorted by gap. The two panels line up. Reptiles carry the largest group gap and Reptilia dissolved at class rank, which is where the largest rank gap sits. Angiosperms and Gymnosperms carry the next two and their revision moved the phylum from Magnoliophyta to Tracheophyta , which is the second largest rank gap. A dash means the gap rounds to zero.

	Amph	Angio	Arach	Birds	Bryo	Fish	Gymno	Insect	Mamm	Rept	mean
Amphibians	1.00	.94	.82	.90	.92	.88	.39	.89	.95	.98	.852
Angiosperms	.94	1.00	.87	.88	.96	.89	.49	.91	.92	.94	.867
Arachnids	.82	.87	1.00	.90	.87	.81	.41	.94	.83	.83	.809
Birds	.90	.88	.90	1.00	.91	.87	.42	.97	.91	.89	.850
Bryophytes	.92	.96	.87	.91	1.00	.94	.46	.95	.89	.91	.868
Fish	.88	.89	.81	.87	.94	1.00	.37	.91	.83	.88	.820

Appendix

Table 24: Spearman rank correlation between organismal groups. The mean column averages the nine off-diagonal cells in each row. Red marks the Gymnosperms row and column, which hold the lowest value in every row of the table without exception, and Arachnids carries the next lowest mean, but almost

2\times

Gymnosperms. The whole matrix averages

0.807

. If we drop Gymnosperms, it jumps to

0.901

, so one group costs the benchmark almost a tenth of its cross-group agreement.

Figure 9: Model rankings remain nearly identical across nine of the ten organism groups. Gymnosperms are the single exception.

Scientific name	As shipped	Lang.	Resolved
Juniperus communis	en	sv/da	Common juniper
Picea abies	gran	sv	Norway spruce
Pinus sylvestris	tall	sv	Scots pine
Rana arvalis	åkergroda	sv	Moor frog
Rana arvalis	Spidssnudet frø	da	Moor frog
Bufotes viridis	grønfläckig padda	sv	European green toad

Appendix

Table 25: Contaminated ground truth labels and their resolutions.

Format	Axis	Question it answers
Open recall	Prevalence	How much of the visual world does this taxon occupy?
4-way MCQ	Confusability	How close is the nearest wrong option in the tree?

Appendix

Table 26: The difficulty axis used for each question format. Prevalence is unsuitable for a choice item: recognizing a common organism among three close relatives is harder than recognizing a rare organism among three distant relatives.

Cell	Distinct taxa	Outcome
(Birds, phylum)	1, Chordata	escalate
(phylum, all groups)	a handful	degenerate
(Birds, class)	1, Aves	escalate
(class, all groups)	many	tertiles, global

Appendix

Table 27: The escalation protocol for resolving thin cells prevents the fabrication of non-existent contrasts. Because every bird belongs to the phylum Chordata , and the entire benchmark spans only a few phyla, there is genuinely no rare-versus-common phylum contrast to measure within birds.

Distractor	LCA	depth	$d_{j}$	$2^{-d_{j}}$
same family	family	4	2	0.250
same class	class	2	4	0.063
other phylum	root	0	6	0.016

Appendix

Table 28: Worked example for a ground truth node Zootoca vivipara at depth

6

. The nearest distractor sits two edges away, so the item is Medium under the weakest-link NDD rule and scores

0.110

on the continuous axis.

Step	Rule	Example
exact	the answer names a node at the asked rank	Lacertidae as family
vernacular	a common name through the vernacular index	Orb weavers to Araneidae
lineage	the asked rank read off this image’s lineage	vivipara through the image
anchor	lowest resolvable node in the lineage	an order question with a null order

Appendix

Table 29: The four-step resolution protocol. The lineage step provides essential robustness; for example, a question asking for an organism’s class resolves through the organism’s full metadata rather than just the isolated string.

#	Model	$\mathrm{EM}$	GS ${}_{\text{raw}}$	cags	95% CI	sev
1	Qwen3-VL-32B	.570	.687	.593	.592–.594	0.946
2	Qwen3-VL-30B-A3B	.538	.665	.569	.568–.570	0.933
3	InternVL3-38B	.535	.665	.568	.567–.569	0.929
4	Gemma-4-12B	.526	.651	.554	.553–.555	0.941
5	Qwen3-VL-8B	.528	.654	.553	.551–.554	0.947
6	InternVL3.5-38B	.528	.648	.548	.546–.549	0.959

Appendix

Table 30: C2 leaderboard over

824\,104

questions per model. Intervals are 95% bootstrap over

B=1000

resamples of the per-item cags . Adjacent ranks separated by less than

0.001

are statistical ties. Phi-3.5-Vision is the only model with severity above 1 and the only one whose cags falls below its

\mathrm{EM}

Model	Acc	MCQ	Open	parse	mism.	choose A
Qwen3-VL-32B	57.31	57.63	51.96	100.0	0.00	19.15
Qwen3-VL-30B-A3B	54.08	54.82	41.53	100.0	0.03	24.30
InternVL3-38B	53.70	54.42	41.72	100.0	0.02	17.08
Qwen3-VL-8B	52.98	53.92	37.11	100.0	0.04	14.56
InternVL3.5-38B	52.88	54.39	27.57	100.0	0.02	22.85
Gemma-4-12B	52.71	55.06	13.33	98.90	0.01	17.00

Appendix

Table 31: C2

\mathrm{EM}

in percent, alongside three format diagnostics. The choose A metric tracks the frequency of multiple-choice answers equal to option A (against a uniform expectation near 20%). Every model’s MCQ score exceeds its open-recall score, often by tens of percentage points.

Model	Anat. loc.	Morph.	Salience	Naming
Qwen3-VL-32B	.4754	.6441	.5839	.7310
Qwen3-VL-30B-A3B	.4525	.6313	.5582	.6584
InternVL3-38B	.4132	.6451	.5621	.6691
Gemma-4-12B	.4306	.6205	.5915	.3742
Qwen3-VL-8B	.4411	.6244	.5309	.6429
InternVL3.5-38B	.3747	.6379	.5595	.5363

Appendix

Table 32: cags by category. Colors mark the best , second best , second worst and worst model in each column. The category leaders vary across domains: InternVL3-38B leads morphological attributes, Gemma-4-12B leads relative salience, and Qwen3-VL-32B leads anatomical localization and naming. Molmo2-8B is second on anatomical localization at

.4741

, which is

.0013

behind a model four times its size. A single aggregated scalar masks these localized strengths.

#	Model	attr_bare	attr_group	cat_mcq	marker	common	genus	kind_of	sci	vis_set	vis_side
1	Qwen3-VL-32B	.644	.584	.959	.727	.619	.696	.763	.629	.435	.526
2	Qwen3-VL-30B-A3B	.631	.558	.951	.660	.608	.423	.733	.594	.434	.445
3	InternVL3-38B	.645	.562	.948	.603	.495	.636	.665	.614	.373	.484
4	Gemma-4-12B	.621	.592	.772	.720	.286	.121	.615	.105	.385	.488
5	Qwen3-VL-8B	.624	.531	.958	.608	.543	.486	.691	.553	.426	.435
6	InternVL3.5-38B	.638	.560	.920	.656	.389	.417	.518	.454	.310	.498

Appendix

Table 33: Colors mark the best , second best , second worst and worst model in each column. cags by question family for all 18 models. Hardest questions are concentrated in two areas: open-ended taxonomic naming and anatomical set/side reasoning. In contrast, coarse recognition via category_mcq is nearing saturation with a field mean of

.833

#	Model	attr_bare	attr_group	cat_mcq	marker	common	genus	kind_of	sci	vis_set	vis_side
1	Qwen3-VL-32B	64.51	59.01	95.90	61.15	35.49	69.46	51.15	51.80	34.12	55.65
2	Qwen3-VL-30B-A3B	63.16	55.53	95.11	51.92	33.57	40.96	44.17	47.61	32.07	46.34
3	InternVL3-38B	64.44	56.54	94.81	47.25	18.94	63.22	34.14	50.38	23.02	49.94
4	Gemma-4-12B	63.17	60.40	77.20	59.93	12.73	12.53	23.78	4.73	19.30	52.84
5	Qwen3-VL-8B	62.81	53.35	95.77	47.12	25.13	47.83	33.85	41.56	33.40	46.10
6	InternVL3.5-38B	64.46	56.46	92.00	54.01	13.18	40.89	21.68	34.38	21.63	53.08

Appendix

Table 34:

\mathrm{EM}

accuracy in percent by question family, reported on the identical item partition as Tab. 33 . Colors mark the best , second best , second worst and worst model in each column. Coarse recognition peaks at 96.16%, whereas the highest open common-name score reaches only 35.49%. Reading the mean row against the one in Tab. 33 isolates where partial credit does work. The two agree to within half a point on attr_bare , attr_group , cat_mcq , genus and vis_side , and they diverge most on kind_of and common , where near misses are frequent and cags credits them.

Family	mean	min	max	reading
visibility_partset	.341	.108	.435	hardest
open_genus	.353	.023	.696	widest spread
open_scientific_name	.359	.049	.629	broadly hard
open_common_name	.406	.141	.627	free naming
visibility_side	.411	.225	.746	laterality
attr_group_value	.495	.317	.592	moderate

Appendix

Table 35: Mean cags per family across all 18 models. The most tightly compressed family is attr_bare_value (minimum

.454

), underscoring that basic attribute extraction is a foundational competence for modern VLMs. Consequently, the harder families remain the primary discriminators of model capability.

	Taxonomic questions
Model	MCQ	OE	gap
InternVL3.5-30B	.869	.301	.568
Phi-3.5-Vision	.716	.177	.539
InternVL3.5-38B	.920	.389	.531
InternVL3-8B	.884	.374	.510
InternVL3.5-8B	.783	.277	.506

Appendix

Table 36: Recognition–recall gap for taxonomic questions.

Model	cags rank	$\mathrm{EM}$ rank	$\Delta$	Why
Gemma-4-12B	4	6	$+2$	near misses earn
LLaVA-NeXT-Vicuna	15	18	$+3$	misses land close
Qwen2.5-VL-7B	8	9	$+1$
InternVL3.5-8B	11	12	$+1$
Phi-3.5-Vision	16	15	$-1$	confidently wrong
LLaVA-NeXT-Mistral	18	16	$-2$	misses land far

Appendix

Table 37: Instances where cags and

\mathrm{EM}

disagree. The signature shift is LLaVA-NeXT-Vicuna surpassing Phi-3.5-Vision. While Vicuna achieves fewer

\mathrm{EM}

s, its misses consistently land closer to the truth; thus, a closeness-aware, chance-corrected metric appropriately ranks it higher.

Figure 10: Color terms are compared perceptually rather than lexically; thus, predicting purple instead of blue receives significantly more partial credit than predicting yellow instead of blue.

Figure 11: Partial credit elevates every model above the identity line, with one exception. Phi-3.5-Vision falls below the line because its prediction errors land further from the ground truth than a random distractor.

Task	Best model	Score	Floor
Counting	InternVL3-38B	.799	modal count .501
Viewpoint	Qwen3-VL-32B	.358	majority .520, $\kappa$ VAS 0
Absolute distance	Qwen3-VL-32B	.858	per-type constant .410
Relative distance	Gemma-4-12B	.443	chance .250
Relative counting	Qwen3-VL-32B	.403	chance .250
Localization	Qwen3-VL-32B	.866	mean box .415

Appendix

Table 38: Per-task headline. Counting is macro- cas , viewpoint is

\kappa

VAS, absolute distance is within-plate Spearman

\rho

, relative distance and relative counting are balanced accuracy, localization is mean IoU, relative position is accuracy, and OCR is suffix character accuracy.

Contrast	Effect	$\sigma$	Holds for
easy to medium	$+.064$	$.037$	18 of 18
medium to hard	$-.005$	$.027$	6 of 18

Appendix

Table 39: Cutting 50% the area a target occupies costs every model accuracy. Drawing both panels from the same corpus makes no further difference.

Model	$\mathrm{EM}$	rank	$\kappa$ VAS	rank	shift
Molmo2-8B	.228	10	.358	1.5	$+8.5$
Qwen2.5-VL-7B	.050	15	.182	7.5	$+7.5$
Qwen3-VL-32B	.417	7	.358	1.5	$+5.5$
InternVL3.5-38B	.483	2	.208	6	$-4.0$
InternVL3.5-8B	.456	3.5	.117	10	$-6.5$
Phi-3.5-Vision	.456	3.5	.083	11	$-7.5$

Appendix

Table 40: The six largest rank shifts between

\mathrm{EM}

and

\kappa

VAS on viewpoint. Tied models share the average of the positions they span. InternVL3.5-8B and Phi-3.5-Vision both score

\mathrm{EM}=.456

and fill positions 3 and 4, so both read 3.5. Molmo2-8B and Qwen3-VL-32B both score

\kappa\mathrm{VAS}=.358

and fill positions 1 and 2, so both read 1.5. A shift is the difference between two midranks and is fractional for the same reason. Rising models place a large share of answers in the over class, which means they name a compound direction where the annotation records a cardinal one.

\mathrm{EM}

scores every one of those as zero.

GT	Answer	steps	angle	$\mathrm{EM}$
left	left	0	$0^{\circ}$	1
left	front-left	1	$45^{\circ}$	0
left	front	2	$90^{\circ}$	0
left	right	4	$180^{\circ}$	0

Appendix

Table 41: One ground truth (GT) viewpoint under both scorers. The rubric reads the steps column and the angular metric reads the angle column.

\mathrm{EM}

reads neither and gives the last three rows the same zero. Labels that sit off the ring carry no bearing, so angular check covers the ring classes only, which is why we use it as a check and not as the headline.

Model	$\rho_{\text{within}}$	$\mathrm{acc}@20$	calib	diagnosis
InternVL3.5-8B	.703	.263	.509	measures, no calibration
Gemma-4-12B	.753	.385	.512	measures, no calibration
Qwen3-VL-8B	.682	.303	.453	measures, no calibration
Qwen3-VL-32B	.858	.622	.614	measures and calibrates
Qwen2.5-VL-3B	.081	.194	.306	neither
LLaVA-NeXT-Vicuna	$-.023$	.217	.326	neither

Appendix

Table 42: Measurement and calibration come apart. InternVL3.5-8B orders the specimens well at

\rho_{\text{within}}=.703

but sits below the floor on precision at

.263

. Removing its per-type bias lifts it to

.509

, from below the floor to above it.

#	Model	IoU ${}_{\text{all}}$	a.25	a.50	a.75	parse	convention
1	Qwen3-VL-32B	.866	.995	.984	.877	1.000	n1000 xyxy
2	Qwen3-VL-8B	.862	.995	.985	.868	1.000	n1000 xyxy
3	Qwen3-VL-30B-A3B	.861	.995	.984	.867	1.000	n1000 xyxy
4	InternVL3-38B	.744	.941	.832	.675	0.999	n1000 xyxy
5	LLaVA-NeXT-Vicuna	.701	.977	.885	.444	1.000	n01 xyxy
6	Gemma-4-12B	.589	.749	.631	.475	0.999	n1000 xywh

Appendix

Table 43: Localization. The headline column charges unparsed replies as zero. Twelve of eighteen models fall below the mean gold box, which is a rule that draws the same box on every image. The convention column records which of eight box formats the model emitted, detected by scoring under all eight. Molmo2-8B and Phi-3.5-Vision mostly repeat the example box given in the prompt.

#	Model	lift	balAcc	acc	posBias
1	Qwen3-VL-32B	$+.403$	.653	.672	.445
2	Gemma-4-12B	$+.388$	.638	.658	.479
3	InternVL3-38B	$+.366$	.616	.634	.513
4	InternVL3.5-38B	$+.338$	.588	.599	.409
5	Molmo2-8B	$+.329$	.579	.594	.235
6	Qwen3-VL-8B	$+.320$	.570	.587	.239

Appendix

Table 44: Relative counting. Lift is accuracy above the .250 chance floor, normalized. The bottom model sits marginally below chance and answers one option 100% of the time.

#	Model	acc	lift	angErr	within45
1	Qwen3-VL-32B	.550	$+.425$	27.0	.895
2	Gemma-4-12B	.495	$+.370$	35.4	.815
3	Molmo2-8B	.457	$+.332$	32.9	.864
4	Qwen3-VL-8B	.421	$+.296$	40.1	.803
5	Qwen3-VL-30B-A3B	.355	$+.230$	55.0	.673
6	Pixtral-12B	.266	$+.141$	64.4	.576

Appendix

Table 45: Relative position over eight directions on the beetle trays. The majority-direction floor is

.125

. Angular error is the mean error on the 45-degree ring and within45 is the share within one step. Two models sit exactly at the floor and answer one direction every time.

Statistic	Seven spatial	$+$ OCR
Mean inter-task $\rho$	$+0.703$	$+0.619$
Krippendorff $\alpha$	$0.940$	$0.921$
Variance on PC1	$75\%$	$68\%$
Mean $\rho$ against $\log$ params	$\mathbf{+0.588}$	$\mathbf{+0.028}$

Appendix

Table 46: OCR acts as the negative control. It correlates

+0.366

with the seven spatial tasks, against

+0.703

among themselves. Every spatial task individually beats its correlation with the control.

Model	charAcc	suffixCharAcc	inflation
Phi-3.5-Vision	.568	.161	.407
Pixtral-12B	.593	.199	.394
LLaVA-NeXT-Vicuna	.607	.214	.393
InternVL3-38B	.620	.246	.374
Qwen2.5-VL-3B	.969	.940	.029
Qwen3-VL-30B-A3B	.993	.987	.007

Appendix

Table 47: Whole-code character accuracy against suffix character accuracy. The inflation column is what the common prefix boosts a model. Phi-3.5-Vision reads

.161

of the informative characters against a floor of

.349

Statistic	Value
Mean inter-task $\rho$	$+0.703$
Range	$+0.351$ to $+0.909$
Krippendorff $\alpha$	$0.940$
Variance on PC1	$75\%$

Appendix

Table 48: The seven spatial tasks cohere. The tightest pairs are viewpoint with relative counting at

+0.909

and viewpoint with relative position at

+0.887

, which are both spatial-relation judgments.

Figure 12: The seven spatial tasks track parameter count and the control does not, on the same imagery with the same models. The coherence is spatial rather than a general capability gradient.

Figure 13: Several tasks sit below what a rule that never looks at the image achieves. A benchmark reporting only accuracy would record these as partial competence.

Model	easy	random	hard	verdict
Qwen3-VL-32B	1.455	0.696	$-0.604$	weak
InternVL3-8B	2.062	0.684	$-0.738$	weak
Qwen2.5-VL-3B	2.290	0.655	$-0.764$	weak
InternVL3.5-30B	1.563	0.568	$-0.529$	weak
Phi-3.5-Vision	1.919	0.562	$-0.932$	weak
Qwen3-VL-30B-A3B	1.281	0.518	$-0.248$	weak

Appendix

Table 49: Discriminability

d^{\prime}

pooled across the five species. The highest random-condition

d^{\prime}

in the field is

0.696

, which is approximately 64% balanced accuracy. No model reaches

d^{\prime}=1.0

. Seventeen of eighteen models exhibit negative

d^{\prime}

under adversarial pairing.

Model	$c$ easy	$c$ random	$c$ hard
LLaVA-NeXT-Vicuna	$-3.09$	$-3.09$	$-3.09$
LLaVA-NeXT-Mistral	$-2.41$	$-2.46$	$-2.54$
Gemma-3-12B	$-1.60$	$-1.99$	$-1.99$
Llama-3.2-Vision	$-1.13$	$-1.59$	$-1.59$
Qwen2.5-VL-7B	$-1.18$	$-1.34$	$-1.51$
Qwen2.5-VL-3B	$-0.88$	$-1.32$	$-1.53$

Appendix

Table 50: Criterion

c

, pooled. Negative values indicate a standing preference for Yes . 17 of the 18 models lean toward Yes , and this bias deepens from easy to hard in nearly every case. Six models exceed

|c|=1.0

in the neutral condition, for which we report

d^{\prime}

and balanced accuracy rather than raw accuracy.

Model	easy $-$ hard	95% CI
InternVL3-8B	$+38.8$	$[+37.0,+40.6]$
InternVL3.5-30B	$+38.4$	$[+36.6,+40.1]$
InternVL3-38B	$+38.4$	$[+36.6,+40.2]$
Phi-3.5-Vision	$+37.0$	$[+35.2,+38.8]$
Qwen3-VL-32B	$+36.9$	$[+35.1,+38.7]$
Molmo2-8B	$+36.0$	$[+34.2,+37.7]$

Appendix

Table 51: Accuracy difference in percentage points between the easy and hard conditions, pooled, with two-proportion difference intervals. Seventeen models confirm the effect, one remains flat, and none reverse.

Species	easy	random	hard	easy $-$ hard
Whale shark	2.812	0.276	$-0.906$	3.718
Spotted hyena	1.743	0.672	$-0.126$	1.869
Grevy’s zebra	1.346	0.391	$-0.596$	1.942
Leopard	1.280	0.517	$-0.188$	1.468
Beluga whale	0.508	0.383	$-0.758$	1.267

Appendix

Table 52: Mean

d^{\prime}

across models, partitioned by species. Beluga is diagnostically useful, as its easy-condition separation is notably the weakest.

Condition	AUC	$d^{\prime}$	accuracy
easy	0.927	2.055	0.902
random	0.698	0.733	0.683
hard	0.247	$-0.969$	0.317

Appendix

Table 53: MegaDescriptor evaluation on Grevy’s zebra. The threshold is fit on the random condition and applied unchanged to both easy and hard conditions, making the easy and hard rows out-of-fit evaluations.

Species	easy AUC	random AUC	hard AUC
Leopard	1.000	0.835	0.114
Whale shark	1.000	0.718	0.053
Spotted hyena	1.000	0.784	0.095
Grevy’s zebra	1.000	0.658	0.068
Beluga whale	0.990	0.386	0.010

Appendix

Table 54: The DINOv2 similarity itself, scored on the pairs it defines. An easy-condition AUC of

1.000

confirms the construction operates exactly as intended. We report this to illustrate the empirical upper bound of the circularity concern.

Model	$d^{\prime}$ random	$d^{\prime}$ hard	$\|d^{\prime}\|$
InternVL3-8B	0.684	$-0.738$	0.68 to 0.74
Qwen3-VL-32B	0.696	$-0.604$	0.70 to 0.60
Qwen2.5-VL-3B	0.655	$-0.764$	0.66 to 0.76
Molmo2-8B	0.491	$-0.411$	0.49 to 0.41

Appendix

Table 55: The magnitude of the sensitivity survives the manipulation; only the sign changes. This rules out the hypothesis that the model lacks signal and is merely guessing, which would drive

d^{\prime}

toward 0 rather than to

-0.7

Figure 14: Difficulty is set by DINOv2 embedding similarity. In the hard condition appearance and identity disagree, so a model reading resemblance rather than identity scores below chance.

Figure 15: Seventeen of eighteen models and the specialist all fall below chance under adversarial pairing, while preserving the magnitude of their sensitivity.

Config	$\rho$	$\tau$	disp.	top-3	mean $\|\Delta\|$
default	1.0000	1.0000	0	✓	0.0000
$\tau=20$	0.9979	0.9869	1	✓	0.0008
$\tau=40$	0.9979	0.9869	1	✓	0.0008
$\tau=50$	0.9979	0.9869	1	✓	0.0015
$\alpha=0.3$	1.0000	1.0000	0	✓	0.0011
$\alpha=0.4$	1.0000	1.0000	0	✓	0.0006

Appendix

Table 56: The cags sensitivity sweep across eleven configurations. The base grid varies the color-kernel temperature

\tau

, the region-value weight

\alpha

, and the color kernel shape. The policy grid adds the letter against text resolution preference and one substantive modeling toggle.

Model	default	anat in vis	$\Delta$
Gemma-3-12B	12	9	$+3$
Llama-3.2-Vision	17	15	$+2$
Qwen2.5-VL-7B	8	10	$-2$
InternVL3.5-8B	11	13	$-2$
LLaVA-NeXT-Vicuna	15	18	$-3$
Top three	1, 2, 3	1, 2, 3	$0$

Appendix

Table 57: Models movement under one substantive toggle. Top three set is unchanged and the movement is confined to families where crediting near-miss parts is a defensible design choice.

Statistic	Value	$n$
Krippendorff $\alpha$ , ordinal	0.7937	904
Mean pairwise quadratic $\kappa$	0.7914
$\kappa$ , annotator 1 against 2	0.7579	292
$\kappa$ , annotator 1 against 3	0.8312	306
$\kappa$ , annotator 2 against 3	0.7851	292
Abstention, any annotator	0.0458	306

Appendix

Table 58: Inter-annotator reliability. Every item receives at least one rating.

Statistic	All, $n=306$	Complete, $n=292$
$\tau_{b}$ , cags against human	0.5883	0.5879
95% interval	$[0.515,0.657]$	$[0.514,0.659]$
$\tau_{b}$ , $\mathrm{EM}$ against human	0.7178	0.7245
Difference, cags minus $\mathrm{EM}$	$-0.1295$	$-0.1366$
$\rho$ , cags against human	0.7328	0.7330
$\rho$ , $\mathrm{EM}$ against human	0.8044	0.8102

Appendix

Table 59: Agreement between the metric and the annotators. The ceiling ratio is the metric against human

\kappa

divided by the mean pairwise annotator

\kappa

. cags reaches

0.908

of the rate at which the annotators agree with each other.

Statistic	Value	Counts
Items with $\mathrm{EM}=0$	204	102 low, 102 mid
AUC, credited against rejected	0.6755	$[0.591,0.757]$
AUC, partial against rejected	0.6823	strict variant

Appendix

Table 60: Within the 204 items

\mathrm{EM}

scores zero, cags separates the answers annotators credited from the ones they rejected.

\mathrm{EM}

cannot, because it assigns all 204 the same score.

Band	$n$	mean cags	mean $h$
low	102	0.0357	0.2484
mid	102	0.4647	0.3595
high	102	1.0000	0.9861

Appendix

Table 61: The human label tracks the band monotonically. The mid band is where the two diverge most, which is where partial credit is being tested.

Family	$n$	$\tau_{b}$ cags	mean cags	mean $h$
attr_bare_value	102	0.6692	0.4999	0.5596
attr_group_value	102	0.5544	0.5130	0.5441
visibility_side	41	0.5638	0.5415	0.6301
visibility_partset	40	0.8063	0.3611	0.4542
marker_localization	21	0.4431	0.6231	0.2857

Appendix

Table 62: Agreement by family. marker_localization is the weakest agreement and the largest gap between metric and human, with mean cags at

0.6231

against mean human at

0.2857

Reference	Response	cags	$h$
(C) left wing	(D) right wing	0.5884	0.0
(A) left wing	(C) right wing	0.5693	0.0
(D) left wing	(C) right wing	0.5594	0.0

Appendix

Table 63: The three items where cags and the annotators disagree most. The anatomy kernel places the left and right members of a paired part close together, so it over-credits exactly the confusion that matters. We state this as a limitation in the main paper and we compute the laterality analysis on

\mathrm{EM}

for that reason.

Dropped	$\tau_{b}$ cags	AUC	$\kappa$ metric-human
none	0.5883	0.6755	0.7187
annotator 1	0.5306	0.6071	0.6464
annotator 2	0.6294	0.7063	0.7189
annotator 3	0.6039	0.6805	0.6782

Appendix

Table 64: Leave-one-annotator-out. The conclusion holds in every subset. The unique-dissent rates are

0.103

0.182

and

0.055

respectively.

Figure 16: The annotators see the image, the question, the reference and the response. They do not see the model identity, the metric value or the stratum.

Figure 17: Inside the

\mathrm{EM}

-zero region, cags separates credited from rejected answers. The three circled points are the laterality items, which are the metric’s one systematic disagreement with the annotators.

Component	Internal $\rho$	Krippendorff $\alpha$
C1	.841	.981
C3	.662	.925
C2	.645	.901
C4	.617	.889

Appendix

Table 65: Within-component agreement. Tasks inside a component agree enough for the component score to summarize them, and differ enough to be worth reporting on their own.

Spearman $\rho$	C1	C2	C3	C4
C1	—	.689	.787	$.433^{\dagger}$
C2		—	.927	$.514^{\dagger}$
C3			—	.610
C4				—

Appendix

Table 66: Component-level correlation over the 18 models. † not significant after Holm correction. The mean across components is

.660

against

.691

within them.

Statistic	Value	Note
Median model rank movement	5.5 of 18	across components
Largest movement	11 of 18	Molmo2-8B
Molmo2-8B C1 percentile	3rd
Molmo2-8B C2 percentile	79th
Phi-3.5 C1 percentile	8th
Phi-3.5 C4 percentile	64th

Appendix

Table 67: Rank movement between components. Three models appear in every component’s top five. Below them the ranking reorders substantially, so anyone choosing a mid-field model cannot do it from one number.

Component	Lowest-loading task	Loading
C4	all five species	.091 to .185
C2	laterality	.154
C3	relative position	.168
C3	localization	.176
C1	gymnosperms	.183

Appendix

Table 68: The tasks that load lowest on the shared factor. Each is a task we designed to isolate something the rest of the benchmark does not measure, and each sits where we designed it to sit.

Figure 18: The effective structure is three blocks rather than four. C2 and C3 merge at the reliability we can measure. C4 separates.

Figure 19: All tasks load positively on the first component, so a single ability drives most of the variance. The tasks with the lowest loadings are exactly the ones we designed to dissociate.

Figure 20: Three models stay in the top five everywhere. The median model moves 5.5 of 18 places between components, so one leaderboard answers which model is best and not which model is best for a given task.

Family	Template
attr_bare_value	What is the {part} {property} of this bird? Options: (A) …
attr_group_value	Which of the following is the most prominently visible physical trait of the bird in this image? Options: …
marker_localization	What is the body part of this bird marked with the {color} {shape}? Options: …
visibility_partset	Which of the following parts of the bird are visible in this image? Options: (A) {x}, (B) {y}, (C) {z}, (D) None, (E) All three
visibility_side	Which of the {eye—leg—wing}s of the bird are visible in this image? Options: (A) Left, (B) Right, (C) Both, (D) None
category_mcq	What type of bird is this in the image? Options: …

Appendix

Table 69: The ten C2 question templates. The marker family specifies the marker using both color and shape, as neither cue alone is sufficient for identification.

Task	Question
Counting	How many {animals} are there in this image? (two-panel tiers: … in the left/right photograph?)
Relative counting	Which of these 4 images contains {most—least} number of {animals}: Image A (top-left), Image B (top-right), Image C (bottom-left), Image D (bottom-right)?
Absolute distance	What is the measurement of the elytra {length—width} (marked with red line with white border) with respect to the scalebar (marked with blue line with green border) in cm?
Relative distance	Among the four highlighted beetle specimens, which one has the largest elytral {length—width}? Beetle A (red dotted box/line), Beetle B (blue dotted box/line), Beetle C (green dotted box/line), and Beetle D (white dotted box/line)
Localization	Provide bounding box coordinates for the subject animal ({species}) in this image.
Relative position	This image is marked with six boxes: two red, two blue, one green and one cyan. Exactly one red box and exactly one blue box contain a specimen; the other red box and the other blue box are empty. The green and cyan boxes are irrelevant to the question. Where is the specimen inside the red box located with respect to the specimen inside the blue box? Options: {four of the eight directions}

Appendix

Table 70: The eight C3 questions. Each names the target, so a non-localization question does not require the model to find the subject first.

Stage	Det.	Mechanism
Image download	✗	Thread interleaving dictates which concurrent downloads fulfill the allocated budget. The finalized manifests serve as the artifact of record.
QA generation	✓	Deterministically seeded per component.
Manifest sharding	✓	Seed 42, applied subsequent to row identifier assignment.
Inference	✓	Greedy decoding, evaluated zero-shot with one attempt per item.
Difficulty tiers	✓	Strictly deterministic. Tie-breaking relies on taxon nomenclature.
Scoring	✓	A pure function applied to the merged prediction files. Bootstraps are fixed-seed.

Appendix

Table 71: Determinism across the evaluation pipeline. Given a finalized manifest, all downstream operations are exactly reproducible. Consequently, we archive and distribute the manifests rather than expecting researchers to regenerate identical image sets byte-for-byte.

Model	phy	cls	ord	fam	gen	spc	sci	com	mcq
BioCLIP-2 kingdom	.985	.871	.816	.772	.691	.547	.546	.502	.966
BioCLIP-2 tol	.925	.874	.821	.775	.691	.545	.544	.500	.966
Qwen3-VL-30B-A3B	.864	.731	.757	.612	.417	.072	.221	.107	.772
Qwen3-VL-32B	.831	.615	.712	.612	.444	.066	.244	.078	.809
Qwen3-VL-8B	.887	.680	.720	.589	.406	.046	.202	.065	.775
InternVL3-38B	.911	.683	.638	.484	.355	.054	.189	.044	.752

Appendix

Table 72:

\mathrm{EM}

by rank for every configuration. Tab. 19 gives the column means. Every model falls monotonically from phylum to the species epithet, and the common-name MCQ column sits far above the open common-name column for all 20.

Model	phy	cls	ord	fam	gen	spc	sci	com	mcq
BioCLIP-2 kingdom	.985	.871	.816	.772	.692	.547	.547	.502	.966
BioCLIP-2 tol	.925	.874	.821	.775	.692	.545	.545	.500	.966
Qwen3-VL-30B-A3B	.882	.829	.758	.615	.420	.072	.233	.107	.772
Qwen3-VL-32B	.861	.713	.718	.615	.448	.067	.258	.078	.809
Qwen3-VL-8B	.895	.776	.721	.592	.409	.047	.213	.065	.775
InternVL3-38B	.932	.779	.640	.488	.359	.054	.199	.044	.752

Appendix

Table 73:

\mathrm{EM}_{\mathrm{soft}}

by rank for every configuration, on the same items as Tab. 72 . The two specialist rows barely move, because a specialist emits current names.

Model	phy	cls	ord	fam	gen	spc	sci	com	mcq
BioCLIP-2 kingdom	.000	.000	.000	.000	.001	.000	.001	.000	.000
BioCLIP-2 tol	.000	.000	.000	.000	.001	.000	.001	.000	.000
Qwen3-VL-30B-A3B	.018	.098	.001	.003	.003	.000	.012	.000	.000
Qwen3-VL-32B	.030	.098	.006	.003	.004	.001	.014	.000	.000
Qwen3-VL-8B	.008	.096	.001	.003	.003	.001	.011	.000	.000
InternVL3-38B	.021	.096	.002	.004	.004	.000	.010	.000	.000

Appendix

Table 74:

\mathrm{EM}_{\mathrm{soft}}-\mathrm{EM}

by rank, the difference between the two tables above. The gain concentrates at class and phylum for nearly every model and is zero from the species epithet rightward. Tab. 23 pools this by rank and by group.

Model	Amph	Angio	Arach	Birds	Bryo	Fish	Gymno	Insect	Mamm	Rept
BioCLIP-2 kingdom	.766	.875	.781	.887	.659	.719	.215	.921	.819	.795
BioCLIP-2 tol	.765	.864	.777	.886	.653	.718	.182	.921	.817	.793
Qwen3-VL-30B-A3B	.495	.561	.497	.691	.322	.345	.482	.605	.614	.406
Qwen3-VL-32B	.485	.536	.525	.653	.328	.334	.388	.645	.567	.407
Qwen3-VL-8B	.460	.595	.478	.645	.285	.327	.469	.601	.574	.385
InternVL3-38B	.457	.480	.444	.640	.265	.297	.483	.573	.552	.346

Appendix

Table 75:

\mathrm{EM}

by organismal group for every configuration. Birds and Insects are the easiest groups for almost every model, Bryophytes the hardest. Gymnosperms is the one column where the specialist loses to most generalists, which is the same dissociation Tab. 24 reports.

Model	Easy	Medium	Hard	Degen.
BioCLIP-2 kingdom	.800	.698	.616	.985
BioCLIP-2 tol	.798	.699	.619	.925
Qwen3-VL-30B-A3B	.606	.429	.295	.864
Qwen3-VL-32B	.613	.428	.245	.831
Qwen3-VL-8B	.592	.412	.246	.887
InternVL3-38B	.562	.375	.207	.911

Appendix

Table 76:

\mathrm{EM}

by difficulty tier for every configuration. All 20 satisfy Easy

>

Medium

>

Hard without exception. The Degenerate column holds items in cells too thin to tier, described in Appendix E .

Rank	Easy	Medium	Hard
class	.796	.691	.233
order	.786	.605	.299
family	.580	.387	.270
genus	.444	.265	.165
species epithet	.101	.083	.074
scientific name	.236	.148	.116

Appendix

Table 77: Mean

\mathrm{EM}

by rank and difficulty tier, pooled over all 20 configurations. The tier gap is widest at class and order and narrowest at the species epithet, where every tier is already near the floor. Phylum carries no tier, because a single group holds too few phyla to cut tertiles.

Model	open-ended	4-way MCQ	gap
Qwen3-VL-32B	.078	.809	$+$ .731
Qwen3-VL-8B	.065	.775	$+$ .710
InternVL3-38B	.044	.752	$+$ .708
Qwen2.5-VL-7B	.062	.753	$+$ .691
InternVL3.5-38B	.025	.707	$+$ .682
Qwen3-VL-30B-A3B	.107	.772	$+$ .665

Appendix

Table 78: Open-ended against four-way multiple choice on the common name, all 20 configurations, sorted by gap. Every configuration recognizes far better than it recalls, including both specialists. Tab. 20 gives the five widest and the two means.

Model	unresolved	severity	$\mathrm{EM}$
LLaVA-NeXT-Mistral	.233	2.739	.203
LLaVA-NeXT-Vicuna	.220	3.135	.202
Llama-3.2-Vision	.217	2.493	.338
Phi-3.5-Vision	.179	2.992	.235
Qwen2.5-VL-3B	.177	2.221	.350
InternVL3-8B	.160	2.692	.355

Appendix

Table 79: Unresolved share, severity and

\mathrm{EM}

for every configuration, sorted by unresolved. An unresolved answer names no node in the tree, so it carries no severity. The share correlates with

\mathrm{EM}

\rho=-0.854

Model	$\mathrm{EM}$	sev	$\mathrm{EM}$ rk	sev rk	shift
BioCLIP-2 kingdom	.743	2.523	1	7	$-6$
BioCLIP-2 tol	.737	2.638	2	11	$-9$
Qwen3-VL-30B-A3B	.503	2.472	3	4	$-1$
Qwen3-VL-32B	.488	2.459	4	3	$+1$
Qwen3-VL-8B	.483	2.480	5	5	$+0$
InternVL3-38B	.455	2.614	6	10	$-4$

Appendix

Table 80:

\mathrm{EM}

rank against severity rank for every configuration. A positive shift means the model ranks better on severity than on accuracy. Qwen2.5-VL-3B moves 11 places and is the clearest case of a model that is inaccurate and still misses closely.

Model	Family	Params	$\mathrm{EM}$	sev
Qwen2.5-VL-3B	Qwen2.5-VL	3.0	.350	2.221
Phi-3.5-Vision	Phi-3.5	4.2	.235	2.992
Qwen2.5-VL-7B	Qwen2.5-VL	7.0	.445	2.444
LLaVA-NeXT-Mistral	LLaVA-1.6	7.0	.203	2.739
LLaVA-NeXT-Vicuna	LLaVA-1.6	7.0	.202	3.135
InternVL3.5-8B	InternVL3.5	8.0	.325	2.740

Appendix

Table 81: Parameters against pooled

\mathrm{EM}

and severity for the 18 generalists, sorted by size. Within Qwen3-VL, 8B scores

.483

and 32B scores

.488

. Within InternVL3, 8B scores

.355

and 38B scores

.455

. Scale helps one family and not the other, and it never closes the gap to the specialist.

Metric	$\rho$ with $\mathrm{EM}$	reading
Wu–Palmer	$+0.959$	near-redundant with $\mathrm{EM}$
Lin	$+0.965$	near-redundant with $\mathrm{EM}$
severity	$-0.662$	orders the field differently
unresolved	$-0.854$	tracks $\mathrm{EM}$ , reported beside it

Appendix

Table 82: Spearman correlation of each C1 metric with

\mathrm{EM}

over the 20 configurations. Wu–Palmer and Lin reorder almost nothing, so we keep them as corroboration. Severity is the one metric that carries information

\mathrm{EM}

does not.

Model	attr_bare	attr_group	cat_mcq	marker	common	genus	kind_of	sci	vis_set	vis_side
Qwen3-VL-32B	1.003	1.015	1.000	.968	.503	.995	.374	.769	.857	1.069
Qwen3-VL-30B-A3B	1.001	.993	1.000	.919	.522	.977	.398	.775	.833	1.034
InternVL3-38B	.998	1.008	1.000	.983	.592	.990	.442	.777	.814	1.031
Gemma-4-12B	1.031	1.032	1.000	1.078	.790	1.005	.470	.939	.763	1.087
Qwen3-VL-8B	1.010	1.006	1.000	.940	.555	.985	.411	.765	.861	1.049
InternVL3.5-38B	1.019	1.012	1.000	.951	.695	.987	.593	.832	.881	1.071

Appendix

Table 83: Mistake severity by question family, the mean

1-\textsc{cags}{}

over wrong items only. Lower is better. A value above 1 means the average miss lands farther from the gold than a random distractor. Tab. 30 gives the pooled column. Colors mark the best , second best , second worst and worst in each family.

Model	attr_bare	attr_group	cat_mcq	marker	common	genus	kind_of	sci	vis_set	vis_side
Qwen3-VL-32B	.725	.721	.959	.787	.624	.699	.766	.655	.480	.613
Qwen3-VL-30B-A3B	.714	.697	.951	.755	.613	.426	.737	.622	.479	.554
InternVL3-38B	.724	.704	.948	.721	.501	.638	.670	.641	.423	.572
Gemma-4-12B	.710	.728	.772	.756	.294	.131	.621	.168	.432	.575
Qwen3-VL-8B	.706	.681	.958	.732	.549	.489	.695	.584	.475	.546
InternVL3.5-38B	.724	.704	.920	.754	.396	.422	.524	.492	.366	.584

Appendix

Table 84: Raw graded similarity

\mathrm{GS}_{\text{raw}}

by question family, before the chance adjustment. Reading this against Tab. 33 shows what the adjustment removes. The two are close on the open families, where distractors are sampled and

b

is small, and far apart on visibility_partset and attr_group_value , where the option sets sit close to the gold and a raw score flatters every model.

Model	easy	random	hard
Qwen3-VL-32B	$-$ .270	$-$ .076	$-$ .382
InternVL3-8B	$-$ .652	$-$ .765	$-$ 1.024
Qwen2.5-VL-3B	$-$ .843	$-$ 1.264	$-$ 1.440
InternVL3.5-30B-A3B	$-$ .003	$-$ .066	$-$ .263
Phi-3.5-Vision	$-$ .740	$-$ .885	$-$ 1.133
Qwen3-VL-30B-A3B	$-$ .010	.118	$-$ .052

Appendix

Table 85: Criterion

c

for every model in all three conditions, pooled over the five species by summing the confusion counts. Negative means a standing preference for yes . Colors mark the smallest and largest

|c|

, so best is the least biased. Every model leans toward yes , and 17 of 18 lean further as the pairs harden.

Model	Whale shark	Hyena	Leopard	Grevy’s zebra	Beluga
Qwen3-VL-32B	.496	.814	.517	.931	1.027
InternVL3-8B	.611	1.097	.981	.798	.882
Qwen2.5-VL-3B	.287	1.048	1.243	.203	.470
InternVL3.5-30B-A3B	.412	.968	.443	.506	.700
Phi-3.5-Vision	.364	.880	.896	.344	.709
Qwen3-VL-30B-A3B	.332	.861	.593	1.088	.061

Appendix

Table 86:

d^{\prime}

by model and species under random pairing, the neutral condition. Tab. 52 gives the column means. No model reaches

d^{\prime}=1.5

on any species, and beluga is the hardest column for almost every model.

Model	acc	bal. acc	MCC	$\kappa$
Qwen3-VL-32B	.636	.636	.284	.272
InternVL3-8B	.601	.601	.256	.202
Qwen2.5-VL-3B	.559	.559	.176	.119
InternVL3.5-30B-A3B	.612	.612	.231	.223
Phi-3.5-Vision	.576	.576	.198	.151
Qwen3-VL-30B-A3B	.602	.602	.216	.203

Appendix

Table 87: Accuracy, balanced accuracy, Matthews correlation and Cohen’s

\kappa

under random pairing, averaged over the five species. We report

d^{\prime}

as the headline because these four all conflate discrimination with response bias, and Tab. 85 shows the bias is large. The four agree with each other and with

d^{\prime}

on the broad ordering.

Model	easy	hard	easy $-$ hard
InternVL3-8B	80.1	41.3	38.8
InternVL3-38B	77.7	39.3	38.4
InternVL3.5-30B-A3B	78.3	39.9	38.4
Phi-3.5-Vision	77.1	40.1	37.0
Qwen3-VL-32B	75.9	38.9	36.9
Molmo2-8B	80.2	44.2	36.0

Appendix

Table 88: The difficulty manipulation for every model, in percentage points of accuracy averaged over the five species. Seventeen models confirm the ordering, one is flat, and none reverses. The flat model emits a constant answer and cannot respond to any manipulation, which is what makes it a useful control.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 1,017 papers

Sep 10, 2026

Can Edge-Deployable Vision-Language Models Identify Species?

Camera traps often run in the field on edge hardware with limited or no connectivity, making small, locally-deployable vision-language models (VLMs) -- not frontier-scale ones -- the practically relevant class to evaluate for species identification. We test whether models in this deployment-relevant 2--8B range carry genuine taxonomic knowledge, evaluating four such VLMs (Qwen3-VL 2B/4B/8B, Gemma3 4B) against the domain-specific specialist BioCLIP (300M parameters) on a 96-species task, comparing clean iNaturalist photographs against camera-trap imagery from 6 LILA.science collections, on two independently-sampled evaluation sets. All models identify species far above chance, but every model -- general-purpose or specialist -- degrades sharply on field imagery (domain gaps of 9.6--26.6 percentage points, consistent across taxonomic levels and both evaluation sets), indicating the degradation reflects general image legibility rather than fine-grained discrimination failure. BioCLIP substantially outperforms every VLM tested (by 33.2--59.2 percentage points across an expanded 200-image sample for every model) despite its far smaller size, suggesting the gap reflects specialized training data rather than model scale; yet BioCLIP's own domain gap (18.0 points) is statistically indistinguishable from the best VLM's (22.3 points), suggesting the clean-to-field degradation itself is a property of the image-quality shift rather than a general-purpose-model weakness. Under open-set prompting, 5.9--9.6% of responses are syntactically valid but taxonomically nonexistent species names; the relative fabrication-rate ranking across models replicates exactly across both evaluation sets, a more robust finding than any single point estimate.

William Zhou, Mayukha Siripuram, Xiao Yan +2

Plano West Senior High School

VLM Evaluation VLM Robustness Vision-Language Models Image Classification

Ask PDF

Jun 21, 2026

Benchmarking Vision-Language Models for Microscopic Plant Image Understanding

Microscopic imaging provides essential visual evidence for studying plant biology and pathology at the cellular and subcellular levels. However, existing benchmarks on vision-language models primarily focus on macroscopic plant imagery, while the microscopic domain remains underexplored. To address this gap, we present PlantMicro, a comprehensive benchmark for evaluating vision-language models (VLMs) in microscopic plant imagery. PlantMicro integrates more than 5,000 images collected across diverse hosts, biological domains, and imaging modalities. Building on this diversity, we design a set of complementary tasks that capture different facets of microscopic image understanding. To support these tasks, we construct over 9,000 VQA pairs that systematically evaluate the capabilities of VLMs. Experiments on PlantMicro show that current VLMs struggle with fine-grained recognition and biologically grounded reasoning. For example, GPT-5 achieves 34.93% accuracy on the pathogen classification task, which is only modestly above the random-guessing baseline. The results highlight a significant gap in current VLMs' ability to comprehend plant microscopic images. PlantMicro provides a standardized foundation for advancing VLMs toward reliable and comprehensive microscopy-level plant understanding.

Tianqi Wei, Xin Yu, Zhi Chen +2

The University of Queensland · Adelaide University · University of Southern Queensland

VLM Evaluation Microscopy Image Analysis

Ask PDF

Sep 29, 2026

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.

L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5

NIT Silchar · MBZUAI · BML Munjal University +1

VLM Evaluation Visual Question Answering Vision-Language Grounding VLM Reasoning Compositional Reasoning in VLMs

Ask PDF

Abstract

VLM Evaluation

Explore similar work

Can Edge-Deployable Vision-Language Models Identify Species?

Benchmarking Vision-Language Models for Microscopic Plant Image Understanding

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread