Fine-grained bird species identification in the wild is frequently unanswerable from a single image: key cues may be non-visual (e.g. vocalization), or obscured due to occlusion, camera angle, or low resolution. Yet today's multimodal systems are typically judged on answerable, in-schema cases, encouraging confident guesses rather than principled abstention. We propose the RealBirdID benchmark: given an image of a bird, a system should either answer with a species or abstain with a concrete, evidence-based rationale: "requires vocalization," "low quality image," or "view obstructed". For each genus, the dataset includes a validation split composed of curated unanswerable examples with labeled rationales, paired with a companion set of clearly answerable instances. We find that (1) the species identification on the answerable set is challenging for a variety of open-source and proprietary models (less than 13% accuracy for MLLMs including GPT-5 and Gemini-2.5 Pro), (2) models with greater classification ability are not necessarily more calibrated to abstain from unanswerable examples, and (3) that MLLMs generally fail at providing correct reasons even when they do abstain. RealBirdID establishes a focused target for abstention-aware fine-grained recognition and a recipe for measuring progress.
Figures & tables
Figure 1: Preview of RealBirdID. In contrast to previous species identification datasets, in each of the genus associated with RealBirdID there is a corresponding set of unanswerable (UA) examples . The summary metric proposed gauges both (1) the ability for the classifier to disambiguate between confusing classes and (2) abstain from predicting on unanswerable examples. Incorrect abstention reasoning is shown in red whereas correct reasoning is shown in green .
Figure 2: Peek into the dataset . Few examples for the answerable and unanswerable images in RealBirdID. Unanswerable examples are grouped by possible unanswerable reasons. Images from “Sound needed” may be harder to abstain than “Quality” for example, as more detailed knowledge about the particular genus would be needed to understand these reasons.
Figure 3: Distribution of images across the answerable (A) and unanswerable (UA) subsets in RealBirdID. The genera within the unanswerable (UA) subset exhibit a highly imbalanced, long-tailed distribution. For example, 85% of UA images originate from just 61 genera that each contain more than five unanswerable samples. For details on the distribution of the 3,442 species, see Appendix D .
Figure 4: Multiple choice question (MCQ) formatting is a problem for encoder models. A straightforward implementation of an abstention class with CLIP models is to simply expose the genus level as a text prompt and treat its prediction as abstention. However, this approach greatly underperforms a modification from a previous hierarchical method, TreeGT [ 13 ] . “ HM ” refers to the harmonic mean of the “ Answerable ” and “ Unanswerable ” accuracies. Bird photographed by Tom Murray on iNaturalist ( link ).
Dataset
Has Train Set?
# A
# UA
# Species
NABirds
✓
24633
0
555
CUB-200
✓
5794
0
200
iNat19-Birds
✓
14860
0
1486
RealBirdID
✗
31885
3253
3442
Table 1: RealBirdID vs. related fine-grained bird benchmarks. “iNat19-Birds” refers to the validation split of iNat19 subset to the Aves class (namely only birds). “ #A ” and “ #UA ” refer to the number of answerable and unanswerable images, respectively.
CLIP
MetaCLIP
WildCLIP
Metric
B/32
B/16
L/14
B/32
B/16
L/14
Base
Lite
SigLIP
BioCLIP
InternVL3
Qwen-2.5VL
Gemma-3
Llama-3.2V
Gemini-2.5P
GPT-5
IG
50.6
55.3
62.0
54.2
59.1
66.0
55.1
53.3
53.7
68.9
46.5
54.2
46.3
48.4
56.4
57.7
UA/A
47.8
44.6
48.1
47.2
44.7
42.5
46.7
48.4
53.2
49.6
45.6
41.7
39.2
43.0
46.2
44.1
Table 2: Summary of classification and abstention performance on RealBirdID. “IG” and “UA/A” refer to the proposed Metrics 1 and 2 in Sec. 4 . InternVL3, Qwen-2.5VL, Gemma3, and Llama-3.2V are their 8B, 7B, 12B, and 11B instruction-tuned variants. Best performance for each metric is bolded whereas the next highest is underlined .
Figure 5: Visualization of sweeping parameters for classification and abstention metrics on popular CLIP-based models and MLLMs. To summarize how deep in the hierarchical classifiers can go while staying accurate we use Information Gain vs. Accuracy (a) . Each classifier admits a tradeoff curve when predicting species (b) and genus (d) . To measure separation between the unanswerable and answerable set, we measure the AUC of the model entropy (c) .
Answerable
Unanswerable
Species
Genus
Genus
AUC
Acc
AUC
Acc
AUC
Acc
Encoder-Based
CLIP-B/32
09.9
04.6
46.8
33.8
55.2
40.3
CLIP-B/16
12.3
05.8
53.3
40.3
63.2
46.5
CLIP-L/14
15.5
07.7
60.7
48.7
65.9
54.0
Table 3: Summary of classification performance of popular vision-language models in terms of taxonomic classification metrics. Columns report Area Under Curve (visualization in Fig. 5 ) and classification accuracy at the Species and Genus levels for the Answerable set and at the Genus level for the Unanswerable set. MLLMs are their instruction-tuned variant.
Figure 6: Effect of range-map information on encoders and MLLMs. Adding species range map information increases classification ability for both encoders and MLLMs, as shown by positive information gain across all models. For encoders, it also increases abstention ability, namely by increasing separation between A / UA. Notably, abstention tradeoff performance decreases for MLLMs. We provide distributions and examples showing this in Appendix B .
Figure 7: Abstention rate of MLLM on the UA set under equivalent writings of instructions. Each distribution comprises 15 semantically equivalent ways of writing the same instruction, depicted in gray on the left.
Figure 8: Confusion matrix of abstention reasons over various MLLMs. For each MLLM, the confusion matrix is computed only over abstained examples, comparing the human-labeled cause: image quality issues (Q), angle or occlusion (A), or dependence on vocalization (V). Cells show counts with row-normalized percentages in parentheses, and each panel title reports the abstention rate (AR), the fraction of all validation examples on which the model abstains.
Ground-Truth Issue
Model
quality
angle/occlusion
vocalization
Qwen2.5-VL-7B
0.158
0.144
0.098
Llama-3.2V-11B
0.086
0.080
0.077
InternVL3-8B
0.279
0.291
0.278
Gemma 3 12B
0.041
0.052
0.044
Table 4: Reason-agnostic abstention probability conditioned on ground-truth issue. Each value reports the probability that a model abstains for any reason over all ground truth abstention reasons. Higher values indicate that the model is more likely to abstain when that type of issue is present, regardless of which abstention label it chooses.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Abstention tradeoff curves for different abstention criteria.
Figure 10: Accuracy vs thresholding for different abstention criteria.
Figure 11: Frequency, probability entropy, and max probability for various choice counts in the Range Info Species Subset data. For the Range Info data, the examples are split up by the amount of choices that the model must choose between. For each choice count, the frequency, entropy, and max probability are shown. Bars are normalized by series. ( Top ) depicts the answerable set, whereas ( bottom ) depicts the unanswerable set. Both are run with Qwen-2.5VL-7B.
Figure 12: Location info of the Unanswerable (top) and Answerable (bottom) sets.
Figure 13: Distribution of species per observation and per taxon.
Figure 14: MANIQA distributions of RealBirdID (ours) vs. CUB200. On average, we find that RealBirdID has lower MANIQA scores.
Figure 15: Observation date distribution by observed and created timestamps.
genus
count
freq
cdf
species
Crows and Ravens
291
0.1261
0.1261
53
Large White-headed Gulls
198
0.0858
0.2119
25
Kingbirds
179
0.0776
0.2894
13
Empidonax Flycatchers
132
0.0572
0.3466
14
Mallards, Pintails, and Allies
125
0.0542
0.4008
33
Dryobates Woodpeckers
116
0.0503
0.4510
26
Appendix
Table 5: Most common generas occuring in the unanswerable data.
Figure 16: Classification results using class-averaging on the answerable and unanswerable sets.
Figure 17: Classification performance of CLIP-based models and popular MLLMs on the species level of answerables (A) and the genus level of unanswerables (UA). For various CLIP encoders, accuracies at percentile-based max probablities are plotted when sweeping over percent of data thresholded. For the answerable set the species label is used to compute accuracy (top) whereas for the unanswerable set, the genus label is used (bottom) . On the (right) we observe the effect of using species range maps to constrict the choice set. Note that the genus accuracy for encoders is not an error : subsetting species list using SINR location info achieves perfect genus performance.
Figure 18: Abstention calibration and entropy–threshold selective classification. For each model we sweep an entropy threshold on the flat species-level softmax and plot the fraction of unanswerable (UA) examples abstained on against the fraction of answerable (A) examples abstained (bottom) . We combine the UA / A performance into a unified classification metric using Information Gain vs. Accuracy as proposed by DARTS ( top ).
Figure 19: Summary of classification and abstention performance on RealBirdID. Blue bars show classification performance (IG), a metric which mixes species and genus-level accuracy over coverage on answerable (A) examples, while red bars show abstention capability, measured as AUROC for separating answerable (A) from unanswerable (UA) instances (higher is better for both). The lighter bars correspond to increases in performance from using species range maps whereas the darker bars indicate performance decreases. Notably, abstention tradeoff performance decreases for MLLMs.
Figure 20: 100 examples of images from the answerable set. The answerable set is sampled from Research Grade iNaturalist images to fill out species corresponding to sampled Unanswerable data.
Figure 21: 30 images with angle/occlusion abstention reasons from the unanswerable set.
Figure 22: 30 images with vocalization abstention reasons from the unanswerable set.
Figure 23: 30 images with image quality abstention reasons from the unanswerable set.
Fine-grained understanding and species-specific multimodal question answering are vital for advancing biodiversity conservation and ecological monitoring. However, existing multimodal large language models face challenges when it comes to specialized topics like avian species, making it harder to provide accurate and contextually relevant information in these areas. To address this limitation, we introduce the MAviS-Dataset, a large-scale multimodal avian species dataset that integrates image, audio, and text modalities for over 1,000 bird species, comprising both pretraining and instruction-tuning subsets enriched with structured question-answer pairs. Building on the MAviS-Dataset, we introduce MAviS-Chat, a multimodal LLM that supports audio, vision, and text and is designed for fine-grained species understanding, multimodal question answering, and scene-specific description generation. Finally, for quantitative evaluation, we present MAviS-Bench, a benchmark of over 25,000 QA pairs designed to assess avian species-specific perceptual and reasoning abilities across modalities. Experimental results show that MAviS-Chat outperforms the baseline MiniCPM-o-2.6 by a large margin, achieving state-of-the-art open-source results and demonstrating the effectiveness of our instruction-tuned MAviS-Dataset. Our findings highlight the necessity of domain-adaptive multimodal LLMs for ecological applications.
Yevheniia Kryklyvets, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly +5
Mohamed bin Zayed University of Artificial Intelligence
Bioacoustic datasets from tropical regions remain limited, in part due to the absence of reproducible workflows for aggregating recordings from public archives. We present MyGardenBird, a curated dataset of bird vocalisations representing twelve common species across Peninsular Malaysia and the Indo-Malayan region. Recordings were sourced from Xeno-canto and processed through species-level filtering, manual spectrogram segmentation, and quality control checks. The primary release comprises 7,200 manually validated audio clips (16 kHz, 16-bit PCM mono WAV), balanced at 600 three-second clips per species (6.0 hours total) derived from 1,381 distinct recordings. Metadata includes geospatial coordinates, vocalisation categories, and signal-to-noise ratio (SNR) values (range: 0.83--59.18 dB; mean: 15.80 dB). A supplementary 44.1 kHz version is also provided. To mitigate data leakage, dataset partitions are defined at the source-recording level. Baseline classification experiments using convolutional neural networks on Mel-spectrograms achieved test accuracies of 92--96%, indicating strong interspecies separability. Limitations include reliance on single-annotator curation; however, validation with BirdNET confirmed label consistency. MyGardenBird is openly available at https://doi.org/10.5281/zenodo.20306877 under a CC BY-NC-SA 4.0 licence. Complete preprocessing code accompanies the release to support reproducibility and future expansion.
Muhammad Mun'im Ahmad Zabidi, Mohd Yamani Idna Idris, Norisma Idris
Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia · Faculty of Electrical Engineering, Universiti Teknologi Malaysia, 81310 Johor Bahru, Johor, Malaysia
Camera traps often run in the field on edge hardware with limited or no connectivity, making small, locally-deployable vision-language models (VLMs) -- not frontier-scale ones -- the practically relevant class to evaluate for species identification. We test whether models in this deployment-relevant 2--8B range carry genuine taxonomic knowledge, evaluating four such VLMs (Qwen3-VL 2B/4B/8B, Gemma3 4B) against the domain-specific specialist BioCLIP (300M parameters) on a 96-species task, comparing clean iNaturalist photographs against camera-trap imagery from 6 LILA.science collections, on two independently-sampled evaluation sets. All models identify species far above chance, but every model -- general-purpose or specialist -- degrades sharply on field imagery (domain gaps of 9.6--26.6 percentage points, consistent across taxonomic levels and both evaluation sets), indicating the degradation reflects general image legibility rather than fine-grained discrimination failure. BioCLIP substantially outperforms every VLM tested (by 33.2--59.2 percentage points across an expanded 200-image sample for every model) despite its far smaller size, suggesting the gap reflects specialized training data rather than model scale; yet BioCLIP's own domain gap (18.0 points) is statistically indistinguishable from the best VLM's (22.3 points), suggesting the clean-to-field degradation itself is a property of the image-quality shift rather than a general-purpose-model weakness. Under open-set prompting, 5.9--9.6% of responses are syntactically valid but taxonomically nonexistent species names; the relative fabrication-rate ranking across models replicates exactly across both evaluation sets, a more robust finding than any single point estimate.