cs.CVMar 27, 2026
SaveRealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs
Organizations: Computer Vision Lab, UMass Amherst
Abstract
Fine-grained bird species identification in the wild is frequently unanswerable from a single image: key cues may be non-visual (e.g. vocalization), or obscured due to occlusion, camera angle, or low resolution. Yet today's multimodal systems are typically judged on answerable, in-schema cases, encouraging confident guesses rather than principled abstention. We propose the RealBirdID benchmark: given an image of a bird, a system should either answer with a species or abstain with a concrete, evidence-based rationale: "requires vocalization," "low quality image," or "view obstructed". For each genus, the dataset includes a validation split composed of curated unanswerable examples with labeled rationales, paired with a companion set of clearly answerable instances. We find that (1) the species identification on the answerable set is challenging for a variety of open-source and proprietary models (less than 13% accuracy for MLLMs including GPT-5 and Gemini-2.5 Pro), (2) models with greater classification ability are not necessarily more calibrated to abstain from unanswerable examples, and (3) that MLLMs generally fail at providing correct reasons even when they do abstain. RealBirdID establishes a focused target for abstention-aware fine-grained recognition and a recipe for measuring progress.
Figures & tables
Figure 1: Preview of RealBirdID. In contrast to previous species identification datasets, in each of the genus associated with RealBirdID there is a corresponding set of unanswerable (UA) examples . The summary metric proposed gauges both (1) the ability for the classifier to disambiguate between confusing classes and (2) abstain from predicting on unanswerable examples. Incorrect abstention reasoning is shown in red whereas correct reasoning is shown in green .
Figure 2: Peek into the dataset . Few examples for the answerable and unanswerable images in RealBirdID. Unanswerable examples are grouped by possible unanswerable reasons. Images from “Sound needed” may be harder to abstain than “Quality” for example, as more detailed knowledge about the particular genus would be needed to understand these reasons.
Figure 3: Distribution of images across the answerable (A) and unanswerable (UA) subsets in RealBirdID. The genera within the unanswerable (UA) subset exhibit a highly imbalanced, long-tailed distribution. For example, 85% of UA images originate from just 61 genera that each contain more than five unanswerable samples. For details on the distribution of the 3,442 species, see Appendix D .
Figure 4: Multiple choice question (MCQ) formatting is a problem for encoder models. A straightforward implementation of an abstention class with CLIP models is to simply expose the genus level as a text prompt and treat its prediction as abstention. However, this approach greatly underperforms a modification from a previous hierarchical method, TreeGT [ 13 ] . “ HM ” refers to the harmonic mean of the “ Answerable ” and “ Unanswerable ” accuracies. Bird photographed by Tom Murray on iNaturalist ( link ).
| Dataset | Has Train Set? | # A | # UA | # Species |
|---|---|---|---|---|
| NABirds | ✓ | 24633 | 0 | 555 |
| CUB-200 | ✓ | 5794 | 0 | 200 |
| iNat19-Birds | ✓ | 14860 | 0 | 1486 |
| RealBirdID | ✗ | 31885 | 3253 | 3442 |
Table 1: RealBirdID vs. related fine-grained bird benchmarks. “iNat19-Birds” refers to the validation split of iNat19 subset to the Aves class (namely only birds). “ #A ” and “ #UA ” refer to the number of answerable and unanswerable images, respectively.
| CLIP | MetaCLIP | WildCLIP | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Metric | B/32 | B/16 | L/14 | B/32 | B/16 | L/14 | Base | Lite | SigLIP | BioCLIP | InternVL3 | Qwen-2.5VL | Gemma-3 | Llama-3.2V | Gemini-2.5P | GPT-5 |
| IG | 50.6 | 55.3 | 62.0 | 54.2 | 59.1 | 66.0 | 55.1 | 53.3 | 53.7 | 68.9 | 46.5 | 54.2 | 46.3 | 48.4 | 56.4 | 57.7 |
| UA/A | 47.8 | 44.6 | 48.1 | 47.2 | 44.7 | 42.5 | 46.7 | 48.4 | 53.2 | 49.6 | 45.6 | 41.7 | 39.2 | 43.0 | 46.2 | 44.1 |
Table 2: Summary of classification and abstention performance on RealBirdID. “IG” and “UA/A” refer to the proposed Metrics 1 and 2 in Sec. 4 . InternVL3, Qwen-2.5VL, Gemma3, and Llama-3.2V are their 8B, 7B, 12B, and 11B instruction-tuned variants. Best performance for each metric is bolded whereas the next highest is underlined .
Figure 5: Visualization of sweeping parameters for classification and abstention metrics on popular CLIP-based models and MLLMs. To summarize how deep in the hierarchical classifiers can go while staying accurate we use Information Gain vs. Accuracy (a) . Each classifier admits a tradeoff curve when predicting species (b) and genus (d) . To measure separation between the unanswerable and answerable set, we measure the AUC of the model entropy (c) .
| Answerable | Unanswerable | |||||
| Species | Genus | Genus | ||||
| AUC | Acc | AUC | Acc | AUC | Acc | |
| Encoder-Based | ||||||
| CLIP-B/32 | 09.9 | 04.6 | 46.8 | 33.8 | 55.2 | 40.3 |
| CLIP-B/16 | 12.3 | 05.8 | 53.3 | 40.3 | 63.2 | 46.5 |
| CLIP-L/14 | 15.5 | 07.7 | 60.7 | 48.7 | 65.9 | 54.0 |
Table 3: Summary of classification performance of popular vision-language models in terms of taxonomic classification metrics. Columns report Area Under Curve (visualization in Fig. 5 ) and classification accuracy at the Species and Genus levels for the Answerable set and at the Genus level for the Unanswerable set. MLLMs are their instruction-tuned variant.
Figure 6: Effect of range-map information on encoders and MLLMs. Adding species range map information increases classification ability for both encoders and MLLMs, as shown by positive information gain across all models. For encoders, it also increases abstention ability, namely by increasing separation between A / UA. Notably, abstention tradeoff performance decreases for MLLMs. We provide distributions and examples showing this in Appendix B .
Figure 7: Abstention rate of MLLM on the UA set under equivalent writings of instructions. Each distribution comprises 15 semantically equivalent ways of writing the same instruction, depicted in gray on the left.
Figure 8: Confusion matrix of abstention reasons over various MLLMs. For each MLLM, the confusion matrix is computed only over abstained examples, comparing the human-labeled cause: image quality issues (Q), angle or occlusion (A), or dependence on vocalization (V). Cells show counts with row-normalized percentages in parentheses, and each panel title reports the abstention rate (AR), the fraction of all validation examples on which the model abstains.
| Ground-Truth Issue | |||
|---|---|---|---|
| Model | quality | angle/occlusion | vocalization |
| Qwen2.5-VL-7B | 0.158 | 0.144 | 0.098 |
| Llama-3.2V-11B | 0.086 | 0.080 | 0.077 |
| InternVL3-8B | 0.279 | 0.291 | 0.278 |
| Gemma 3 12B | 0.041 | 0.052 | 0.044 |
Table 4: Reason-agnostic abstention probability conditioned on ground-truth issue. Each value reports the probability that a model abstains for any reason over all ground truth abstention reasons. Higher values indicate that the model is more likely to abstain when that type of issue is present, regardless of which abstention label it chooses.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Abstention tradeoff curves for different abstention criteria.
Figure 10: Accuracy vs thresholding for different abstention criteria.
Figure 11: Frequency, probability entropy, and max probability for various choice counts in the Range Info Species Subset data. For the Range Info data, the examples are split up by the amount of choices that the model must choose between. For each choice count, the frequency, entropy, and max probability are shown. Bars are normalized by series. ( Top ) depicts the answerable set, whereas ( bottom ) depicts the unanswerable set. Both are run with Qwen-2.5VL-7B.
Figure 12: Location info of the Unanswerable (top) and Answerable (bottom) sets.
Figure 13: Distribution of species per observation and per taxon.
Figure 14: MANIQA distributions of RealBirdID (ours) vs. CUB200. On average, we find that RealBirdID has lower MANIQA scores.
Figure 15: Observation date distribution by observed and created timestamps.
| genus | count | freq | cdf | species |
| Crows and Ravens | 291 | 0.1261 | 0.1261 | 53 |
| Large White-headed Gulls | 198 | 0.0858 | 0.2119 | 25 |
| Kingbirds | 179 | 0.0776 | 0.2894 | 13 |
| Empidonax Flycatchers | 132 | 0.0572 | 0.3466 | 14 |
| Mallards, Pintails, and Allies | 125 | 0.0542 | 0.4008 | 33 |
| Dryobates Woodpeckers | 116 | 0.0503 | 0.4510 | 26 |
Table 5: Most common generas occuring in the unanswerable data.
Figure 16: Classification results using class-averaging on the answerable and unanswerable sets.
Figure 17: Classification performance of CLIP-based models and popular MLLMs on the species level of answerables (A) and the genus level of unanswerables (UA). For various CLIP encoders, accuracies at percentile-based max probablities are plotted when sweeping over percent of data thresholded. For the answerable set the species label is used to compute accuracy (top) whereas for the unanswerable set, the genus label is used (bottom) . On the (right) we observe the effect of using species range maps to constrict the choice set. Note that the genus accuracy for encoders is not an error : subsetting species list using SINR location info achieves perfect genus performance.
Figure 18: Abstention calibration and entropy–threshold selective classification. For each model we sweep an entropy threshold on the flat species-level softmax and plot the fraction of unanswerable (UA) examples abstained on against the fraction of answerable (A) examples abstained (bottom) . We combine the UA / A performance into a unified classification metric using Information Gain vs. Accuracy as proposed by DARTS ( top ).
Figure 19: Summary of classification and abstention performance on RealBirdID. Blue bars show classification performance (IG), a metric which mixes species and genus-level accuracy over coverage on answerable (A) examples, while red bars show abstention capability, measured as AUROC for separating answerable (A) from unanswerable (UA) instances (higher is better for both). The lighter bars correspond to increases in performance from using species range maps whereas the darker bars indicate performance decreases. Notably, abstention tradeoff performance decreases for MLLMs.
Figure 20: 100 examples of images from the answerable set. The answerable set is sampled from Research Grade iNaturalist images to fill out species corresponding to sampled Unanswerable data.
Figure 21: 30 images with angle/occlusion abstention reasons from the unanswerable set.
Figure 22: 30 images with vocalization abstention reasons from the unanswerable set.
Figure 23: 30 images with image quality abstention reasons from the unanswerable set.