Predicting Species Splits: A Challenging Fine-Grained Benchmark for Category Discovery
Organizations: University of Edinburgh · University of Southern Denmark
Abstract
Widely used category discovery benchmarks have become increasingly saturated and often fail to reflect real-world discovery settings, making long-term evaluation of new methods difficult. We introduce PS-SPLIT, a large-scale fine-grained benchmark for category discovery, motivated by taxonomic splitting: the real-world process by which what was once considered a single species is found to comprise multiple distinct ones. PS-SPLIT contains 79K globally distributed, high-quality, community-collected bird images spanning 771 species, derived from 257 recent real expert-verified taxonomic splits, with associated geographic metadata and full taxonomic hierarchy. The visual differences between categories arising from real taxonomic splits can be very subtle, often requiring significant human expertise to tell apart, making PS-SPLIT a challenging benchmark which we hope will assist in advancing fine-grained category discovery research. Based on taxonomic splitting, we introduce the task of discovery by category splitting, where the goal is to determine which known categories should be subdivided into new finer-grained subcategories. We adapt four generalized category discovery methods to this setting, evaluate existing methods, and show that there is substantial room for future progress.
Figures & tables
| Partition | # Images | # Coarse | Splits | Controls | # Fine | Imgs/coarse | Imgs/fine | Fine/coarse | Mean | |
|---|---|---|---|---|---|---|---|---|---|---|
| Full | Known | 39,499 | 256 | 128 | 128 | 384 | 154.3 | 102.9 | 1.50 | 2.4 |
| Novel | 39,562 | 258 | 129 | 129 | 387 | 153.3 | 102.2 | 1.50 | 2.0 | |
| Total | 79,061 | 514 | 257 | 257 | 771 | 153.8 | 102.5 | 1.50 | 2.2 | |
| Small | Known | 12,645 | 64 | 32 | 32 | 96 | 197.6 | 131.7 | 1.50 | 1.3 |
| Novel | 12,640 | 64 | 32 | 32 | 96 | 197.5 | 131.7 | 1.50 | 0.0 | |
| Total | 25,285 | 128 | 64 | 64 | 192 | 197.5 | 131.7 | 1.50 | 0.7 |
| Backbone | Cl. Acc. | Cl. Acc. | AMI | AMI | K Acc. | K Acc. | Split Prec. | ||
|---|---|---|---|---|---|---|---|---|---|
| Brent-KMeans | DINOv1 | 0.70 | 0.52 | 0.31 | 0.00 | 0.73 | 0.66 | -10 | 0.45 |
| DINOv2 | 0.77 | 0.57 | 0.49 | 0.07 | 0.82 | 0.69 | -17 | 0.80 | |
| UNO+ | 0.75 | 0.51 | 0.50 | 0.00 | 0.75 | 0.50 | -32 | – | |
| GCD | 0.73 | 0.54 | 0.42 | 0.06 | 0.74 | 0.59 | -19 | 0.46 | |
| SimGCD | 0.75 | 0.51 | 0.50 | 0.00 | 0.75 | 0.50 | -32 | – | |
| SelEx | 0.75 | 0.54 | 0.42 | 0.03 | 0.75 | 0.59 | -20 | 0.50 |
| Method | Cl. Acc. | Cl. Acc. | AMI | AMI | K Acc. | K Acc. | Split Prec. | ||
|---|---|---|---|---|---|---|---|---|---|
| DINOv2, w/o Geo | Oracle | 0.83 | 0.66 | 0.60 | 0.20 | 1.00 | 1.00 | +0 | 1.00 |
| Tuning-Agglo. | 0.59 | 0.58 | 0.12 | 0.15 | 0.56 | 0.67 | +74 | 0.51 | |
| Tuning-Spectral | 0.70 | 0.55 | 0.32 | 0.05 | 0.71 | 0.62 | -11 | 0.38 | |
| Tuning-XMeans | 0.75 | 0.51 | 0.50 | 0.00 | 0.75 | 0.50 | -32 | – | |
| Brent-KMeans | 0.77 | 0.57 | 0.49 | 0.07 | 0.82 | 0.69 | -17 | 0.80 | |
| Brent-Agglo. | 0.74 | 0.53 | 0.43 | 0.02 | 0.76 | 0.59 | -21 | 0.55 |
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | Cl. Acc. | Cl. Acc. | AMI | AMI | K Acc. | K Acc. | Split Prec. | ||
|---|---|---|---|---|---|---|---|---|---|
| Brent-KMeans | DINOv1 | 0.75 | 0.60 | 0.35 | 0.00 | 0.74 | 0.64 | -52 | 0.48 |
| DINOv2 | 0.77 | 0.64 | 0.40 | 0.06 | 0.78 | 0.69 | -47 | 0.59 | |
| UNO+ | 0.80 | 0.61 | 0.48 | 0.00 | 0.74 | 0.50 | -124 | 0.00 | |
| GCD | 0.77 | 0.67 | 0.39 | 0.14 | 0.77 | 0.72 | -25 | 0.56 | |
| SimGCD | 0.80 | 0.62 | 0.48 | 0.01 | 0.75 | 0.52 | -118 | 0.45 | |
| SelEx | 0.77 | 0.64 | 0.39 | 0.08 | 0.77 | 0.69 | -42 | 0.55 |
| Dataset | Partition | # Images | # Coarse | # Fine | Imgs/coarse | Imgs/fine | Fine/coarse | Max K | K=1 | K>1 |
|---|---|---|---|---|---|---|---|---|---|---|
| CUB-200-2011 | Known | 5,930 | 19 | 100 | 312.1 | 59.3 | 5.26 | 30 | 6 | 13 |
| Novel | 5,858 | 19 | 100 | 308.3 | 58.6 | 5.26 | 22 | 6 | 13 | |
| Total | 11,788 | 38 | 200 | 310.2 | 58.9 | 5.26 | 30 | 12 | 26 | |
| Stanford Cars | Known | 8,026 | 24 | 98 | 334.4 | 81.9 | 4.08 | 22 | 10 | 14 |
| Novel | 8,159 | 25 | 98 | 326.4 | 83.3 | 3.92 | 15 | 9 | 16 | |
| Total | 16,185 | 49 | 196 | 330.3 | 82.6 | 4.00 | 22 | 19 | 30 |
| CUB-200-2011 | Stanford Cars | FGVC Aircraft | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | AMI | Cl. Acc. | K Acc. | AMI | Cl. Acc. | K Acc. | AMI | Cl. Acc. | K Acc. | ||||
| Brent-KMeans | DINOv1 | 0.38 | 0.52 | 0.36 | +65 | 0.14 | 0.41 | 0.36 | +30 | 0.19 | 0.61 | 0.52 | +37 |
| DINOv2 | 0.49 | 0.61 | 0.31 | +83 | 0.36 | 0.50 | 0.28 | +131 | 0.53 | 0.76 | 0.73 | +10 | |
| UNO+ | 0.54 | 0.57 | 0.49 | -60 | 0.63 | 0.71 | 0.57 | -38 | 0.77 | 0.87 | 0.86 | -10 | |
| GCD | 0.56 | 0.71 | 0.55 | -13 | 0.64 | 0.73 | 0.58 | +14 | 0.72 | 0.84 | 0.87 | -2 | |
| SimGCD | 0.52 | 0.60 | 0.49 | -37 | 0.64 | 0.73 | 0.61 | -26 | 0.75 | 0.86 | 0.85 | -8 | |
| CUB-200-2011 | Stanford Cars | FGVC Aircraft | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | AMI | Cl. Acc. | K Acc. | AMI | Cl. Acc. | K Acc. | AMI | Cl. Acc. | K Acc. | ||||
| Brent-KMeans | DINOv1 | 0.40 | 0.47 | 0.30 | +52 | 0.09 | 0.33 | 0.35 | -9 | 0.00 | 0.39 | 0.49 | -7 |
| DINOv2 | 0.64 | 0.63 | 0.27 | +64 | 0.51 | 0.53 | 0.29 | +79 | -0.00 | 0.37 | 0.42 | -4 | |
| UNO+ | 0.33 | 0.37 | 0.25 | -60 | 0.48 | 0.57 | 0.36 | -40 | 0.09 | 0.47 | 0.42 | -11 | |
| GCD | 0.58 | 0.64 | 0.48 | -19 | 0.63 | 0.67 | 0.46 | +8 | 0.10 | 0.48 | 0.62 | -7 | |
| SimGCD | 0.37 | 0.45 | 0.29 | -38 | 0.62 | 0.67 | 0.51 | -31 | 0.11 | 0.48 | 0.44 | -10 | |
| PS-SPLIT | CUB-200 | Scars | Aircraft | CIFAR-10 | CIFAR-100 | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | All | Old | New | All | Old | New | All | Old | New | All | Old | New | All | Old | New | All | Old | New |
| DINO | 43.2 | 45.7 | 39.0 | 42.1 | 48.2 | 36.4 | 12.7 | 11.7 | 14.0 | 17.6 | 12.4 | 21.9 | 88.8 | 91.0 | 87.8 | 65.0 | 68.5 | 58.0 |
| DINOv2 | 68.8 | 70.8 | 65.3 | 80.9 | 80.0 | 81.7 | 56.1 | 49.3 | 65.6 | 30.7 | 22.8 | 37.3 | 96.3 | 96.3 | 96.2 | 81.7 | 81.2 | 82.5 |
| UNO | 47.8 | 78.8 | 32.1 | 36.3 | 30.6 | 39.2 | 34.1 | 29.8 | 36.2 | 33.1 | 31.6 | 33.9 | 42.5 | 59.4 | 34.1 | 39.5 | 45.1 | 28.4 |
| GCD | 69.1 | 74.2 | 60.3 | 67.0 | 66.3 | 67.7 | 68.5 | 62.3 | 77.0 | 57.5 | 54.2 | 60.1 | 95.8 | 96.6 | 95.5 | 69.6 | 75.2 | 58.4 |
| SimGCD | 76.5 | 66.9 | 81.3 | 73.7 | 79.9 | 70.6 | 70.6 | 83.9 | 64.3 | 66.4 | 70.0 | 64.6 | 98.5 | 96.0 | 99.7 | 87.7 | 87.6 | 87.9 |
| Threshold | 0.60 (paper) | 0.70 | 0.80 | 0.90 |
|---|---|---|---|---|
| Spearman vs. 0.60 ordering | 1.00 (ref.) | 0.96 | 0.88 | 0.78 |
| % of coarse classes discarded | 0% | 6.2% | 30.7% | 62.6% |
| % of images discarded | 0% | 4.8% | 28.6% | 58.0% |
| Threshold | # Images | # Coarse | Splits | Controls | # Fine | Imgs/coarse | Imgs/fine | Fine/coarse | Mean |
|---|---|---|---|---|---|---|---|---|---|
| All retained | 80,842 | 526 | 263 | 263 | 789 | 153.7 | 102.5 | 1.50 | 2.2 |
| 0.60 (paper) | 79,061 | 514 | 257 | 257 | 771 | 153.8 | 102.5 | 1.50 | 2.2 |
| 0.65 | 77,789 | 504 | 252 | 252 | 756 | 154.3 | 102.9 | 1.50 | 2.3 |
| 0.70 | 75,283 | 482 | 241 | 241 | 723 | 156.2 | 104.1 | 1.50 | 2.4 |
| 0.75 | 68,349 | 432 | 216 | 216 | 648 | 158.2 | 105.5 | 1.50 | 2.4 |
| 0.80 | 56,451 | 356 | 178 | 178 | 534 | 158.6 | 105.7 | 1.50 | 2.5 |
| Overall | Asia | N. America | S. America | Africa | Europe | Oceania | |||||||||
| ( =258) | ( =72) | ( =60) | ( =40) | ( =38) | ( =30) | ( =18) | |||||||||
| Backbone | Features | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. |
| DINOv2 | w/o Geo | 0.75 | 0.63 | 0.74 | 0.63 | 0.75 | 0.55 | 0.75 | 0.64 | 0.74 | 0.66 | 0.78 | 0.62 | 0.80 | 0.69 |
| w/ Geo | 0.78 | 0.69 | 0.76 | 0.70 | 0.78 | 0.66 | 0.80 | 0.73 | 0.77 | 0.71 | 0.78 | 0.62 | 0.79 | 0.67 | |
| +0.02 | +0.06 | +0.03 | +0.07 | +0.03 | +0.11 | +0.04 | +0.09 | +0.03 | +0.05 | +0.00 | +0.00 | -0.01 | -0.02 | ||
| SelEx | w/o Geo | 0.76 | 0.65 | 0.76 | 0.67 | 0.76 | 0.57 | 0.77 | 0.65 | 0.75 | 0.67 | 0.78 | 0.64 | 0.83 | 0.69 |
| Overall ( =258) | Asia ( =72) | North America ( =60) | South America ( =40) | Africa ( =38) | Europe ( =30) | Oceania ( =18) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Row | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | |
| Brent-KMeans | DINOv1 | 0.75 | 0.60 | 0.70 | 0.61 | 0.77 | 0.56 | 0.78 | 0.66 | 0.73 | 0.61 | 0.80 | 0.56 | 0.72 | 0.58 |
| DINOv2 | 0.77 | 0.64 | 0.74 | 0.64 | 0.78 | 0.56 | 0.76 | 0.63 | 0.76 | 0.69 | 0.83 | 0.66 | 0.85 | 0.68 | |
| UNO+ | 0.80 | 0.61 | 0.77 | 0.63 | 0.82 | 0.54 | 0.80 | 0.64 | 0.77 | 0.63 | 0.81 | 0.56 | 0.88 | 0.64 | |
| GCD | 0.77 | 0.67 | 0.76 | 0.68 | 0.75 | 0.58 | 0.78 | 0.67 | 0.82 | 0.75 | 0.76 | 0.71 | 0.83 | 0.65 | |
| SimGCD | 0.80 | 0.62 | 0.77 | 0.63 | 0.82 | 0.54 | 0.80 | 0.64 | 0.76 | 0.63 | 0.84 | 0.60 | 0.88 | 0.64 | |
| Overall ( =258) | Asia ( =72) | North America ( =60) | South America ( =40) | Africa ( =38) | Europe ( =30) | Oceania ( =18) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Row | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | Cl. Acc. | |
| DINOv2, w/o Geo | Oracle | 0.83 | 0.66 | 0.81 | 0.67 | 0.84 | 0.59 | 0.82 | 0.67 | 0.79 | 0.67 | 0.87 | 0.67 | 0.90 | 0.70 |
| Tuning-Agglo. | 0.67 | 0.62 | 0.66 | 0.62 | 0.65 | 0.53 | 0.69 | 0.63 | 0.69 | 0.65 | 0.65 | 0.63 | 0.68 | 0.71 | |
| Tuning-Spectral | 0.78 | 0.63 | 0.77 | 0.64 | 0.77 | 0.57 | 0.77 | 0.63 | 0.75 | 0.63 | 0.79 | 0.63 | 0.88 | 0.72 | |
| Tuning-XMeans | 0.80 | 0.61 | 0.78 | 0.63 | 0.82 | 0.54 | 0.81 | 0.64 | 0.77 | 0.63 | 0.82 | 0.56 | 0.88 | 0.64 | |
| Brent-KMeans | 0.77 | 0.64 | 0.74 | 0.64 | 0.78 | 0.56 | 0.76 | 0.63 | 0.76 | 0.69 | 0.83 | 0.66 | 0.85 | 0.68 | |
| Method | Backbone | Split | Cl. Acc. | Cl. Acc. | AMI | AMI | K Acc. | K Acc. | Split Prec. | |
|---|---|---|---|---|---|---|---|---|---|---|
| Brent-KMeans | DINOv2 | Novel | 0.77 | 0.57 | 0.49 | 0.07 | 0.82 | 0.69 | -17 | 0.80 |
| Train | 0.72 | 0.55 | 0.38 | 0.04 | 0.75 | 0.64 | -14 | 0.50 | ||
| SelEx | Novel | 0.75 | 0.54 | 0.42 | 0.03 | 0.75 | 0.59 | -20 | 0.50 | |
| Train | 0.79 | 0.61 | 0.56 | 0.21 | 0.78 | 0.61 | -22 | 0.70 | ||
| Transformer | DINOv2 | Novel | 0.70 | 0.62 | 0.28 | 0.16 | 0.76 | 0.81 | +7 | 0.51 |
| Train | 0.68 | 0.60 | 0.20 | 0.12 | 0.73 | 0.83 | +12 | 0.48 |
| Region | Count | % | Continent | Count | % |
|---|---|---|---|---|---|
| North America | 9,713 | 12.3 | North America | 17,704 | 22.4 |
| Central America & Caribbean | 7,991 | 10.1 | South America | 12,463 | 15.8 |
| South America | 12,463 | 15.8 | Europe | 9,681 | 12.2 |
| Europe | 5,877 | 7.4 | Africa | 8,960 | 11.3 |
| North Africa | 451 | 0.6 | Asia | 23,397 | 29.6 |
| Sub-Saharan Africa | 8,623 | 10.9 | Oceania | 6,722 | 8.5 |
Generalized Category Discovery
Generalized category discovery learns known classes from labeled examples while grouping unlabeled data into known and previously unseen categories.
See all 30 papers