cs.CVFeb 3, 2026

Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study

Authors: Hugo Markoff, Stefan Hein Bengtson, Michael Ørsted

Organizations: Department of Chemistry and Bioscience, Aalborg University, Aalborg, Denmark · Department of Architecture, Design and Media Technology, Visual Analysis and Perception Lab, Aalborg University, Aalborg, Denmark

Abstract

Manual labelling of animal images remains a significant bottleneck in ecological research, limiting the scale and efficiency of biodiversity monitoring efforts. This study investigates whether state-of-the-art Vision Transformer (ViT) foundation models can reduce thousands of unlabelled animal images directly to species-level clusters. We present a comprehensive benchmarking framework evaluating five ViT models combined with five dimensionality reduction techniques and four clustering algorithms, two supervised and two unsupervised, across 60 species (30 mammals and 30 birds), with each test using a random subset of 200 validated images per species. We investigate when clustering succeeds at species-level, where it fails, and whether clustering within the species-level reveals ecologically meaningful patterns such as sex, age, or phenotypic variation. Our primary, fully unsupervised finding is that DINOv3 embeddings with t-SNE and HDBSCAN reach a species-level V-measure of 0.943 with no prior knowledge of the number of species, rejecting only 1.14% of images as outliers requiring expert review. Assigning one label per cluster reduces manual labelling actions by 98.3%, with 94.6% of clustered images labelled correctly, close to the accuracy of manual annotation. When the true number of species is supplied, supervised hierarchical clustering raises this to 0.958, which we report as an upper bound as the species count is typically unknown in real deployments. We further demonstrate robustness to realistic long-tailed distributions of species and show that intentional over-clustering can reveal intra-specific variation including age classes, sexual dimorphism, and pelage differences. We introduce an open-source benchmarking toolkit and provide recommendations for ecologists to select appropriate methods for sorting their specific taxonomic groups and data.

Explore similar work

CardsList
  1. InfoTaxa: Information-Calibrated Label-Free Clustering for Fine-Grained Visual Taxonomy

    Sep 15, 2026David Ahmedt-Aristizabal, Mohammad Ali Armin, Lars PeterssonVisual Representation LearningClustering

  2. Can Edge-Deployable Vision-Language Models Identify Species?

    Sep 10, 2026William Zhou, Mayukha Siripuram, Xiao Yan +2VLM EvaluationVLM Robustness

  3. How many labels do you need? A decision framework for cross-habitat marine species recognition

    Jun 28, 2026Alzayat Saleh, Mostafa Rahimi AzghadiDomain GeneralizationVision Foundation Model Adaptation