BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

Risa Shinoda; Kaede Shiohara; Nakamasa Inoue; Kuniaki Saito; Hiroaki Santo; Fumio Okura

cs.CVMar 25, 2026

BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

Authors: Risa Shinoda, Kaede Shiohara, Nakamasa Inoue, Kuniaki Saito, Hiroaki Santo, Fumio Okura

Organizations: The University of Osaka · The University of Tokyo · Institute of Science Tokyo · OMRON SINIC X

Abstract

Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/

Figures & tables

Figure 1: We introduce $\mathsf{BioVITA}$ for biological visual-textual-acoustic alignment. We curate a million-scale dataset, train a unified representation model, and develop a comprehensive species-level retrieval benchmark.

Dataset	#Audio Clips	#Images	Taxon	#Species	#Traits
BEANS [ 12 ]	56k	0	Com	391	0
AnimalSpeak [ 36 ]	1.1M	0	Com, Sci	12k	0
iNatSounds [ 3 ]	230k	0	Full	5.5k	0
SSW60 [ 17 ]	4k	31k	Com	60	0
$\mathsf{BioVITA\hskip 1.5ptTrain}$	1.3M	2.3M	Full	14k	34

Table 1: Comparison with existing animal sound datasets. The number of species excludes subspecies. Com and Sci stand for common and scientific names, respectively.

Category	Trait
Diet Type ⋆	Herbivorous
	Carnivorous
	Omnivorous
	Specialized
Activity Pattern ⋆	Diurnal
	Nocturnal

Table 2: Ecological trait labels for

\mathsf{BioVITA}

. ⋆ indicates mutually exclusive traits, where only one trait within a category is active.

Figure 2: Representative taxonomic distribution of the dataset across the five animal classes. Numbers indicate audio recordings.

Figure 5: Examples from $\mathsf{BioVITA\hskip 1.5ptTrain}$ . Images and audio clips are shown with their corresponding scientific names.

Figure 6: $\mathsf{BioVITA\hskip 1.5ptModel}$ consists of three encoders. Building upon BioCLIP 2, we train the audio encoder in Stage 1, and jointly train the audio and text encoders in Stage 2.

Figure 7: Task examples for $\mathsf{BioVITA~Bench}$ . Given a query, models are required to identify the relevant sample (indicated by green rectangles) from databases of 100 samples each. Five samples per database are shown.

Model	Audio $\rightarrow$ Text		Text $\rightarrow$ Audio		Audio $\rightarrow$ Image		Image $\rightarrow$ Audio		Image $\rightarrow$ Text		Text $\rightarrow$ Image		Average
	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5
Random	0.01	0.05	0.01	0.05	0.01	0.05	0.01	0.05	0.01	0.05	0.01	0.05	0.01	0.05
CLIP [ 33 ]	–	–	–	–	–	–	–	–	50.9	65.1	63.8	72.5	–	–
CLAP [ 55 ]	0.7	4.7	1.0	4.7	–	–	–	–	–	–	–	–	–	–
ImageBind [ 9 ]	2.0	8.9	2.8	12.9	1.8	8.7	2.9	12.2	59.5	81.4	67.9	86.7	22.8	35.1
BioCLIP 2 [ 10 ]	–	–	–	–	–	–	–	–	65.1	80.5	84.8	95.3	–	–

Table 3: Species-level cross-modal retrieval results on the seen subset. We report Top 1 and 5 accuracies for each retrieval direction.

Figure 8: Accuracy by taxonomy class for each retrieval scenario.

Prompt	A2T	T2A	A2I	I2A	I2T	T2I	Avg.
Common Name	63.7	81.1	50.3	57.5	86.3	91.2	71.7
Science Name	67.0	84.2	50.2	57.6	88.1	93.6	73.5

Table 4: Prompt selection for

\mathsf{BioVITA}

on species-level retrieval.

Model		Audio $\rightarrow$ Text		Text $\rightarrow$ Audio		Audio $\rightarrow$ Image		Image $\rightarrow$ Audio		Image $\rightarrow$ Text		Text $\rightarrow$ Image		Average
		Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5
Genus	ImageBind [ 9 ]	1.3	6.1	1.6	8.3	1.8	9.0	2.7	12.1	6.6	14.2	7.3	14.0	3.6	10.6
	BioCLIP 2 [ 10 ]	–	–	–	–	–	–	–	–	19.3	34.5	71.7	87.0	–	–
	TaxaBind [ 38 ]	5.9	18.4	14.1	34.2	11.7	33.8	14.9	54.3	30.8	51.7	50.5	72.3	21.3	44.1
	$\mathsf{BioVITA}$ (Stage1)	37.3	58.1	80.9	91.1	41.8	67.9	42.8	72.5	19.3	34.5	71.7	87.0	49.0	68.5
	$\mathsf{BioVITA}$ (Stage2)	53.6	75.4	84.9	94.2	43.2	71.5	48.6	78.0	74.7	89.6	90.7	96.1	66.0	84.1

Table 5: High-level retrieval results at the Genus and Family levels, where queries and targets belong to different species within the same taxon. This setting is more challenging as it requires models to capture semantic relationships beyond exact species-level identification.

Model	Audio $\rightarrow$ Text		Text $\rightarrow$ Audio		Audio $\rightarrow$ Image		Image $\rightarrow$ Audio		Image $\rightarrow$ Text		Text $\rightarrow$ Image		Average
	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5
ImageBind [ 9 ]	2.2	11.8	1.5	8.9	1.1	9.2	3.0	12.1	42.8	67.5	47.6	71.5	16.4	30.2
BioCLIP 2 [ 10 ]	–	–	–	–	–	–	–	–	33.0	59.5	37.1	66.7	–	–
$\mathsf{BioVITA}$ (Stage1)	14.6	32.9	20.0	43.4	7.8	23.6	6.4	25.8	33.0	59.5	37.1	66.7	19.8	42.0
$\mathsf{BioVITA}$ (Stage2)	36.6	62.2	54.1	74.8	29.3	54.3	28.0	55.5	76.8	94.7	86.5	96.2	51.9	73.0

Table 6: Species-level cross-modal retrieval results on the unseen subset. We report Top 1 and 5 accuracies for each retrieval direction.

Figure 9: Genus and family-level consistency of Top-1 misclassifications across retrieval tasks. The $\mathsf{BioVITA}$ model more frequently predicts the correct genus or family despite species-level errors.

Category		CLAP	CLIP	$\mathsf{BioVITA}$	$\mathsf{BioVITA}$
	(Audio)	CLAP	CLIP	(Image)
Diet Type		43.7	41.5	53.9	41.1
Act. Pattern		80.9	82.1	83.7	82.2
Locomotion		79.7	84.3	81.1	83.1
Lifestyle		82.7	92.5	85.1	89.9
Trophic Role		79.4	89.2	89.7	88.0

Table 7: Ecological trait prediction for unseen species. All models are fine-tuned on the proposed training dataset and evaluated using the F1 score.

Stage
Method	A2T	T2A	A2I	I2A	I2T	T2I	Avg.
$\mathsf{BioVITA}$ (full)	63.7	81.1	50.3	57.5	86.3	91.2	71.7
w/o Stage1	20.7	29.6	10.3	15.1	49.9	45.6	34.2
w/o Stage2	60.3	79.3	47.8	48.6	65.1	84.8	64.3
Stage1 Initial Weight
w/o BioCLIP2	15.8	18.2	1.32	1.17	50.9	63.8	25.1

Table 8: Ablation study. We report Top-1 accuracy for each retrieval direction.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Template	Example (‘Anianiau)
Common Name (Com)	‘Anianiau
Scientific Name (Sci)	Magumma Parva
Taxonomic Sequence (Tax)	Aves Passeriformes, Fringillidae Magumma, Magumma Parva
Sci + Com	Magumma Parva with a common name ‘Anianiau
Tax + Com	Aves Passeriformes, Fringillidae Magumma, Magumma Parva,
Tax + Com	with a common name ‘Anianiau

Appendix

Table 9: Examples of textual descriptions following the five templates used in training.

Model	Species				Genus				Family				Average
Model	Audio $\rightarrow$ Text		Text $\rightarrow$ Audio		Audio $\rightarrow$ Text		Text $\rightarrow$ Audio		Audio $\rightarrow$ Text		Text $\rightarrow$ Audio		Audio $\rightarrow$ Text		Text $\rightarrow$ Audio
	Top1	Top5	Top1	Top5	Top1	Top5	Top1	Top5	Top1	Top5	Top1	Top5	Top1	Top5	Top1	Top5
BioLingual [ 36 ]	24.0	45.9	26.5	50.3	23.2	42.8	51.7	68.2	5.4	15.9	20.2	38.9	17.5	34.9	32.8	52.5
$\mathsf{BioVITA}$	24.4	49.2	27.7	56.6	32.5	51.4	58.0	74.7	17.4	42.9	52.3	77.3	24.8	47.9	46.0	69.5

Appendix

Table 10: Classification results of BioLingual and

\mathsf{BioVITA}

. Since BioLingual may include part of our test data in its training split, we construct a new 2024 audio test set and evaluate the classification performance on it.

Table 19

Figure 10: t-SNE visualization. Our model successfully learns all three modalities in Stage 2 without collapsing the audio feature embedding space.

Model		Audio $\rightarrow$ Text		Text $\rightarrow$ Audio		Audio $\rightarrow$ Image		Image $\rightarrow$ Audio		Image $\rightarrow$ Text		Text $\rightarrow$ Image		Average
		Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5	Top 1	Top 5
Sci	25%	48.0	68.1	67.5	85.9	39.6	65.9	41.1	71.4	45.4	59.5	80.9	93.2	53.8	74.0
	50%	53.6	73.4	75.2	90.5	43.7	70.0	44.5	74.5	45.4	59.5	81.1	93.5	57.2	76.9
	$\mathsf{BioVITA}$ (Full)	60.3	80.0	79.3	93.9	47.8	72.8	48.6	78.8	65.1	80.5	84.8	95.3	64.3	83.6
Com	25%	50.0	72.1	65.8	86.7	39.6	65.9	41.1	71.4	65.1	80.4	84.5	95.1	57.7	78.6
	50%	55.7	76.3	73.0	90.6	43.7	70.0	44.5	74.5	65.1	80.5	84.6	95.2	61.1	81.2

Appendix

Table 13: Training dataset size variation. The amount of training data significantly affects model performance. These results correspond to Stage 1. “Sci” and “Com” denote the prompt types used during inference: scientific name and common name, respectively.

Figure 12: Genus distribution. The blue bars represent the training set, while the orange bars represent the test set.

Figure 13: Genus distribution. The blue bars represent the training set, while the orange bars represent the test set.

Figure 14: Genus distribution. The blue bars represent the training set, while the orange bars represent the test set.

Figure 15: Genus distribution. The blue bars represent the training set, while the orange bars represent the test set.

Explore similar work

AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference
Mar 23, 2026Risa Shinoda, Kaede Shiohara, Nakamasa Inoue +2Audio-Language Models Bioacoustics
The University of Osaka · The University of Tokyo · Institute of Science Tokyo
Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inference
Mar 26, 2026Sk Miraj Ahmed, Xi Yu, Yunqi Li +2Hierarchical Representation Learning Multimodal Robustness
Computing and Data Sciences, Brookhaven National Laboratory, Upton, NY 11973, USA · Rutgers University, New Brunswick, NJ, USA
MAviS: A Multimodal Conversational Assistant For Avian Species
Mar 7, 2026Yevheniia Kryklyvets, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly +5Multimodal Large Language Models Multimodal QA
Mohamed bin Zayed University of Artificial Intelligence

Mar 23, 2026J/K move · Enter open · S save

AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference

Risa Shinoda, Kaede Shiohara, Nakamasa Inoue, Hiroaki Santo+1

The University of Osaka · The University of Tokyo · Institute of Science Tokyo

Audio-Language Models Bioacoustics Audio Representation Learning

Open paper AskPDF arXiv

Animal vocalizations provide crucial insights for wildlife assessment, particularly in complex environments such as forests, aiding species identification and ecological monitoring. Recent advances in deep learning have enabled automatic species classification from their vocalizations. However, classifying species unseen during training remains challenging. To address this limitation, we introduce AnimalCLAP, a taxonomy-aware language-audio framework comprising a new dataset and model that incorporate hierarchical biological information. Specifically, our vocalization dataset consists of 4,225 hours of recordings covering 6,823 species, annotated with 22 ecological traits. The AnimalCLAP model is trained on this dataset to align audio and textual representations using taxonomic structures, improving the recognition of unseen species. We demonstrate that our proposed model effectively infers ecological and biological attributes of species directly from their vocalizations, achieving superior performance compared to CLAP. Our dataset, code, and models will be publicly available at https://dahlian00.github.io/AnimalCLAP_Page/.

Abstract

Explore similar work

AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference

Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inference

MAviS: A Multimodal Conversational Assistant For Avian Species