Fine-grained understanding and species-specific multimodal question answering are vital for advancing biodiversity conservation and ecological monitoring. However, existing multimodal large language models face challenges when it comes to specialized topics like avian species, making it harder to provide accurate and contextually relevant information in these areas. To address this limitation, we introduce the MAviS-Dataset, a large-scale multimodal avian species dataset that integrates image, audio, and text modalities for over 1,000 bird species, comprising both pretraining and instruction-tuning subsets enriched with structured question-answer pairs. Building on the MAviS-Dataset, we introduce MAviS-Chat, a multimodal LLM that supports audio, vision, and text and is designed for fine-grained species understanding, multimodal question answering, and scene-specific description generation. Finally, for quantitative evaluation, we present MAviS-Bench, a benchmark of over 25,000 QA pairs designed to assess avian species-specific perceptual and reasoning abilities across modalities. Experimental results show that MAviS-Chat outperforms the baseline MiniCPM-o-2.6 by a large margin, achieving state-of-the-art open-source results and demonstrating the effectiveness of our instruction-tuned MAviS-Dataset. Our findings highlight the necessity of domain-adaptive multimodal LLMs for ecological applications.
Figures & tables
Figure 2: Pipeline for generating image-text and audio-text annotations in the pretraining dataset. The process combines large-scale public resources with structured AI-assisted enrichment to ensure semantic accuracy and species-specific grounding.
Figure 3: The curation process for the fine-tuning dataset , detailing the sources, annotation, and refinement steps to ensure high-quality alignment. Broader multimodal Q&A types are shown in Figure 1 and detailed in Section 3.4 .
Figure 4: Distribution of Image and Audio Samples in the Pretraining and Fine-Tuning Sets.
Model
R1
M
BS
MS
ME-a
ME-c
GPT-4o
30.55
34.08
87.92
54.03
59.88
71.18
GPT-4o-mini
24.28
29.72
86.52
52.58
60.04
69.16
Gemini 1.5
18.95
23.31
84.42
50.63
45.16
47.36
Phi-4-MM-Instruct
32.43
30.18
88.68
54.42
48.04
57.70
MiniCPM-o-2.6
19.77
27.14
85.33
52.10
55.46
52.42
MAviS-Chat (our)
34.17
29.31
87.42
54.76
61.10
59.92
Table 1: Performance comparison on MAviS-Bench. ‘R1’, ‘M’, ‘BS’, and ‘MS’ denote ROUGE-1, METEOR, BERTScore, and MoverScore. ‘ME-a’ denotes MAviS-Eval (audio), generated from model responses to audio only questions. ‘ME-c’ is the MAviS-Eval (combined), evaluated on the full set comprising both audio- and image-based test samples.
Model
B1
R1
M
BS
MS
ME
GPT-4o-mini
10.27
11.53
14.49
83.24
51.06
31.06
MAviS-Chat
18.55
19.42
19.04
85.02
51.18
39.30
Table 2: Results on the real-world audio-visual video evaluation set. ‘B1’, ‘R1’, ‘M’, ‘BS’, and ‘MS’ denote BLEU-1, ROUGE-1, METEOR, BERTScore, and MoverScore. ‘ME’ denotes MAviS-Eval.
Training Layers
B1
R1
M
BS
MS
ME
All Proj. (MiniCPM.)
11.43
19.77
27.14
85.33
52.10
52.42
All Proj. (our model)
25.98
34.17
34.85
87.42
54.76
59.92
FFN only (our model)
39.65
36.33
29.61
86.72
55.11
59.42
Table 3: Ablation on the layers tuning with LoRA. ‘B1’, ‘R1’, ‘M’, ‘BS’, ‘MS’, and ‘ME’ denote BLEU-1, ROUGE-1, METEOR, BERTScore, MoverScore, and MAviS-Eval (combined). The first two rows denote that all projectors are fine-tuned using LoRA adapters, whereas the last row corresponds to training only the Feed-Forward Network (‘FFN’).
Training Strategy
Data Setup
B1
R1
M
BS
MS
ME
MiniCPM
-
11.43
19.77
27.14
85.33
52.10
52.42
Three-step Tuning
A → I → A (full)
25.98
34.17
34.85
87.42
54.76
59.92
Two-step Tuning
A→I (full)
38.88
36.46
29.31
86.66
55.07
58.16
Small+VQ Proj.
A→I→A (small)
30.64
33.99
34.07
87.35
54.58
58.84
Table 4: Ablation on training data setups. ‘B1’, ‘R1’, ‘M’, ‘BS’, ‘MS’, and ‘ME’ denote BLEU-1, ROUGE-1, METEOR, BERTScore, MoverScore, and MAviS-Eval (combined), respectively. The last three rows denote our MAviS-Chat model. A/I : Audio/Image. full/small : 65K (audio)/83K (image) or 35K data.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Methods
ROUGE-1
ROUGE-2
ROUGE-L
ROUGE-W
METEOR
Two-step
36.46
36.21
15.13
29.56
29.31
Three-step
34.17
37.33
16.40
31.36
34.85
Appendix
Table 5: Lexical quality evaluation of Two-step vs. Three-step tuning methods across ROUGE and METEOR metrics. Bold values indicate the best scores.
Figure 5: Additional multimodal question–answer samples from MAviS-Dataset, covering ecological inference such as eating habits, visual behaviour, and appearance interpretation, as well as detailed understanding of bird emotional tone and calling types from audio.
Figure 6: Prompt used for the instruction set generation for audio and video inputs.
Figure 7: Prompt template used in the MAviS-Eval metric to evaluate model performance on avian species identification within MAviS-Bench.
Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia · Faculty of Electrical Engineering, Universiti Teknologi Malaysia, 81310 Johor Bahru, Johor, Malaysia