Behavior-Conditioned Animal Vocalization Generation: Event-Aware Modeling and Multidimensional Evaluation
Authors: Xinlong Deng, Yong Tian, Jian Guan, Qiuqiang Kong, Jie Jiang, Yin Cao
Organizations: College of Artificial Intelligence, China University of Petroleum, Beijing · College of Computer Science and Technology, Harbin Engineering University, Harbin, China · Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, China · Institute of Acoustics, Chinese Academy of Sciences
Behavior-conditioned animal vocalization generation aims to synthesize vocalizations from natural-language descriptions of animal behaviors, social interactions, and contexts. This task remains largely unexplored, with limited behavioral text supervision and substantial variation in vocalization duration. We construct the Behavior-Vocalization Dataset (BeVo) by integrating public bioacoustic data from six species with natural-language behavioral conditions derived from species-specific ethological and bioacoustic literature, totaling 31,191 events. We then adapt a rectified-flow-based generator with Event-Aware variable-length modeling, which encodes vocalizations at their actual durations and uses masks to prevent attention to padded latent positions and exclude these positions from the training loss. We evaluate vocalization discriminability, acoustic distribution fidelity, and language-aligned representation fidelity. Experiments show that task-specific adaptation improves all evaluated metrics compared with its zero-shot baseline, while Event-Aware modeling further increases vocalization-anchor macro accuracy from 59.86% to 61.99% and reduces Frechet Audio Distance from 1.40 to 1.22. Conditioning analyses further show that behavioral conditions improve target-anchor discriminability, while swapping behavior shifts generations toward the corresponding vocalization anchors. Demos are available on our project page.
Figures & tables
Figure 1: Overview of BeVo dataset construction. Recordings from six species are organized by vocalization anchors and linked to literature-grounded behavior cards. An LLM uses these cards to generate natural-language behavioral descriptions, which are paired with recordings sharing the corresponding anchor.
Split
Goat
Hyena
Marmo.
Meerkat
Zebra
Z. Finch
Total
Train
2826
10216
4800
4475
353
2298
24968
Val.
360
1226
600
564
62
311
3123
Test
400
800
600
600
200
500
3100
Total
3586
12242
6000
5639
615
3109
31191
Table 1: Number of vocalization events in each data split.
Figure 2: Overview of the event-aware latent modeling.
System
Macro Acc. ↑
mAP ↑
FAD ↓
AFDD ↓
AnimalCLAP-FAD ↓
BioLingual-FAD ↓
Real Reference
0.8852
0.9240
0.2321
0.2266
0.0075
0.0275
TangoFlux Zero-shot
0.1787
0.2022
6.7080
3.4759
0.3318
0.4904
TangoFlux Fine-tuned
0.5986
0.6345
1.4047
1.2480
0.0491
0.1237
TangoFlux Event-Aware
0.6199
0.6551
1.2239
0.9206
0.0492
0.1231
Table 2: Overall results on behavior-conditioned animal vocalization generation.
Species
Macro Acc. ↑
FAD ↓
BioLi. FAD ↓
Hyena
0.9000 /0.5500
0.1622 /1.1566
0.0227 /0.1029
Meerkat
0.7883 /0.6967
0.2478 /1.5149
0.0155 /0.0739
Marmoset
0.9717 /0.6550
0.1147 /1.8304
0.0225 /0.2129
Goat
0.8300 /0.4050
0.1561 /0.7219
0.0273 /0.0974
Zebra
0.9650 /0.6550
0.5648 /1.1341
0.0520 /0.1151
Zebra Finch
0.8560 /0.7580
0.1470 /0.9853
0.0249 /0.1364
Table 3: Species-wise performance of TangoFlux Event-Aware. Each entry reports Real Reference / Event-Aware.
Figure 3: t-SNE of real and TangoFlux Event-Aware vocalizations in AnimalCLAP space.
Condition Acc. mAP Full .6199 .6551 Species-only .2261 .2427
Behavior swap Value Δptgt +.3139 Increase rate 85.39%
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
Advances in AI-based voice conversion have enabled a wide range of media applications, including films, audiobooks, and games. However, most research and public benchmarks still focus on natural human speech, leaving designed vocalizations, such as monster growls and robotic voices, underexplored, partly due to the lack of publicly available resources. To address this gap, we introduce the Designed Vocalizations Dataset, constructed by curating diverse raw vocal sources, including speech and animal vocalizations, and applying professional vocal effects processing to produce corresponding effect modified variants. We further provide a standardized test set with explicit seen/unseen splits over source timbre groups and preset styles to assess generalization under controlled conditions. Finally, we report baseline benchmark results to support reproducible evaluation and future research. The dataset and demo samples are available at https://ncai-official.github.io/speech/publications/designed-vocalizations-dataset/.
Seolhee Lee, Minsu Kang, Yangsun Lee +3
NC AI Co., Ltd, Republic of Korea · Sogang University, Republic of Korea
Existing video-to-audio (V2A) generation methods predominantly rely on text prompts alongside visual information to synthesize audio. However, two critical bottlenecks persist: semantic granularity gaps in training data, such as conflating acoustically distinct sounds under coarse labels, and textual ambiguity in describing micro-acoustic features. These bottlenecks make it difficult to perform fine-grained sound synthesis using text-controlled modes. To address these limitations, we propose AC-Foley, an audio-conditioned V2A model that directly leverages reference audio to achieve precise and fine-grained control over generated sounds. This approach enables fine-grained sound synthesis, timbre transfer, zero-shot sound generation, and improved audio quality. By directly conditioning on audio signals, our approach bypasses the semantic ambiguities of text descriptions while enabling precise manipulation of acoustic attributes. Empirically, AC-Foley achieves state-of-the-art performance for Foley generation when conditioned on reference audio, while remaining competitive with state-of-the-art video-to-audio methods even without audio conditioning. Code and demo are available at: https://ff2416.github.io/AC-Foley-Page
Pengjun Fang, Yingqing He, Yazhou Xing +3
The Hong Kong University of Science and Technology · University of Central Florida
The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.