Behavior-Conditioned Animal Vocalization Generation: Event-Aware Modeling and Multidimensional Evaluation
Authors: Xinlong Deng, Yong Tian, Jian Guan, Qiuqiang Kong, Jie Jiang, Yin Cao
Organizations: College of Artificial Intelligence, China University of Petroleum, Beijing · College of Computer Science and Technology, Harbin Engineering University, Harbin, China · Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, China · Institute of Acoustics, Chinese Academy of Sciences
Behavior-conditioned animal vocalization generation aims to synthesize vocalizations from natural-language descriptions of animal behaviors, social interactions, and contexts. This task remains largely unexplored, with limited behavioral text supervision and substantial variation in vocalization duration. We construct the Behavior-Vocalization Dataset (BeVo) by integrating public bioacoustic data from six species with natural-language behavioral conditions derived from species-specific ethological and bioacoustic literature, totaling 31,191 events. We then adapt a rectified-flow-based generator with Event-Aware variable-length modeling, which encodes vocalizations at their actual durations and uses masks to prevent attention to padded latent positions and exclude these positions from the training loss. We evaluate vocalization discriminability, acoustic distribution fidelity, and language-aligned representation fidelity. Experiments show that task-specific adaptation improves all evaluated metrics compared with its zero-shot baseline, while Event-Aware modeling further increases vocalization-anchor macro accuracy from 59.86% to 61.99% and reduces Frechet Audio Distance from 1.40 to 1.22. Conditioning analyses further show that behavioral conditions improve target-anchor discriminability, while swapping behavior shifts generations toward the corresponding vocalization anchors. Demos are available on our project page.
Figures & tables
Figure 1: Overview of BeVo dataset construction. Recordings from six species are organized by vocalization anchors and linked to literature-grounded behavior cards. An LLM uses these cards to generate natural-language behavioral descriptions, which are paired with recordings sharing the corresponding anchor.
Split
Goat
Hyena
Marmo.
Meerkat
Zebra
Z. Finch
Total
Train
2826
10216
4800
4475
353
2298
24968
Val.
360
1226
600
564
62
311
3123
Test
400
800
600
600
200
500
3100
Total
3586
12242
6000
5639
615
3109
31191
Table 1: Number of vocalization events in each data split.
Figure 2: Overview of the event-aware latent modeling.
System
Macro Acc. ↑
mAP ↑
FAD ↓
AFDD ↓
AnimalCLAP-FAD ↓
BioLingual-FAD ↓
Real Reference
0.8852
0.9240
0.2321
0.2266
0.0075
0.0275
TangoFlux Zero-shot
0.1787
0.2022
6.7080
3.4759
0.3318
0.4904
TangoFlux Fine-tuned
0.5986
0.6345
1.4047
1.2480
0.0491
0.1237
TangoFlux Event-Aware
0.6199
0.6551
1.2239
0.9206
0.0492
0.1231
Table 2: Overall results on behavior-conditioned animal vocalization generation.
Species
Macro Acc. ↑
FAD ↓
BioLi. FAD ↓
Hyena
0.9000 /0.5500
0.1622 /1.1566
0.0227 /0.1029
Meerkat
0.7883 /0.6967
0.2478 /1.5149
0.0155 /0.0739
Marmoset
0.9717 /0.6550
0.1147 /1.8304
0.0225 /0.2129
Goat
0.8300 /0.4050
0.1561 /0.7219
0.0273 /0.0974
Zebra
0.9650 /0.6550
0.5648 /1.1341
0.0520 /0.1151
Zebra Finch
0.8560 /0.7580
0.1470 /0.9853
0.0249 /0.1364
Table 3: Species-wise performance of TangoFlux Event-Aware. Each entry reports Real Reference / Event-Aware.
Figure 3: t-SNE of real and TangoFlux Event-Aware vocalizations in AnimalCLAP space.
Condition Acc. mAP Full .6199 .6551 Species-only .2261 .2427
Behavior swap Value Δptgt +.3139 Increase rate 85.39%
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.