cs.SDOct 28, 2026

Who Called? V33DA: A Physically Verified Multimodal Benchmark for Vocal Attribution in Zebra Finch Groups

Authors: Maris Basha, Yuhang Wang, Xiaoran Chen, Longbiao Cheng, Luca Yapura, Anja T. Zai, Mathieu Salzmann, Richard Hahnloser

Organizations: Institute of Neuroinformatics, University of Zurich and ETH Zurich · Swiss Data Science Center and CVLab, EPFL

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deciding which member of an animal group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3D candidate geometry. We close this gap with V33DA, a dataset-centered multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an on-body reference ceiling. Given a detected vocalization and the visible candidate birds, the task is to select the caller. We evaluate it under three regimes: session-disjoint testing on familiar birds from held-out recording dates, held-out-experiment transfer to a new cohort, and leave-one-individual-out evaluation on a bird whose vocalizations are held out from training. Reference methods expose a consistent gap between in-domain accuracy and transfer to unseen individuals: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The on-body reference stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels. We release V33DA with fixed protocols and code for the evaluated methods, and V33DA++, an unevaluated companion set of verified overlapping-call events and +/-2 s context windows.

Figures & tables

Figure 1: V33DA overview and representation geometry. a. BirdPark rendering with calibrated microphones, camera poses, the 4-antenna FM receiver array, and triangulated bird positions. b. Synchronized multi-view frame with tracked individuals. c. Candidate-centered crop; the neon-green ring marks the vocalizing bird’s backpack. d. Accelerometer-derived vibration spectrograms; left color bars mark the blue, peach, and red birds. e. Per-antenna (A–D) radio telemetry—baseband signal power, relative phase, and compensation phase—over the event, with the same color bars. f. Five-channel microphone spectrograms. Panels a–f show the same verified event across geometry, video, on-body sensing, radio telemetry, and multichannel audio. g. UMAP of withheld on-body reference embeddings: the reference cleanly separates the caller from silent birds. h. SELDnet++ embeddings organized primarily by bird identity. i. SpatialScorer embeddings organized primarily by vocalizer position. Panels g–i preview the main result: transferable attribution depends on spatial organization more than on identity recognition.
 DomainTaskSizeNConc.MicsVis.Verif.
Physically verified animal caller attribution and localization
Anisimov+ ( 2014 )Zebra finchCAmethods demo991/bird–on-body accel.
Gill+ ( 2016 )Zebra finchCA182k calls881/bird–body-mounted mic
Rookognise ( 2022 )RookCA + ID17.7k vocs15156–obs. + MC audio
MeerKAT ( 2025 )MeerkatSSL pre{}_{\text{pre}} + classif.251k events41–1/collar–body-mounted mic
Prat+ ( 2017 )Fruit batrepertoire90k events332–81/chamb.✓mouth on IR video
Table 1: Comparison of V33DA with related datasets. Task abbreviations are defined in the text (classif. = classification, diar. = diarization, verif. = speaker verification). N : number of individuals; Conc. : individuals recorded concurrently; Mics : microphones; Vis. : synchronized video ( ∘\circ : in-the-wild video without calibrated geometry); Verif. : source of the caller label (on-body accel. = accelerometer on the animal; body-mounted mic = microphone on the animal; perceptual = human annotation); vocs = vocalizations, utt. = utterances, segs = segments.
RegimeHeld out at testHeld fixedWhat it isolates
Session-disjointRecording dateBirds, cage, arrayIn-domain session noise
Held-out experimentBird cohort + acq. contextNo birds sharedIdentity + context shift
Leave-one-bird-outPositive caller examplesCage, array, cohort, datesHeld-out vocalizer
Table 2: Evaluation regimes in order of increasing difficulty. All three regimes evaluate the same task (candidate-conditioned caller attribution); they differ in which axes of variation are held out at test time.
     LOO mean
MethodSessionHeld-out BP01Held-out BP02Held-out BP05BaseΔ\Delta audΔ\Delta geoΔ\Delta rad
SRP ±\pm 20 (physics)51.052.462.235.853.5 ±\pm 27.5–––
SELDnet++ (slot)86.5 ±\pm 0.439.8 ±\pm 0.844.0 ±\pm 1.236.9 ±\pm 3.50.0 ±\pm 0.0+0.0+0.0+0.0+0.0+0.0+0.0
SpatialScorer73.0 ±\pm 1.256.7 ±\pm 1.669.1 ±\pm 1.055.0 ±\pm 1.249.5 ±\pm 22.0−26.2-26.2−25.2-25.2+6.3+6.3
ACCDOA65.6 ±\pm 0.258.1 ±\pm 2.063.4 ±\pm 1.338.6 ±\pm 0.955.4 ±\pm 21.8−27.4-27.4−29.0-29.0+0.0+0.0
EINV265.8 ±\pm 0.358.2 ±\pm 1.966.6 ±\pm 1.438.2 ±\pm 0.355.2 ±\pm 22.1−26.4-26.4−26.9-26.9+0.0+0.0
Table 3: Vocal attribution and LOO modality interventions across benchmark regimes (representative subset) using V33DA. Attribution accuracy (%) for one representative method per family; the full 24-method matrix is in Appendix Table 18 . The last three columns are LOO accuracy changes (points) when one input channel is removed at test time on a single seed-42 checkpoint: zeroing the audio ( Δ\Delta aud), shuffling candidate geometry across events ( Δ\Delta geo), or zeroing the radio features ( Δ\Delta rad). Negative values indicate degradation. Session tests new recordings of known birds; Held-out BP01/BP02/BP05 rotate the test experiment; LOO mean averages within-experiment leave-one-bird-out folds. Learned rows report mean ±\pm std across seeds. The withheld on-body reference is a ceiling.
SettingMethodSessionHeld-out BP01Held-out BP02Held-out BP05LOO mean
FreeEINV220.7 ±\pm 0.623.9 ±\pm 0.821.7 ±\pm 0.427.1 ±\pm 0.725.0 ±\pm 9.6
 CST-former21.8 ±\pm 0.422.8 ±\pm 0.725.8 ±\pm 0.426.9 ±\pm 0.523.2 ±\pm 5.0
Cand.Random candidate27.526.927.128.427.1 ±\pm 1.3
 SpatialScorer18.2 ±\pm 0.117.4 ±\pm 0.119.2 ±\pm 0.224.2 ±\pm 0.319.7 ±\pm 5.5
 BeamFusion11.7 ±\pm 0.717.1 ±\pm 1.617.1 ±\pm 1.120.5 ±\pm 0.623.7 ±\pm 6.7
Ref.On-body ref.0.3 ±\pm 0.02.9 ±\pm 0.40.8 ±\pm 0.21.4 ±\pm 0.13.3 ±\pm 3.1
Table 4: Localization across benchmark regimes (representative subset) using V33DA. Mean error in cm (lower is better). Setting : Free = unconstrained 3D regression; Cand. = candidate-conditioned (one of the visible birds). The full table is in Appendix Table 23 . The random-candidate row picks one visible candidate uniformly at random; the shaded withheld on-body reference row is a ceiling for the candidate-conditioned setting only.
Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

 Audio tasksMultimodalVideo tasksRadio 
 SSLSepVADIDCAASDKPPoseTrackRadTransfer
Anisimov+, Gill+ [ 2014 , 2016 ]–✓––✓––––––
Rookognise [ 2022 ]––✓✓✓––––––
MeerKAT [ 2025 ]––✓––––––––
Prat+ [ 2017 ]––––✓––––––
HyVL [ 2023 ]✓✓✓–✓–––✓––
Appendix
Table 5: Task coverage of V33DA versus related resources. A checkmark indicates that the released data directly support the task. In the V33DA row, a star ( ⋆\star ) marks tasks evaluated in this paper, and a dagger ( †\dagger ) marks tasks for which the unevaluated V33DA++ companion release provides data.
ExperimentDaysH5 FilesRecording Hours
juvExpBP0110∼\sim 400∼\sim 46
juvExpBP023∼\sim 120∼\sim 14
juvExpBP053∼\sim 120∼\sim 14
Appendix
Table 6: Raw data volumes per experiment.
ExperimentClean patchesAfter 48 ×\times augAccuracy
juvExpBP01∼{\sim} 2,000∼{\sim} 96,00098.1%
juvExpBP02∼{\sim} 1,050∼{\sim} 50,40095.2%
juvExpBP05∼{\sim} 1,040∼{\sim} 49,92097.3%
Average  96.9%
Appendix
Table 7: K-NN backpack color classification accuracy per experiment.
Figure 2: Verification interface used during dataset curation. (a) Experiment selector. (b) Synchronized video frame with overlaid keypoints and color-coded identities; the selected vocalizer is highlighted by a green ring. (c) Reviewer controls for navigating events, selecting the vocalizer identity, and assigning the review verdict; the crop strip shows the currently visible tracks. (d) Per-bird accelerometer spectrograms; the selected bird shows the vocalization-associated vibration signature. (e) Five-channel microphone spectrograms aligned to the same event.
MetricBP01BP02BP05
Samples16,21911,5775,829
Reproj error (median, px)7.92.512.3
3D displacement (median, mm)2.92.52.5
2D disp top (median, px)4.04.34.6
2D disp back (median, px)4.43.53.6
Appendix
Table 9: Quality metrics after filtering.
Table 11
Table 12
FamilyModelAudio3DHeadRadioVideoAccelAttribution
PhysicsLoudestMic✓✓––––Nearest to loudest mic
 SRP ±20\pm 20✓✓––––Nearest to SRP peak
 SRP Reranker✓✓✓✓––Learned reranker
 NeuralSRP✓✓–(opt)––Candidate scorer
Slot-indexedSELDnet✓–––––4-way slot head
Candidate- centricSELDnet++✓✓✓✓––Hybrid direct + cand
Appendix
Table 14: Baseline modality usage and attribution strategy. Audio : 5-channel microphone array. 3D : triangulated bird beak position. Head : head orientation vector. Radio : per-bird FM radio telemetry (summary and/or temporal). Video : synchronized RGB crop tubes. Accel : on-body accelerometer (withheld reference only). A checkmark indicates the modality is consumed at inference; (opt) indicates the model has a variant with and without that input.
ParameterValue
Audio sample rate24,414.0625 Hz
Video frame rate47.6837158203125 Hz (= 390625/8192)
Samples per video frame512
Speed of sound343,000 mm/s (20 ∘ C)
Cage dimensions (X ×\times Y ×\times Z)1246 ×\times 896 ×\times 1294 mm
Cage bounds X (left–right)[−90.8,1155.2][-90.8,1155.2] mm
Appendix
Table 15: Recording system parameters and physical constants.
Figure 3: Validation accuracy per epoch for matrix-family baselines on the three held-out-experiment splits. Lines are seed means, shaded bands are ±1\pm 1 standard deviation across the three training seeds.
MethodEvaluation aggregateSubmitted60-epoch
CST-formerLOO15.019.9±5.719.9\pm 5.7
CST-formerHeld-out exp. mean30.033.7
Whisper-candidateLOO12.111.7±1.911.7\pm 1.9
Whisper-candidateHeld-out exp. mean31.430.3
ACCDOALOO55.456.7
ACCDOASession-disjoint65.667.1
Appendix
Table 16: Attribution accuracy (%) with the submitted budget and with 60 epochs. Held-out exp. mean averages BP01, BP02, and BP05.
FamilyArchitectureLRWDEpochsBatchSeedsLabel smooth
SELDnetCRNN of Adavanne et al. [2018]2×10−42{\times}10^{-4}5×10−35{\times}10^{-3}606430.1
SRP rerankerours1×10−31{\times}10^{-3}1×10−21{\times}10^{-2}4012850.05
NeuralSRPSALSA-Lite input [ Nguyen et al., 2022 ]2×10−42{\times}10^{-4}5×10−35{\times}10^{-3}83230.1
PairRankours2×10−42{\times}10^{-4}5×10−35{\times}10^{-3}83230.03
SpatialScorerours2×10−42{\times}10^{-4}5×10−35{\times}10^{-3}81630.1
Contrastiveours2×10−42{\times}10^{-4}5×10−35{\times}10^{-3}81630.1
Appendix
Table 17: Shared hyperparameters across learned baselines. Architecture : source of the model design (ours = introduced in this work). Optimizer settings are shared defaults and were not tuned per method.
MethodSessionHeld-out BP01Held-out BP02Held-out BP05LOO mean
Random28.3 ±\pm 0.524.8 ±\pm 0.233.3 ±\pm 0.624.6 ±\pm 0.327.0 ±\pm 4.6
LoudestMic31.118.939.526.327.3 ±\pm 11.2
SRP ±\pm 2051.052.462.235.853.5 ±\pm 27.5
SELDnet81.8 ±\pm 1.234.3 ±\pm 0.438.6 ±\pm 0.241.7 ±\pm 8.60.0 ±\pm 0.0
SELDnet++86.5 ±\pm 0.439.8 ±\pm 0.844.0 ±\pm 1.236.9 ±\pm 3.50.0 ±\pm 0.0
SELDnet++ open61.5 ±\pm 0.434.6 ±\pm 1.445.4 ±\pm 0.029.2 ±\pm 1.29.3 ±\pm 4.8
Appendix
Table 18: Full vocal attribution matrix across benchmark regimes. All methods solve the same task on the released events: given a synchronized multimodal window, pick the vocalizing bird from the set of candidates visible at that moment. Entries are raw attribution accuracy (%). Session tests new recordings of known birds. Held-out BP01/BP02/BP05 rotate the test experiment and measure transfer across experiments and individuals. LOO mean averages within-experiment leave-one-bird-out folds and isolates identity shift under a fixed sensing setup. Learned rows report mean ±\pm standard deviation; deterministic rows are shown without ±\pm . The withheld on-body reference is included only as a ceiling.
SplitConditionACCDOAEINV2SpatialScorerCST-former
Held-out BP01Base58.4/56.7/23.257.9/56.1/24.258.5/57.1/17.028.4/26.6/22.0
Δ\Delta zero audio-27.9/-27.2/+9.2-24.5/-23.8/+3.8-35.9/-35.4/+15.3+4.1/+4.2/+5.9
Δ\Delta shuffle geometry-33.7/-32.5/+0.0-33.1/-31.9/+0.0-33.5/-32.5/+0.0-0.7/-0.7/+0.0
Δ\Delta zero radio+0.0/+0.0/+0.0+0.0/+0.0/+0.0+1.2/+1.4/+0.0+5.1/+5.8/+0.0
Held-out BP02Base61.2/59.4/22.766.6/65.2/21.866.3/65.7/19.836.3/35.3/25.8
Δ\Delta zero audio-25.2/-24.3/+10.2-29.8/-29.7/+11.5-27.1/-28.8/+14.2-2.7/-2.3/+3.8
Appendix
Table 19: Full modality interventions. Base rows report accuracy / legacy fold-wise macro-F1 / localization error in cm ; delta rows report changes from base in points / points / cm . The macro-F1 entries are retained only as intervention diagnostics within the same split and should not be read as the LOO class-balanced benchmark summary. Positive localization deltas indicate worse error. Interventions are evaluated on a single seed-42 checkpoint, so base accuracies may differ slightly from the multi-seed means reported in Table 18 while remaining within the reported seed variance.
BirdTest nnSRP ±\pm 20SpatialScorerACCDOAEINV2ContrastiveVideoCandidateOn-body ref.AudioANN-locfirst
BP01-0396375.761.9 ±\pm 3.367.4 ±\pm 0.567.8 ±\pm 0.746.7 ±\pm 1.330.6 ±\pm 0.498.6 ±\pm 0.263.3 ±\pm 0.4
BP01-1227731.860.9 ±\pm 2.665.0 ±\pm 1.666.3 ±\pm 0.858.7 ±\pm 2.746.9 ±\pm 0.193.8 ±\pm 0.856.8 ±\pm 0.8
BP01-2424471.262.9 ±\pm 0.673.5 ±\pm 0.473.9 ±\pm 0.863.4 ±\pm 1.326.5 ±\pm 0.286.8 ±\pm 2.572.5 ±\pm 0.3
BP01-3573530.571.1 ±\pm 1.268.6 ±\pm 1.269.1 ±\pm 0.855.1 ±\pm 3.238.2 ±\pm 1.098.1 ±\pm 0.569.3 ±\pm 1.1
BP02-0206588.840.7 ±\pm 1.171.1 ±\pm 1.071.0 ±\pm 1.635.8 ±\pm 1.214.2 ±\pm 0.397.2 ±\pm 0.462.8 ±\pm 1.7
BP02-1458961.063.8 ±\pm 1.162.3 ±\pm 0.760.7 ±\pm 0.828.9 ±\pm 0.527.8 ±\pm 1.579.1 ±\pm 9.559.6 ±\pm 2.1
Appendix
Table 20: Per-bird leave-one-bird-out accuracy (%). Birds are rows and representative completed methods are columns. Test nn is the number of held-out test events in that fold. For learned methods, entries report mean ±\pm standard deviation across seeds within that held-out bird fold.
Methodρ(acc,ntest)\rho(\mathrm{acc},n_{\mathrm{test}})ρ(acc,dNN)\rho(\mathrm{acc},d_{\mathrm{NN}})
SRP ±20\pm 20-0.30- 0.45
SpatialScorer- 0.94-0.23
ACCDOA- 0.44-0.11
EINV2- 0.44-0.11
AudioANN-locfirst- 0.65-0.27
On-body ref.-0.06-0.20
Appendix
Table 21: Simple predictors of LOO fold difficulty. Spearman correlation ρ\rho across the 10 held-out-bird folds between fold-wise LOO accuracy, test-set size, and the held-out bird’s mean nearest-neighbor distance.
MethodUniform / per-bird recallSample-weighted
Random27.027.9
SRP ±20\pm 2053.552.9
SELDnet0 0.00 0.0
SpatialScorer49.558.3
NeuralSRP19.222.8
BeamFusion27.630.5
Appendix
Table 22: LOO accuracy / per-bird recall, uniform vs. sample-weighted across the 10 held-out-bird folds. Uniform columns match Table 3 and are also the per-bird recall average. Weighted columns use the test-set sizes in Table 11 . Both columns average seeds within each fold before aggregating.
MethodSessionHeld-out BP01Held-out BP02Held-out BP05LOO mean
Free 3D localization
ACCDOA22.7 ±\pm 0.424.0 ±\pm 1.422.2 ±\pm 0.428.6 ±\pm 0.825.1 ±\pm 7.5
EINV220.7 ±\pm 0.623.9 ±\pm 0.821.7 ±\pm 0.427.1 ±\pm 0.725.0 ±\pm 9.6
CST-former21.8 ±\pm 0.422.8 ±\pm 0.725.8 ±\pm 0.426.9 ±\pm 0.523.2 ±\pm 5.0
AudioANN-locfirst21.7 ±\pm 0.725.0 ±\pm 0.427.2 ±\pm 2.727.2 ±\pm 0.724.6 ±\pm 8.0
RadioANN-locfirst41.1 ±\pm 0.627.2 ±\pm 0.149.4 ±\pm 1.849.1 ±\pm 2.234.9 ±\pm 6.7
Appendix
Table 23: Localization results (mean error in cm, lower is better). Free 3D localization and candidate-conditioned localization are reported separately. The withheld on-body reference is a ceiling only for the candidate-conditioned setting.
Table 24
ComparisonΔ\Delta accpppHolmp_{\mathrm{Holm}}
SpatialScorer −- SRP ±20\pm 20-0.0070.946041.00000
ACCDOA −- SRP ±20\pm 20+0.0220.454831.00000
EINV2 −- SRP ±20\pm 20+0.0220.375731.00000
AudioANN-locfirst −- SRP ±20\pm 20-0.0000.497311.00000
CST-former −- SRP ±20\pm 20-0.3430.000490.00391
RadioANN-locfirst −- SRP ±20\pm 20-0.2310.004640.03247
Appendix
Table 26: Selected paired transfer significance tests for attribution accuracy. Positive Δ\Delta acc means that the row method improves mean fold-wise transfer accuracy over the column method.
RegimeMethodCleanσ=1\sigma{=}1 cmσ=5\sigma{=}5 cmσ=10\sigma{=}10 cmΔ10\Delta_{10}
SessionSpatialScorer71.871.568.362.1-9.7
Contrastive72.071.969.964.7-7.3
BeamFusion67.967.363.055.1-12.8
ACCDOA64.564.461.956.2-8.3
EINV265.265.162.656.5-8.7
NeuralSRP43.843.843.743.6-0.2
Appendix
Table 27: Mean attribution accuracy under test-time perturbation of candidate 3D position. Rows are grouped by evaluation regime. Δ10\Delta_{10} is the drop at σ=10\sigma{=}10 cm relative to the clean column. Orientation perturbations (not shown) moved every cell by <0.1<0.1 pt.
Nearest competitorACCDOARandomWithheld on-body reference
<10<10 cm46.626.391.2
10–20 cm54.226.791.1
20–30 cm61.027.891.2
30–50 cm67.328.990.8
>50>50 cm73.031.190.2
Appendix
Table 28: LOO attribution accuracy (%) by distance from the vocalizer to its nearest competing candidate.
Front endSessionHeld-out exp. meanLOO
Random candidate28.327.627.0
Mel spectrogram75.2±0.575.2\pm 0.549.1±0.549.1\pm 0.523.8±0.223.8\pm 0.2
AVES (self-supervised)75.4±0.775.4\pm 0.749.3±1.049.3\pm 1.024.1±0.524.1\pm 0.5
BirdNET (supervised)75.3±0.875.3\pm 0.850.0±0.450.0\pm 0.422.8±0.522.8\pm 0.5
animal2vec (self-supervised)74.3±0.574.3\pm 0.550.0±1.450.0\pm 1.422.8±0.622.8\pm 0.6
Appendix
Table 29: Frozen audio front ends under a fixed candidate-scoring head (accuracy, %). The held-out column averages BP01, BP02, and BP05.
More in this topic

Bioacoustics

Bioacoustics uses sound to study animal activity, communication, biodiversity, and ecological change.

See all 65 papers

Explore similar work

CardsList

  1. Time-frequency localization of bird calls in dense soundscapes

    Jun 9, 2026Simen Hexeberg, Fanghui Tong, Hari Vishnu +1Sound Event DetectionObject Detection

    Tropical Marine Science Institute, National University of Singapore, Singapore · Dept. of Electrical and Computer Engineering, National University of Singapore, Singapore · School of Computing, National University of Singapore, Singapore

  2. Trajectory Variance: An Unsupervised Measure of Developmental Vocal Plasticity in Birdsong

    Jul 3, 2026Kanghwi LeeBioacoustics

    Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland

  3. MyGardenBird: A Machine-Learning-Ready Bird Sound Dataset for Twelve Common Malaysian Birds

    Jun 5, 2026Muhammad Mun'im Ahmad Zabidi, Mohd Yamani Idna Idris, Norisma IdrisTraining Data CurationBioacoustics

    Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia · Faculty of Electrical Engineering, Universiti Teknologi Malaysia, 81310 Johor Bahru, Johor, Malaysia