Audio-Visual Dog Identification and Tracking in Real-World Videos

Aryahvishwa Babu; Kingsley Jin; Lillian Wang; Tuan M. Dang; Theron S. Wang; Jeorge Joshi; Annie Cao; Kenny Q. Zhu

cs.CVOct 14, 2026

Audio-Visual Dog Identification and Tracking in Real-World Videos

Authors: Aryahvishwa Babu, Kingsley Jin, Lillian Wang, Tuan M. Dang, Theron S. Wang, Jeorge Joshi, Annie Cao, Kenny Q. Zhu

Organizations: University of Maryland, College Park · University of California, Merced · Cornell University · University of Texas at Arlington ·

Abstract

Identifying individual dogs from barks in real-world videos remains challenging and largely unexplored. Solving this problem would enable long-term behavioral analysis, veterinary monitoring, and animal welfare monitoring by attributing vocalizations and behaviors to the correct individual. Real-world videos are challenging because multiple dogs may bark simultaneously, become occluded, or move in and out of view. Prior studies show that canine vocalizations and visual appearance contain discriminative identity information, but these modalities have largely been studied independently. We present a multimodal dog-fingerprinting framework combining visual identification, multiple-object tracking, and canine vocal fingerprinting to associate bark events with individual dogs while maintaining persistent profiles. The framework synchronizes bark events with tracked dogs and updates profiles using visual appearance and vocal characteristics, enabling identification despite occlusion, overlapping vocalizations, and off-screen barking. We also introduce a benchmark of 46 real-world videos containing 213 manually annotated bark events. Our framework correctly localizes the active visible dog in 61.83% of visible events at IoU $\geq 0.30$ and achieves a 91.33% F1 score for visible versus off-screen barking detection.

Figures & tables

Panel of Figure 1 — Figure 1: Challenges in dog identification under real-world conditions.

Dataset	Use	Scale	Samples
Automated visual set	Barker classifier: train	1,690 videos	12,738 windows
Tracking benchmark	Visual tracking: test	21 videos	3,890 frames / 15,407 boxes
Multimodal benchmark	Visual/fusion: test	46 usable videos	747 windows / 213 events
YouTube Channel Benchmark	Audio fingerprint: train	426 dogs	415,361 clips
DogSpeak	Open-set audio: test	150 dogs	65,263 clips

Table 1: Datasets used to train and evaluate each framework component.

Figure 2: Overview of the proposed multimodal dog-fingerprinting framework. Visual tracking and active-barker classification are combined with open-set vocal fingerprinting to maintain persistent dog identities.

Tracker	HOTA $\uparrow$	MOTA $\uparrow$	DetA $\uparrow$	AssA $\uparrow$	IDF1 $\uparrow$	FP $\downarrow$	FN $\downarrow$	IDs $\downarrow$
BoT-SORT	42.61	26.78	40.14	47.10	40.52	3,447	5,718	75
ByteTrack	41.44	27.11	40.16	44.68	36.25	3,367	5,758	74
SORT	39.21	25.59	38.14	42.09	34.12	3,051	6,265	75
StableTracker	46.85	27.34	40.38	56.44	46.42	3,368	5,768	34
StableTracker + Profile Merging (Ours)	49.81	25.17	38.42	66.71	51.11	3,221	6,211	11

Table 2: Tracking performance on the manually annotated benchmark. Best results are shown in bold.

Method	Window Acc. $\uparrow$	Window AUC $\uparrow$	Track Acc. $\uparrow$	Track AUC $\uparrow$	Multi-dog Top-1 $\uparrow$
Zero-shot ImageBind	45.4	0.487	43.7	0.472	47.8
ImageBind + LR	56.8	0.572	59.2	0.588	53.3
8-frame mean	58.6	0.594	61.9	0.626	73.3
12-frame mean	58.8	0.581	61.9	0.619	65.6
Temporal statistics	58.1	0.561	61.2	0.590	65.6
Temporal attention	50.5	0.507	50.3	0.556	50.0

Table 3: Visual barking-classification results on the manually annotated benchmark. Multi-dog top-1 accuracy uses 90 multi-candidate events.

Method	Features	Acc. $\uparrow$	F1 $\uparrow$	EER $\downarrow$	minDCF $\downarrow$	DIR@1% $\uparrow$
MFCC	MFCC	15.73	6.81	37.90	0.0100	0.33
Dog2vec	Dog2vec	28.10	11.99	26.09	0.0095	5.00
Feature Fusion	MFCC + Dog2vec	28.25	12.24	27.24	0.0095	4.89
ResNet-TSP	80-d log-Mel	56.89	32.71	14.33	0.0073	22.30
Ours	Hybrid embedding	65.36	37.14	12.47	0.0063	29.48

Table 4: Audio-representation comparison on DogSpeak using five enrollment barks. Accuracy, macro F1, EER, and DIR are percentages; DIR@1% is measured at a 1% false-accept rate.

Barks	Acc. $\uparrow$	F1 $\uparrow$	EER $\downarrow$	minDCF $\downarrow$	DIR@1% $\uparrow$
1	40.69	21.13	20.38	0.0084	11.49
3	59.23	32.60	14.32	0.0069	23.59
5	65.36	37.14	12.47	0.0063	29.48

Table 5: Effect of enrollment size on DogSpeak. Accuracy, macro F1, EER, and DIR are percentages; DIR@1% is measured at a 1% false-accept rate.

Metric	Result
Correct localization at IoU $\geq 0.30$	61.83%
Mean spatial IoU	0.589
Visible/off-screen precision	86.89%
Visible/off-screen recall	96.24%
Visible/off-screen F1	91.33%
Profile-assignment coverage	99.67%

Table 6: Performance of the complete multimodal framework.

Animal Re-Identification

Animal re-identification matches observations of the same individual across time, cameras, viewpoints, and changing conditions.

See all 33 papers

From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification
Feb 28, 2026Vasiliy Kudryavtsev, Kirill Borodin, German Berezin +3Animal Re-Identification Multimodal Fusion
Faculty of IT, Technical University of Communication and Informatics, Moscow, Russia, 111024 · AI lab, Avito, Moscow, Russia, 125196
Counting Without Numbers and Finding Without Words
Mar 25, 2026Badri Narayana PatroAnimal Re-Identification Audio-Visual Understanding
Microsoft
Uncertainty-Aware Crossmodal Fusion for Classification of Animal Behavior
Aug 3, 2026Ehsan Yaghoubi, Florian HaselbeckUncertainty-Aware Multimodal Fusion Bioacoustics
Weihenstephan-Triesdorf University of Applied Sciences, Department of Sustainable Agricultural and Energy Systems, Freising, Am Staudengarten 1, 85354, Germany

Feb 28, 2026J/K move · Enter open · S save

From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification

Vasiliy Kudryavtsev, Kirill Borodin, German Berezin, Kirill Bubenchikov+2

Faculty of IT, Technical University of Communication and Informatics, Moscow, Russia, 111024 · AI lab, Avito, Moscow, Russia, 125196

Animal Re-Identification Multimodal Fusion Cross-Modal Representation Learning

Open paper AskPDF arXiv

Automated animal identification is a practical task for reuniting lost pets with their owners, yet current systems often struggle due to limited dataset scale and reliance on unimodal visual cues. This study introduces a multimodal verification framework that enhances visual features with semantic identity priors derived from synthetic textual descriptions. We constructed a massive training corpus of 1.9 million photographs covering 695,091~unique animals to support this investigation. Through systematic ablation studies, we identified SigLIP2-Giant and E5-Small-v2 as the optimal vision and text backbones. We further evaluated fusion strategies ranging from simple concatenation to adaptive gating to determine the best method for integrating these modalities. Our proposed approach utilizes a gated fusion mechanism and achieved a Top-1 accuracy of 84.28% and an Equal Error Rate of 0.0422 on a comprehensive test protocol. These results represent an 11% improvement over leading unimodal baselines and demonstrate that integrating synthesized semantic descriptions significantly refines decision boundaries in large-scale pet re-identification.\\ Conference update. This work was accepted to the FGVC13 Workshop at CVPR 2026. We thank Vadim Ezhov and Nikolai Shcheglov for their help with the poster presentation and demonstration at the conference.

Abstract

Animal Re-Identification

Explore similar work

From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification

Counting Without Numbers and Finding Without Words

Uncertainty-Aware Crossmodal Fusion for Classification of Animal Behavior