cs.CVSep 28, 2026

VastMAT: A Large-Scale Multi-Category Benchmark for Multi-Animal Tracking

Authors: Zhizhen Li, Zan Wang, Huidong Peng, Bohan Tan, Shimin Shan, Yu Liu, Liang Peng

Organizations: Dalian University of Technology · Wuhan University · University of North Texas · The Hong Kong University of Science and Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-animal tracking (MAT) supports the study of animal movement, behavior, and group interactions. However, general multi-object tracking (MOT) benchmarks primarily focus on pedestrians and vehicles, whereas dedicated MAT benchmarks remain limited in jointly supporting broad animal coverage, large-scale video data, and extensive within-video multi-instance association. To address this gap, we introduce VastMAT, which has four key characteristics: (1) Large scale. It comprises 2,947 videos with 1,002,562 annotated frames, totaling 27.85 hours. (2) Broad category coverage. These videos cover 337 animal categories with diverse morphologies and motion patterns. (3) Extensive instance annotations. It provides 3,663,248 bounding boxes and 22,883 identity trajectories---to our knowledge, the largest numbers of both among dedicated MAT benchmarks. (4) High-quality annotations. To ensure reliability, annotations undergo iterative expert review and correction, and quality is assessed through an independent reannotation audit. To systematically assess tracking performance and cross-category generalization, we establish Seen-category and category-disjoint Unseen-category protocols, and evaluate eight representative MOT methods under both protocols. Under these protocols, the highest baseline HOTA scores are 66.37% and 52.90%, respectively, highlighting the challenge of tracking unseen animals. To address the low-overlap association challenge revealed by our analysis, we propose Center-Distance-Augmented Association (CDA), a lightweight module that adaptively combines IoU with center similarity normalized by the boxes' own scales. Without additional training, CDA improves TrackTrack's HOTA by 1.58 and 1.31 percentage points under the two protocols, respectively. To facilitate further MAT research, we will publicly release our benchmark and code.

Figures & tables

Figure 1: Category and video counts of VastMAT and representative tracking datasets.
Figure 2: Annotated examples across animal groups and scenes in VastMAT.
Figure 3: Motion and box-geometry distributions across tracking datasets.
Figure 4: Scoring non-overlapping candidates with (a) IoU and (b) CDA.
Figure 5: Qualitative comparison of eight trackers on Protocol 1.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative comparison of eight baseline trackers on selected VastMAT sequences.
Figure 7: Video-level distributions of (a) trajectory count, (b) average active instances per frame, (c) duration, and (d) category count in VastMAT.
Figure 8: Per-category distributions of videos, trajectories, and bounding boxes in VastMAT.
Figure 9: All 337 categories in VastMAT grouped into six major animal groups, with inner-ring proportions and outer-ring category labels.
Figure 10: Category statistics in VastMAT: (a) composition of the 20 most frequent categories and (b) connected components in the category co-occurrence graph.
Figure 11: Representative visually similar category pairs in VastMAT: (a) Silver Dollar vs. Piranha, (b) Grebe vs. Duck, (c) Moorish Idol vs. Butterflyfish, (d) Goldfish vs. Cichlid (Parrot Cichlid), (e) Yellow Sailfin Tang vs. Tangs (Yellow Tang), and (f) Guppy vs. Molly.
Figure 12: Empirical CDF of per-video HOTA for all methods under Protocol 1 (left) and Protocol 2 (right). The curves show the fraction of videos below a given HOTA threshold.
Figure 13: Fusion-weight sensitivity of TrackTrack and DiffMOT.

Explore similar work

CardsList
Jul 1, 2026cs.CV

HieDG: A Hierarchical Discrete Geometry-Guided Framework for Multi-Animal Tracking

Multi-animal tracking (MAT) is critical for wildlife monitoring and behavioral analysis, yet remains challenging due to uniform appearance, high density, and irregular motion. Existing methods typically follow heuristic- or query-based paradigms: the former relies on handcrafted geometric associations without end-to-end optimization, whereas the latter enables joint optimization but relies heavily on appearance embeddings. In such conditions, continuous geometric embeddings can be unstable, as small coordinate perturbations may disproportionately alter cross-frame attention weights, degrading identity association performance. To address this limitation, we propose HieDG, a Hierarchical Discrete Geometry-guided tracking framework that reformulates geometric dynamics as structured discrete representations within a query-based tracker. Instead of directly using raw geometric signals, HieDG employs a two-stage residual codebook to discretize position, scale, and velocity cues, transforming unstable continuous geometry into structured, stable discrete tokens. These tokens are aligned with visual embeddings and integrated into the tracking queries to enhance identity consistency. Extensive experiments on animal-specific benchmarks (AnimalTrack, BFT, and BuckTales) demonstrate state-of-the-art association performance with significant improvements in HOTA, AssA, and IDF1. Additional evaluations on generic multi-object tracking benchmarks, including DanceTrack and SportsMOT, show competitive performance, indicating the broader applicability of discretized geometric modeling beyond animal-specific scenarios.

Chenxun Deng, Zhongde Zhang, Ye Yuan +7

State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Technology, Beijing Forestry University +6

Multi-Object TrackingSpherical Latent SpaceHierarchical
AskPDF
Mar 25, 2026cs.CV

COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm

Multi-Object Tracking (MOT) has traditionally focused on a few specific categories, restricting its applicability to real-world scenarios involving diverse objects. Open-Vocabulary Multi-Object Tracking (OVMOT) addresses this by enabling tracking of arbitrary categories, including novel objects unseen during training. However, current progress is constrained by two challenges: the lack of continuously annotated video data for training, and the lack of a customized OVMOT framework to synergistically handle detection and association. We address the data bottleneck by constructing C-TAO, the first continuously annotated training set for OVMOT, which increases annotation density by 26x over the original TAO and captures smooth motion dynamics and intermediate object states. For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules: (1) Multi-Cue Adaptive Fusion (MCF) dynamically balances appearance, motion, and semantic cues for association feature learning; (2) Multi-Granularity Hierarchical Aggregation (MGA) exploits hierarchical spatial relationships in dense detections, where visible child nodes (e.g., object parts) assist occluded parent objects (e.g., whole body) for association feature enhancement; (3) Temporal Confidence Propagation (TCP) recovers flickering detections through high-confidence tracked objects boosting low-confidence candidates across frames, stabilizing trajectories. Extensive experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods, and show strong zero-shot generalization on BDD100K.

Zekun Qian, Wei Feng, Ruize Han +1

School of Computer Science and Technology, Tianjin University, Tianjin, China · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology

Multi-Object TrackingOpen-Vocabulary Object DetectionIntermediate States
AskPDF
Aug 5, 2026cs.CV

Promptable Animal Pose Tracking Across Species

Animal pose estimation and tracking is important for wildlife monitoring and conservation research, and with limited expert time for labelling automated approaches are imperative. While human pose estimation and tracking has seen rapid progress thanks to large annotated datasets, animal pose remain challenging, due to large morphological and behavioural differences between species and limited annotated data. Existing approaches either optimise generic keypoint localisation from annotated datasets (such as APTv2) with poor generalisation, or track custom keypoints using visual tracking, at the cost of performance. In this paper, we demonstrate that vision foundation models trained on large datasets can be used effectively to track animal pose with limited labelled data. We propose two models, one unsupervised and the other supervised, to track user-selected keypoints in videos. The supervised approach delivers superior tracking accuracy by employing a keypoint prompt encoder to explicitly inject structural priors from a reference frame into feature matching. In parallel, the unsupervised route provides strong cross-species robustness by leveraging diverse foundation-model features for training-free correspondence matching. Extensive evaluation on challenging animal video benchmarks APTv2 and TigDog demonstrates that our framework achieves strong performance while maintaining an effective balance between accuracy and generalisation, offering a practical solution for real-world animal behaviour analysis and conservation applications.

Le Li, Daniela Ivanova, Nicolas Pugeault

School of Computing Science University of Glasgow

SpeciesKeypoint DetectionLong-Video Benchmarks
AskPDF