Cat Royale Comprehensive Dataset: A Dataset for Pose Detection, Segmentation, Animal Action Recognition and Scene Understanding
Authors: Zane K. J. Hartley, Joseph Ho, Alan Chamberlain, Michael P. Pound, Steve Benford
Organizations: School of Computer Science, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, Nottinghamshire, United Kingdom.
Animal focused computer vision datasets are of a growing interest in the fields of both Computer Vision and Animal behaviour. The ability to make sense of complex scenes via a variety of Computer Vision tasks has substantial implications for behavioural understanding and welfare. The perfection of these tasks will help lead to better health and emotional outcomes for wild and domestic animals. In this work we present the Cat Royale Comprehensive Dataset (CRCD), a large, comprehensive, multi-modal dataset for feline understanding, based on a huge database of over 500 hours of video captured during the Cat Royale artwork and research project. Our dataset contains annotations for a range of computer vision problems including 20,000 images with instance segmentation masks, 5000 images with multi-subject pose coordinates, over 800 video clips with action classes, and over 800 clips with detailed text description. This is the first such dataset to link this wide range of annotation types together for multi-modal use. We release this entire dataset, and our custom annotation tools will also be made available, in order to allow for ongoing expansion. We also provide a series of benchmarks for the tasks of Instance Segmentation, Pose detection, Temporal Action Localisation, and finally video-text retrieval. The totality of this multi-modal dataset is the data needed to solve both these individual computer vision tasks as well as enabling more complex multi-modal tasks making use of a combination of annotations.
Figures & tables
Figure 1: Four example frames from the CRCD dataset, shown in a 2×2 grid. The images illustrate key tasks: (a) Segmentation of animal boundaries, (b) Pose Estimation with skeletal keypoints, (c) Action Recognition based on behavioural labels, and (d) an excerpt from a Video Caption frame summarizing the scene.
Dataset Subset
Annotation Type
Total Items
CRCD-seg
Instance Segmentation Masks
20,000 Images
CRCD-pos
18-Point Pose Coordinates
5,000 Images
CRCD-act
Ethogram-based Action Classes
813 Video Clips
CRCD-cap
Detailed Text Descriptions
802 Video Clips
Table 1: Overview and Composition of the Cat Royale Comprehensive Dataset (CRCD)
Figure 2: Examples from CRCD-pos with keypoints overlaid over each cat. Keypoint colour distinguishes individual cats in each frame.
Figure 3: Examples from CRCD-seg with segmentation masks overlaid over each cat. Mask colour distinguishes individual cats in each frame.
Figure 4: Examples of the 12 most common actions from CRCD-act with categories and bounding boxes superimposed over the image.
ID
Action Class
Total Instances
1
Resting (lying down/loaf)
365
2
Staring
336
3
Exploring the area or objects
327
4
Resting (sitting)
142
5
Object play
61
6
Eating
56
Table 2: Distribution of Ethogram-based Action Classes in CRCD-act
Model
AP mask
AP 50mask
AP 75mask
Params (M)
Mask R-CNN
0.854
0.973
0.944
42.9
YOLOv8s-seg
0.890
0.980
—
11.2
Table 3: Benchmark Results for Instance Segmentation .
Model
mAP
AP 50
Params (M)
Keypoint RCNN
0.64
0.82
59.1
Yolov8-Pose
0.53
0.95
26.2
Table 4: Benchmark Results for Cat Pose Estimation using Keypoint R-CNN (ResNet-50 FPN backbone) and Yolov8-Pose .
Animal pose estimation and tracking is important for wildlife monitoring and conservation research, and with limited expert time for labelling automated approaches are imperative. While human pose estimation and tracking has seen rapid progress thanks to large annotated datasets, animal pose remain challenging, due to large morphological and behavioural differences between species and limited annotated data. Existing approaches either optimise generic keypoint localisation from annotated datasets (such as APTv2) with poor generalisation, or track custom keypoints using visual tracking, at the cost of performance. In this paper, we demonstrate that vision foundation models trained on large datasets can be used effectively to track animal pose with limited labelled data. We propose two models, one unsupervised and the other supervised, to track user-selected keypoints in videos. The supervised approach delivers superior tracking accuracy by employing a keypoint prompt encoder to explicitly inject structural priors from a reference frame into feature matching. In parallel, the unsupervised route provides strong cross-species robustness by leveraging diverse foundation-model features for training-free correspondence matching. Extensive evaluation on challenging animal video benchmarks APTv2 and TigDog demonstrates that our framework achieves strong performance while maintaining an effective balance between accuracy and generalisation, offering a practical solution for real-world animal behaviour analysis and conservation applications.
Deciphering animal intent is a fundamental challenge in computational ethology, largely because of semantic aliasing, the phenomenon where identical external signals (e.g., a cat's purr) correspond to radically different internal states depending on physiological context. Existing Multimodal Large Language Models (MLLMs) are blind to high-frequency biological time-series data, restricting them to superficial behavioural pattern matching rather than genuine latent-state reasoning. To bridge this gap, we introduce Meow-Omni 1, the first open-source, quad-modal MLLM purpose-built for computational ethology. It natively fuses video, audio, and physiological time-series streams with textual reasoning. Through targeted architectural adaptation, we integrate specialized scientific encoders into a unified backbone and formalize intent inference via physiologically grounded cross-modal alignment. Evaluated on MeowBench, a novel, expert-verified quad-modal benchmark, Meow-Omni 1 achieves state-of-the-art intent-recognition accuracy (71.16%), substantially outperforming leading vision-language and omni-modal baselines. We release the complete open-source pipeline including model weights, training framework, and the Meow-10K dataset, to establish a scalable paradigm for inter-species intent understanding and to advance foundation models toward real-world veterinary diagnostics and wildlife conservation.
Jucheng Hu, Zhangquan Chen, Yulin Chen +9
University College London · Tsinghua University · Nanjing University +1
We present LAION-BVD (LAION - Big Video Dataset), a large-scale open video dataset for multimodal learning, which contains 1.3B platform-specific video URLs collected from CommonCrawl. From these, we download 80M videos with a total duration of 13.5 million hours. The dataset is designed for multimodal pre-training across the video, audio, and image modalities. Using content-aware scene detection, we extract clips for which we synthetically generate video and audio captions. Models trained on these data achieve competitive performance on standard video-text and audio-text benchmarks, with consistent improvements as training or model scale increases. Additionally, we explore video frames as an alternative source of image-text data by extracting scene-changing frames. These frames exhibit a visual distribution distinct from standard web image corpora, and models trained on this dataset achieve strong image-text retrieval performance. We release LAION-BVD to the research community. It significantly expands open access to multimodal videos at an unprecedented scale.
Andreas Hochlehnert, Marianna Nezhurina, Mehdi Cherti +9
University of Tübingen, Tübingen AI Center · LAION · JSC, FZJ +3