Cat Royale Comprehensive Dataset: A Dataset for Pose Detection, Segmentation, Animal Action Recognition and Scene Understanding
Authors: Zane K. J. Hartley, Joseph Ho, Alan Chamberlain, Michael P. Pound, Steve Benford
Organizations: School of Computer Science, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, Nottinghamshire, United Kingdom.
Animal focused computer vision datasets are of a growing interest in the fields of both Computer Vision and Animal behaviour. The ability to make sense of complex scenes via a variety of Computer Vision tasks has substantial implications for behavioural understanding and welfare. The perfection of these tasks will help lead to better health and emotional outcomes for wild and domestic animals. In this work we present the Cat Royale Comprehensive Dataset (CRCD), a large, comprehensive, multi-modal dataset for feline understanding, based on a huge database of over 500 hours of video captured during the Cat Royale artwork and research project. Our dataset contains annotations for a range of computer vision problems including 20,000 images with instance segmentation masks, 5000 images with multi-subject pose coordinates, over 800 video clips with action classes, and over 800 clips with detailed text description. This is the first such dataset to link this wide range of annotation types together for multi-modal use. We release this entire dataset, and our custom annotation tools will also be made available, in order to allow for ongoing expansion. We also provide a series of benchmarks for the tasks of Instance Segmentation, Pose detection, Temporal Action Localisation, and finally video-text retrieval. The totality of this multi-modal dataset is the data needed to solve both these individual computer vision tasks as well as enabling more complex multi-modal tasks making use of a combination of annotations.
Figures & tables
Figure 1: Four example frames from the CRCD dataset, shown in a 2×2 grid. The images illustrate key tasks: (a) Segmentation of animal boundaries, (b) Pose Estimation with skeletal keypoints, (c) Action Recognition based on behavioural labels, and (d) an excerpt from a Video Caption frame summarizing the scene.
Dataset Subset
Annotation Type
Total Items
CRCD-seg
Instance Segmentation Masks
20,000 Images
CRCD-pos
18-Point Pose Coordinates
5,000 Images
CRCD-act
Ethogram-based Action Classes
813 Video Clips
CRCD-cap
Detailed Text Descriptions
802 Video Clips
Table 1: Overview and Composition of the Cat Royale Comprehensive Dataset (CRCD)
Figure 2: Examples from CRCD-pos with keypoints overlaid over each cat. Keypoint colour distinguishes individual cats in each frame.
Figure 3: Examples from CRCD-seg with segmentation masks overlaid over each cat. Mask colour distinguishes individual cats in each frame.
Figure 4: Examples of the 12 most common actions from CRCD-act with categories and bounding boxes superimposed over the image.
ID
Action Class
Total Instances
1
Resting (lying down/loaf)
365
2
Staring
336
3
Exploring the area or objects
327
4
Resting (sitting)
142
5
Object play
61
6
Eating
56
Table 2: Distribution of Ethogram-based Action Classes in CRCD-act
Model
AP mask
AP 50mask
AP 75mask
Params (M)
Mask R-CNN
0.854
0.973
0.944
42.9
YOLOv8s-seg
0.890
0.980
—
11.2
Table 3: Benchmark Results for Instance Segmentation .
Model
mAP
AP 50
Params (M)
Keypoint RCNN
0.64
0.82
59.1
Yolov8-Pose
0.53
0.95
26.2
Table 4: Benchmark Results for Cat Pose Estimation using Keypoint R-CNN (ResNet-50 FPN backbone) and Yolov8-Pose .