BigMaQ: A Big Macaque Motion and Animation Dataset Bridging Image and 3D Pose Representations

Lucas Martini; Alexander Lappe; Anna Bognár; Rufin Vogels; Martin A. Giese

cs.CVFeb 23, 2026

BigMaQ: A Big Macaque Motion and Animation Dataset Bridging Image and 3D Pose Representations

Authors: Lucas Martini, Alexander Lappe, Anna Bognár, Rufin Vogels, Martin A. Giese

Organizations: Hertie Institute, University of Tübingen · IMPRS-IS · KU Leuven

Abstract

The recognition of dynamic and social behavior in animals is fundamental for advancing ethology, ecology, medicine and neuroscience. Recent progress in deep learning has enabled automated behavior recognition from video, yet an accurate reconstruction of the three-dimensional (3D) pose and shape has not been integrated into this process. Especially for non-human primates, mesh-based tracking efforts lag behind those for other species, leaving pose descriptions restricted to sparse keypoints that are unable to fully capture the richness of action dynamics. To address this gap, we introduce the $\textbf{Big Ma}$ ca $\textbf{Q}$ ue 3D Motion and Animation Dataset ( $\texttt{BigMaQ}$ ), a large-scale dataset comprising more than 750 scenes of interacting rhesus macaques with detailed 3D pose descriptions. Extending previous surface-based animal tracking methods, we construct subject-specific textured avatars by adapting a high-quality macaque template mesh to individual monkeys. This allows us to provide pose descriptions that are more accurate than previous state-of-the-art surface-based animal tracking methods. From the original dataset, we derive BigMaQ500, an action recognition benchmark that links surface-based pose vectors to single frames across multiple individual monkeys. By pairing features extracted from established image and video encoders with and without our pose descriptors, we demonstrate substantial improvements in mean average precision (mAP) when pose information is included. With these contributions, $\texttt{BigMaQ}$ establishes the first dataset that both integrates dynamic 3D pose-shape representations into the learning task of animal action recognition and provides a rich resource to advance the study of visual appearance, posture, and social interaction in non-human primates. The code and data are publicly available at https://martinivis.github.io/BigMaQ/ .

Figures & tables

Figure 1: BigMaQ : A motion capture dataset with surface-based modeling for non-human primate behavior recognition. The dataset provides colored avatars that can be rendered from arbitrary views, including dynamic action reconstructions (left), rich individual and social interactions (right).

Dataset	Species	Action Rec.	Video	Pose
				Type	3D-Data	Frame#
APT-36K ( Yang et al., 2022b )	G	✗	✓	2D	✗	36,000
AnimalKingdom ( Ng et al., 2022 )	G	✓	✓	2D	✗	33,099
LoTE-animal ( Liu et al., 2023 )	G	✓	✓	2D	✗	35,000
Animal3D ( Xu et al., 2023 )	G	✗	✗	3D-S	S	3,400
AniMer ( Lyu et al., 2025b )	G	✗	✗	3D-S	S	41,300

Table 1: Comparison of BigMaQ with existing pose estimation and action recognition datasets for NHPs, and animals in general. Species-specific datasets other than those containing non-human primates are not included. In the ”Species” column, G denotes general, P primates, C chimpanzees, and M macaques. The column ”Type” differentiates between 2D keypoint, 3D keypoint, and 3D-shape (3D-S) representations. The origin of this 3D data is further specified by S for synthetic and R for real recordings.

Figure 2: Distribution of the ethogram-aligned action category labels.

Figure 3: Pipeline overview for generating BigMaQ and the BigMaQ500 action–pose recognition benchmark. The 3D Labeling Tool provides annotations used to train the detection and keypoint models as well as to optimize subject-specific avatars. These optimized avatars are then combined with video-inferred labels to obtain dynamic pose reconstructions. BigMaQ500 includes all annotations available in BigMaQ , and additionally contains video encodings for more than 500 actions for which complete video-to-3D pose correspondences could be established.

Panel of Figure 4 — Figure 4: Qualitative comparison of different surface-tracking approaches for BigMaQ , illustrated for four actions of different individuals (rows). Only perspectives that were not used for multi-view optimization are shown. The second and third columns show the surface fits of BigMaQ with color texture (BigMaQ-C) and without (BigMaQ-M).

Action	IoU $\uparrow$		MPJPE $\downarrow$ [mm]		MPJTD $\downarrow$ [mm/frame]
Action	MAMMAL	Ours	MAMMAL	Ours	MAMMAL	Ours
Walk	0.771	0.883	23.493	20.402	9.961	6.875
Food Picking	0.756	0.855	25.521	16.489	13.362	9.062
Branch Shake	0.757	0.831	22.163	13.633	17.676	15.515
Scratch	0.722	0.843	28.843	20.481	8.498	4.422

Table 2: Mean sequence-level surface reconstruction metrics for the actions shown in Figure 4 . MPJPE is reported in mm and MPJTD in mm/frame. IoU scores are averaged over time and across cameras involved in the surface reconstruction.

	BigMaQ	MAMMAL	AniMer+
IoU	0.844	0.714	0.591

Table 3: Mean IoU across single-subject actions in BigMaQ , aggregating all viewpoints used in optimization for the first frame of the respective action.

Vision Model	Feat.	mAP	mAP ${}_{\text{L}}$	mAP ${}_{\text{OI}}$	mAP ${}_{\text{SI}}$	mAP ${}_{\text{O}}$
—	Pose	$43.5\pm 1.4$	$57.3\pm 2.1$	$\mathbf{54.4}\pm 1.3$	$28.7\pm 1.3$	$47.4\pm 4.0$
ResNet50	Vis	$34.3\pm 0.5$	$50.9\pm 1.7$	$38.6\pm 1.3$	$22.2\pm 0.9$	$35.9\pm 0.9$
	Vis+Pose	$\mathbf{44.0}\pm 0.8$	$58.1\pm 4.5$	$53.7\pm 1.0$	$28.8\pm 1.1$	$\mathbf{48.5}\pm 2.3$
ViT-base-cls	Vis	$32.9\pm 0.7$	$51.8\pm 1.0$	$37.4\pm 1.5$	$18.0\pm 0.9$	$35.9\pm 1.1$
	Vis+Pose	$\mathbf{44.0}\pm 0.1$	$60.6\pm 4.5$	$50.2\pm 3.0$	$\mathbf{29.9}\pm 3.5$	$47.1\pm 3.9$
DINOv2-base-cls	Vis	$37.8\pm 0.7$	$53.4\pm 0.8$	$49.9\pm 0.8$	$25.1\pm 1.2$	$37.8\pm 1.4$

Table 4: Comparison of action recognition based on pose representations

\bm{\theta}

and visual features including confidence intervals (mean

\pm

std) for multiple training runs: Including pose features improves performance across visual backbones. The overall mean average precision (mAP) is further broken down by the categories defined in our ethogram: the subscripts L, OI, SI, and O denote Locomotion, Object Interaction, Social Interaction, and Others, respectively.

Features	—	ViT-base-cls	DINOv2-base	VideoPrism-base
2D-KP	$35.6\pm 0.3$	$35.5\pm 0.4$	$40.2\pm 1.2$	$40.0\pm 0.6$
3D-KP	$40.8\pm 0.7$	$34.4\pm 1.1$	$34.5\pm 1.4$	$34.0\pm 2.0$
3D-M	$35.2\pm 4.4$	$34.4\pm 2.4$	$36.6\pm 0.8$	$36.0\pm 3.3$
3D-Rot	$\mathbf{43.5}\pm 1.4$	$\mathbf{44.0}\pm 0.1$	$\mathbf{41.4}\pm 1.7$	$\mathbf{43.8}\pm 2.9$

Table 5: Overall mAP scores for different pose representations in the BigMaQ action recognition task. Results are shown first for the pose-only stream, followed by ViT-base-cls, DINOv2-base, and VideoPrism visual features in combination with pose features. KP denotes keypoint positions in 2D and 3D, M the posed mesh’s vertex positions, and Rot the matrix form of

\bm{\theta}

that consistently outperforms other pose descriptors.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The recording setup of BigMaQ includes 16 cameras positioned around the monkey enclosure. The cameras were oriented toward the center of the space, specifically focusing the table area. Camera IDs are shown in black adjacent to each camera, with odd-numbered IDs indicating cameras providing a top-down view.

Panel of Figure 6 — Figure 6: Detection performance illustrated through the confusion matrix (top) and the precision–recall curve (bottom), demonstrating robust recognition across individuals.

Metric	Score
AP (all IoU)	0.9268
AP @ 0.5	0.9900
AP @ 0.75	0.9664
AR (all IoU)	0.9355
AR @ 0.5	0.9918
AR @ 0.75	0.9696

Appendix

Table 6: COCO evaluation metrics for HRNet-w48 at trained epoch 180. The model was trained with an 80%/10%/10% train/validation/test split. AP and AR denote average precision and average recall, respectively. The notation ‘ x’ indicates evaluation at an IoU threshold of x (e.g., [email protected] = average precision at IoU = 0.5), while ‘all IoU’ refers to averaging across thresholds from 0.5 to 0.95 in steps of 0.05.

Metric	Value
Precision (P)	0.9526
Recall (R)	0.9561
mAP @ 0.5	0.9762
mAP @ 0.5:0.95	0.8538

Appendix

Table 7: YOLOv8l model validation results at epoch 49. The model was trained with a 5% validation split for early stopping. mAP @ 0.5 indicates mean average precision at IoU = 0.5, while mAP @ 0.5:0.95 is the mean average precision averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05.

Figure 7: Predicted 2D keypoints of an exemplary frame.

Keypoint index	Keypoint name	Mean	Std
1	Right eye	0.431	0.753
2	Left eye	0.341	0.502
3	Nose	0.369	0.500
4	Neck	0.980	1.252
5	Right shoulder	0.937	1.280
6	Right elbow	0.956	1.234

Appendix

Table 8: Reprojection error statistics in pixels (px) of the 20 body keypoints in the 3D Labeling Tool across the 306 labeled monkey bodies.

# Cameras	$\sigma=1$ px	$\sigma=2$ px	$\sigma=5$ px	$\sigma=10$ px
2	1.440 $\pm$ 1.636	2.990 $\pm$ 5.374	7.329 $\pm$ 8.318	15.506 $\pm$ 31.656
3	1.007 $\pm$ 0.578	2.011 $\pm$ 1.219	5.015 $\pm$ 3.048	10.045 $\pm$ 5.768
4	0.818 $\pm$ 0.427	1.638 $\pm$ 0.841	4.087 $\pm$ 2.126	8.202 $\pm$ 4.216

Appendix

Table 9: Mean

\pm

std of the 3D Euclidean error (mm) of estimated 3D ground-truth points under different annotation noise levels and number of annotated cameras. Projected ground-truth points per annotation view were shifted randomly, and independently in

(x,y)

image dimensions for [

-\sigma

, 0,

\sigma

] pixels (px) before triangulation. Columns indicate the annotation noise

\sigma

; rows indicate the number of annotation cameras used for triangulation.

Figure 8: Box plots showing the distribution of action sequence lengths for BigMaQ and BigMaQ500 across the four behavioral categories. The filtering step does not substantially affect the action length distribution in the action recognition benchmark.

Figure 9: Side view of the high-poly mesh.

Figure 10: An overview of the individual mesh adaptations for all eight monkeys in BigMaQ .

Parameter	Min	Max	Mean	Std
Global Scale $\gamma$	0.791	1.001	0.908	0.060
Bone Lengths $\bm{\alpha}$	0.833	1.179	0.973	0.124
Vertex Offsets $\bm{\xi}$	-0.030	0.039	0.000	0.005
Vertex Colors $\mathbf{C}$	26.854	254.747	164.602	35.398

Appendix

Table 10: Summary statistics for adapted parameters in individual shape adaptation (min, max, mean, std).

	Pose	Mesh	Color	Time
$\lambda_{P}$	3	0	0	3
$\lambda_{b}$	100	100	0	0
$\lambda_{sm}$	0	5	0	0
$\lambda_{kp}$	2	1	0	1
$\lambda_{sil}$	1500	100	0	50
$\lambda_{C}$	0	0	1	0

Appendix

Table 11: A typical set of hyper-parameters for shape and dynamic pose estimation used for the BigMaQ dataset. The first three columns are involved in individual shape fitting, whereas the last column refers to the stage of dynamic pose fitting.

Panel of Figure 11 — Figure 11: Surface fitting results when disabling relevant $\lambda$ entries in dynamic pose estimation. The red outline indicates that this particular camera was not involved in surface fitting, ensuring an unbiased different viewpoint to evaluate reconstruction quality.

Ablation Setting	MPJPE $\downarrow$ (mm)	MPJTD $\downarrow$ (mm/frame)	IoU $\uparrow$
None (full model)	16.479	9.076	0.855
$\lambda_{\text{kp}}=0$ .	210.081	1.844	0.375
$\lambda_{\text{sil}}=0$ .	16.593	9.344	0.812
$\lambda_{T}=0$ .	17.082	10.620	0.853

Appendix

Table 12: Ablation experiment on weighting factors

\lambda

in dynamic optimization: mean error metrics for the entire time sequence of the action shown in Figure 11 . IoU scores are averaged over time and across cameras involved in the surface reconstruction. Arrows indicate whether lower or higher values are better.

Ablation Setting	MPJPE $\downarrow$ (mm)	MPJTD $\downarrow$ (mm/frame)	IoU $\uparrow$
None (full model)	16.479	9.076	0.855
w/o conf. values	18.676	9.756	0.847
$N_{\text{views}}=5$ .	19.052	9.971	0.848
$N_{\text{views}}=4$ .	29.322	11.085	0.822
$N_{\text{views}}=3$ .	26.286	11.138	0.825
$N_{\text{views}}=2$ .	36.053	11.902	0.801

Appendix

Table 13: Ablation experiment on confidence-value integration and viewpoint reduction in dynamic optimization: mean error metrics for the entire time sequence of the action shown in Figure 11 . IoU scores are averaged over time and across cameras of the full model (

N_{\text{views}}=6

). Arrows indicate whether lower or higher values are better.

Panel of Figure 12 — Figure 12: Failure cases of a frame in an action. The first and third column shows the rendered mesh on top of an image of the particular scene, including bounding box and keypoints labels, where the green frame outline indicates the camera’s participation in optimization. The associated SAM 2 silhouette is shown in the second and fourth column, respectively. Rows show results for different individuals.

Video Action Recognition

Video action recognition identifies and classifies human or animal activities by learning from visual appearance, motion, and temporal context.

See all 111 papers

Promptable Animal Pose Tracking Across Species
Aug 5, 2026Le Li, Daniela Ivanova, Nicolas PugeaultKeypoint Detection
School of Computing Science University of Glasgow
PRIMA: Boosting Animal Mesh Recovery with Taxonomic Priors and Adaptation-Driven Data Bootstrapping
Jun 1, 2026Xiaohang Yu, Ti Wang, Mackenzie Weygandt MathisTest-Time Adaptation 3D Reconstruction
´Ecole Polytechnique F´ed´erale de Lausanne (EPFL)
UniMo: Unifying Human and Animal Motion Generation
Sep 11, 2026Zeyu Zhang, Zhiyuan Zhang, Siheng Wang +4Text-to-Motion Generation Human Motion Generation
Westlake University, China · The Australian National University, Australia

Aug 5, 2026J/K move · Enter open · S save

Promptable Animal Pose Tracking Across Species

Le Li, Daniela Ivanova, Nicolas Pugeault

School of Computing Science University of Glasgow

Keypoint Detection

Open paper AskPDF arXiv

Animal pose estimation and tracking is important for wildlife monitoring and conservation research, and with limited expert time for labelling automated approaches are imperative. While human pose estimation and tracking has seen rapid progress thanks to large annotated datasets, animal pose remain challenging, due to large morphological and behavioural differences between species and limited annotated data. Existing approaches either optimise generic keypoint localisation from annotated datasets (such as APTv2) with poor generalisation, or track custom keypoints using visual tracking, at the cost of performance. In this paper, we demonstrate that vision foundation models trained on large datasets can be used effectively to track animal pose with limited labelled data. We propose two models, one unsupervised and the other supervised, to track user-selected keypoints in videos. The supervised approach delivers superior tracking accuracy by employing a keypoint prompt encoder to explicitly inject structural priors from a reference frame into feature matching. In parallel, the unsupervised route provides strong cross-species robustness by leveraging diverse foundation-model features for training-free correspondence matching. Extensive evaluation on challenging animal video benchmarks APTv2 and TigDog demonstrates that our framework achieves strong performance while maintaining an effective balance between accuracy and generalisation, offering a practical solution for real-world animal behaviour analysis and conservation applications.

Abstract

Video Action Recognition

Explore similar work

Promptable Animal Pose Tracking Across Species

PRIMA: Boosting Animal Mesh Recovery with Taxonomic Priors and Adaptation-Driven Data Bootstrapping

UniMo: Unifying Human and Animal Motion Generation