BigMaQ: A Big Macaque Motion and Animation Dataset Bridging Image and 3D Pose Representations

Lucas Martini; Alexander Lappe; Anna Bognár; Rufin Vogels; Martin A. Giese

cs.CVFeb 23, 2026

BigMaQ: A Big Macaque Motion and Animation Dataset Bridging Image and 3D Pose Representations

Authors: Lucas Martini, Alexander Lappe, Anna Bognár, Rufin Vogels, Martin A. Giese

Organizations: Hertie Institute, University of Tübingen · IMPRS-IS · KU Leuven

Abstract

The recognition of dynamic and social behavior in animals is fundamental for advancing ethology, ecology, medicine and neuroscience. Recent progress in deep learning has enabled automated behavior recognition from video, yet an accurate reconstruction of the three-dimensional (3D) pose and shape has not been integrated into this process. Especially for non-human primates, mesh-based tracking efforts lag behind those for other species, leaving pose descriptions restricted to sparse keypoints that are unable to fully capture the richness of action dynamics. To address this gap, we introduce the $\textbf{Big Ma}$ ca $\textbf{Q}$ ue 3D Motion and Animation Dataset ( $\texttt{BigMaQ}$ ), a large-scale dataset comprising more than 750 scenes of interacting rhesus macaques with detailed 3D pose descriptions. Extending previous surface-based animal tracking methods, we construct subject-specific textured avatars by adapting a high-quality macaque template mesh to individual monkeys. This allows us to provide pose descriptions that are more accurate than previous state-of-the-art surface-based animal tracking methods. From the original dataset, we derive BigMaQ500, an action recognition benchmark that links surface-based pose vectors to single frames across multiple individual monkeys. By pairing features extracted from established image and video encoders with and without our pose descriptors, we demonstrate substantial improvements in mean average precision (mAP) when pose information is included. With these contributions, $\texttt{BigMaQ}$ establishes the first dataset that both integrates dynamic 3D pose-shape representations into the learning task of animal action recognition and provides a rich resource to advance the study of visual appearance, posture, and social interaction in non-human primates. The code and data are publicly available at https://martinivis.github.io/BigMaQ/ .

Figures & tables

Figure 1: BigMaQ : A motion capture dataset with surface-based modeling for non-human primate behavior recognition. The dataset provides colored avatars that can be rendered from arbitrary views, including dynamic action reconstructions (left), rich individual and social interactions (right).

Dataset	Species	Action Rec.	Video	Pose
				Type	3D-Data	Frame#
APT-36K ( Yang et al., 2022b )	G	✗	✓	2D	✗	36,000
AnimalKingdom ( Ng et al., 2022 )	G	✓	✓	2D	✗	33,099
LoTE-animal ( Liu et al., 2023 )	G	✓	✓	2D	✗	35,000
Animal3D ( Xu et al., 2023 )	G	✗	✗	3D-S	S	3,400
AniMer ( Lyu et al., 2025b )	G	✗	✗	3D-S	S	41,300

Table 1: Comparison of BigMaQ with existing pose estimation and action recognition datasets for NHPs, and animals in general. Species-specific datasets other than those containing non-human primates are not included. In the ”Species” column, G denotes general, P primates, C chimpanzees, and M macaques. The column ”Type” differentiates between 2D keypoint, 3D keypoint, and 3D-shape (3D-S) representations. The origin of this 3D data is further specified by S for synthetic and R for real recordings.

Figure 2: Distribution of the ethogram-aligned action category labels.

Figure 3: Pipeline overview for generating BigMaQ and the BigMaQ500 action–pose recognition benchmark. The 3D Labeling Tool provides annotations used to train the detection and keypoint models as well as to optimize subject-specific avatars. These optimized avatars are then combined with video-inferred labels to obtain dynamic pose reconstructions. BigMaQ500 includes all annotations available in BigMaQ , and additionally contains video encodings for more than 500 actions for which complete video-to-3D pose correspondences could be established.

Panel of Figure 4 — Figure 4: Qualitative comparison of different surface-tracking approaches for BigMaQ , illustrated for four actions of different individuals (rows). Only perspectives that were not used for multi-view optimization are shown. The second and third columns show the surface fits of BigMaQ with color texture (BigMaQ-C) and without (BigMaQ-M).

Action	IoU $\uparrow$		MPJPE $\downarrow$ [mm]		MPJTD $\downarrow$ [mm/frame]
Action	MAMMAL	Ours	MAMMAL	Ours	MAMMAL	Ours
Walk	0.771	0.883	23.493	20.402	9.961	6.875
Food Picking	0.756	0.855	25.521	16.489	13.362	9.062
Branch Shake	0.757	0.831	22.163	13.633	17.676	15.515
Scratch	0.722	0.843	28.843	20.481	8.498	4.422

Table 2: Mean sequence-level surface reconstruction metrics for the actions shown in Figure 4 . MPJPE is reported in mm and MPJTD in mm/frame. IoU scores are averaged over time and across cameras involved in the surface reconstruction.

	BigMaQ	MAMMAL	AniMer+
IoU	0.844	0.714	0.591

Table 3: Mean IoU across single-subject actions in BigMaQ , aggregating all viewpoints used in optimization for the first frame of the respective action.

Vision Model	Feat.	mAP	mAP ${}_{\text{L}}$	mAP ${}_{\text{OI}}$	mAP ${}_{\text{SI}}$	mAP ${}_{\text{O}}$
—	Pose	$43.5\pm 1.4$	$57.3\pm 2.1$	$\mathbf{54.4}\pm 1.3$	$28.7\pm 1.3$	$47.4\pm 4.0$
ResNet50	Vis	$34.3\pm 0.5$	$50.9\pm 1.7$	$38.6\pm 1.3$	$22.2\pm 0.9$	$35.9\pm 0.9$
	Vis+Pose	$\mathbf{44.0}\pm 0.8$	$58.1\pm 4.5$	$53.7\pm 1.0$	$28.8\pm 1.1$	$\mathbf{48.5}\pm 2.3$
ViT-base-cls	Vis	$32.9\pm 0.7$	$51.8\pm 1.0$	$37.4\pm 1.5$	$18.0\pm 0.9$	$35.9\pm 1.1$
	Vis+Pose	$\mathbf{44.0}\pm 0.1$	$60.6\pm 4.5$	$50.2\pm 3.0$	$\mathbf{29.9}\pm 3.5$	$47.1\pm 3.9$
DINOv2-base-cls	Vis	$37.8\pm 0.7$	$53.4\pm 0.8$	$49.9\pm 0.8$	$25.1\pm 1.2$	$37.8\pm 1.4$

Table 4: Comparison of action recognition based on pose representations

\bm{\theta}

and visual features including confidence intervals (mean

\pm

std) for multiple training runs: Including pose features improves performance across visual backbones. The overall mean average precision (mAP) is further broken down by the categories defined in our ethogram: the subscripts L, OI, SI, and O denote Locomotion, Object Interaction, Social Interaction, and Others, respectively.

Features	—	ViT-base-cls	DINOv2-base	VideoPrism-base
2D-KP	$35.6\pm 0.3$	$35.5\pm 0.4$	$40.2\pm 1.2$	$40.0\pm 0.6$
3D-KP	$40.8\pm 0.7$	$34.4\pm 1.1$	$34.5\pm 1.4$	$34.0\pm 2.0$
3D-M	$35.2\pm 4.4$	$34.4\pm 2.4$	$36.6\pm 0.8$	$36.0\pm 3.3$
3D-Rot	$\mathbf{43.5}\pm 1.4$	$\mathbf{44.0}\pm 0.1$	$\mathbf{41.4}\pm 1.7$	$\mathbf{43.8}\pm 2.9$

Table 5: Overall mAP scores for different pose representations in the BigMaQ action recognition task. Results are shown first for the pose-only stream, followed by ViT-base-cls, DINOv2-base, and VideoPrism visual features in combination with pose features. KP denotes keypoint positions in 2D and 3D, M the posed mesh’s vertex positions, and Rot the matrix form of

\bm{\theta}

that consistently outperforms other pose descriptors.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The recording setup of BigMaQ includes 16 cameras positioned around the monkey enclosure. The cameras were oriented toward the center of the space, specifically focusing the table area. Camera IDs are shown in black adjacent to each camera, with odd-numbered IDs indicating cameras providing a top-down view.

Panel of Figure 6 — Figure 6: Detection performance illustrated through the confusion matrix (top) and the precision–recall curve (bottom), demonstrating robust recognition across individuals.

Metric	Score
AP (all IoU)	0.9268
AP @ 0.5	0.9900
AP @ 0.75	0.9664
AR (all IoU)	0.9355
AR @ 0.5	0.9918
AR @ 0.75	0.9696

Appendix

Table 6: COCO evaluation metrics for HRNet-w48 at trained epoch 180. The model was trained with an 80%/10%/10% train/validation/test split. AP and AR denote average precision and average recall, respectively. The notation ‘ x’ indicates evaluation at an IoU threshold of x (e.g., [email protected] = average precision at IoU = 0.5), while ‘all IoU’ refers to averaging across thresholds from 0.5 to 0.95 in steps of 0.05.

Metric	Value
Precision (P)	0.9526
Recall (R)	0.9561
mAP @ 0.5	0.9762
mAP @ 0.5:0.95	0.8538

Appendix

Table 7: YOLOv8l model validation results at epoch 49. The model was trained with a 5% validation split for early stopping. mAP @ 0.5 indicates mean average precision at IoU = 0.5, while mAP @ 0.5:0.95 is the mean average precision averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05.

Figure 7: Predicted 2D keypoints of an exemplary frame.

Keypoint index	Keypoint name	Mean	Std
1	Right eye	0.431	0.753
2	Left eye	0.341	0.502
3	Nose	0.369	0.500
4	Neck	0.980	1.252
5	Right shoulder	0.937	1.280
6	Right elbow	0.956	1.234

Appendix

Table 8: Reprojection error statistics in pixels (px) of the 20 body keypoints in the 3D Labeling Tool across the 306 labeled monkey bodies.

# Cameras	$\sigma=1$ px	$\sigma=2$ px	$\sigma=5$ px	$\sigma=10$ px
2	1.440 $\pm$ 1.636	2.990 $\pm$ 5.374	7.329 $\pm$ 8.318	15.506 $\pm$ 31.656
3	1.007 $\pm$ 0.578	2.011 $\pm$ 1.219	5.015 $\pm$ 3.048	10.045 $\pm$ 5.768
4	0.818 $\pm$ 0.427	1.638 $\pm$ 0.841	4.087 $\pm$ 2.126	8.202 $\pm$ 4.216

Appendix

Table 9: Mean

\pm

std of the 3D Euclidean error (mm) of estimated 3D ground-truth points under different annotation noise levels and number of annotated cameras. Projected ground-truth points per annotation view were shifted randomly, and independently in

(x,y)

image dimensions for [

-\sigma

, 0,

\sigma

] pixels (px) before triangulation. Columns indicate the annotation noise

\sigma

; rows indicate the number of annotation cameras used for triangulation.

Figure 8: Box plots showing the distribution of action sequence lengths for BigMaQ and BigMaQ500 across the four behavioral categories. The filtering step does not substantially affect the action length distribution in the action recognition benchmark.

Figure 9: Side view of the high-poly mesh.

Figure 10: An overview of the individual mesh adaptations for all eight monkeys in BigMaQ .

Parameter	Min	Max	Mean	Std
Global Scale $\gamma$	0.791	1.001	0.908	0.060
Bone Lengths $\bm{\alpha}$	0.833	1.179	0.973	0.124
Vertex Offsets $\bm{\xi}$	-0.030	0.039	0.000	0.005
Vertex Colors $\mathbf{C}$	26.854	254.747	164.602	35.398

Appendix

Table 10: Summary statistics for adapted parameters in individual shape adaptation (min, max, mean, std).

	Pose	Mesh	Color	Time
$\lambda_{P}$	3	0	0	3
$\lambda_{b}$	100	100	0	0
$\lambda_{sm}$	0	5	0	0
$\lambda_{kp}$	2	1	0	1
$\lambda_{sil}$	1500	100	0	50
$\lambda_{C}$	0	0	1	0

Appendix

Table 11: A typical set of hyper-parameters for shape and dynamic pose estimation used for the BigMaQ dataset. The first three columns are involved in individual shape fitting, whereas the last column refers to the stage of dynamic pose fitting.

Panel of Figure 11 — Figure 11: Surface fitting results when disabling relevant $\lambda$ entries in dynamic pose estimation. The red outline indicates that this particular camera was not involved in surface fitting, ensuring an unbiased different viewpoint to evaluate reconstruction quality.

Ablation Setting	MPJPE $\downarrow$ (mm)	MPJTD $\downarrow$ (mm/frame)	IoU $\uparrow$
None (full model)	16.479	9.076	0.855
$\lambda_{\text{kp}}=0$ .	210.081	1.844	0.375
$\lambda_{\text{sil}}=0$ .	16.593	9.344	0.812
$\lambda_{T}=0$ .	17.082	10.620	0.853

Appendix

Table 12: Ablation experiment on weighting factors

\lambda

in dynamic optimization: mean error metrics for the entire time sequence of the action shown in Figure 11 . IoU scores are averaged over time and across cameras involved in the surface reconstruction. Arrows indicate whether lower or higher values are better.

Ablation Setting	MPJPE $\downarrow$ (mm)	MPJTD $\downarrow$ (mm/frame)	IoU $\uparrow$
None (full model)	16.479	9.076	0.855
w/o conf. values	18.676	9.756	0.847
$N_{\text{views}}=5$ .	19.052	9.971	0.848
$N_{\text{views}}=4$ .	29.322	11.085	0.822
$N_{\text{views}}=3$ .	26.286	11.138	0.825
$N_{\text{views}}=2$ .	36.053	11.902	0.801

Appendix

Table 13: Ablation experiment on confidence-value integration and viewpoint reduction in dynamic optimization: mean error metrics for the entire time sequence of the action shown in Figure 11 . IoU scores are averaged over time and across cameras of the full model (

N_{\text{views}}=6

). Arrows indicate whether lower or higher values are better.

Panel of Figure 12 — Figure 12: Failure cases of a frame in an action. The first and third column shows the rendered mesh on top of an image of the particular scene, including bounding box and keypoints labels, where the green frame outline indicates the camera’s participation in optimization. The associated SAM 2 silhouette is shown in the second and fourth column, respectively. Rows show results for different individuals.

Video Action Recognition

Video action recognition identifies and classifies human or animal activities by learning from visual appearance, motion, and temporal context.

See all 111 papers

Aug 5, 2026

Promptable Animal Pose Tracking Across Species

Animal pose estimation and tracking is important for wildlife monitoring and conservation research, and with limited expert time for labelling automated approaches are imperative. While human pose estimation and tracking has seen rapid progress thanks to large annotated datasets, animal pose remain challenging, due to large morphological and behavioural differences between species and limited annotated data. Existing approaches either optimise generic keypoint localisation from annotated datasets (such as APTv2) with poor generalisation, or track custom keypoints using visual tracking, at the cost of performance. In this paper, we demonstrate that vision foundation models trained on large datasets can be used effectively to track animal pose with limited labelled data. We propose two models, one unsupervised and the other supervised, to track user-selected keypoints in videos. The supervised approach delivers superior tracking accuracy by employing a keypoint prompt encoder to explicitly inject structural priors from a reference frame into feature matching. In parallel, the unsupervised route provides strong cross-species robustness by leveraging diverse foundation-model features for training-free correspondence matching. Extensive evaluation on challenging animal video benchmarks APTv2 and TigDog demonstrates that our framework achieves strong performance while maintaining an effective balance between accuracy and generalisation, offering a practical solution for real-world animal behaviour analysis and conservation applications.

Le Li, Daniela Ivanova, Nicolas Pugeault

School of Computing Science University of Glasgow

Keypoint Detection

Ask PDF

Jun 1, 2026

PRIMA: Boosting Animal Mesh Recovery with Taxonomic Priors and Adaptation-Driven Data Bootstrapping

Recovering articulated 3D animal meshes from monocular images is challenging due to the long-tailed distribution of species and limited large-scale 3D supervision. Existing methods rely on small annotated datasets and thus regress to average shapes and poses, leading to poor generalization to rare species and articulations. We present PRIMA (PRIors for Mesh Adaptation), a framework that tackles these limitations with two components: taxonomic shape priors and adaptation-driven data bootstrapping. First, we derive taxonomic shape priors from BioCLIP image features, which are learned under taxonomic supervision, to inject species-level knowledge into mesh reconstruction, improving shape estimation across diverse quadrupeds. Second, we propose an adaptation-driven bootstrapping scheme that turns abundant 2D animal annotations into reliable pseudo-3D supervision via lightweight instance-level mesh refinement and quality-aware filtering. This yields Quadruped3D, a large-scale pseudo-3D dataset constructed from real-world animal images, containing approximately 9K pseudo-3D annotations. Together, these components alleviate the long-tail problem: extensive experiments on Animal3D, CtrlAni3D, Quadruped2D, and Animal Kingdom demonstrate state-of-the-art performance for mesh reconstruction, with particularly strong gains on challenging poses and underrepresented species. The code is available at https://github.com/AdaptiveMotorControlLab/PRIMA.

Xiaohang Yu, Ti Wang, Mackenzie Weygandt Mathis

´Ecole Polytechnique F´ed´erale de Lausanne (EPFL)

Test-Time Adaptation 3D Reconstruction 3D Mesh Reconstruction

Ask PDF

Sep 11, 2026

UniMo: Unifying Human and Animal Motion Generation

The conditional generation of 3D motion has emerged as a key research topic due to its wide applicability across robotics, AR/VR, gaming, and content creation. However, extending recent advances in text-driven human motion generation to the animal domain remains challenging due to two core limitations. First, animals exhibit highly diverse skeletal topologies, unlike the standard human structure, making unified modeling across species difficult and leading to inefficient per-species models. Second, existing animal motion datasets suffer from limited scale and annotation quality, constraining model performance. To address these challenges, we propose UniMo, a unified point cloud-based motion generation framework that bypasses topological discrepancies by converting parametric skeletons into unparametric representations, further enhanced by dynamic sampling that allocates more points to active joints. Additionally, we present UniML3D, a large-scale motion-language dataset spanning both human and animal categories, containing 145,907 motion sequences and 433,388 captions-over 102x larger than existing animal datasets. Our method achieves state-of-the-art results on UniML3D and three public benchmarks including HumanML3D, KIT-ML, and AnimalML3D, demonstrating the feasibility and effectiveness of unified human-animal motion generation. Website: https://steve-zeyu-zhang.github.io/UniMo.

Zeyu Zhang, Zhiyuan Zhang, Siheng Wang +4

Westlake University, China · The Australian National University, Australia

Text-to-Motion Generation Human Motion Generation Motion Representation Learning

Ask PDF

Abstract

Video Action Recognition

Explore similar work

Promptable Animal Pose Tracking Across Species

PRIMA: Boosting Animal Mesh Recovery with Taxonomic Priors and Adaptation-Driven Data Bootstrapping

UniMo: Unifying Human and Animal Motion Generation