Photogrammetric 3D Model-Based Animal Behavior Classification
Authors: Olaf Hellwich, Moritz Heine, Patrik Reiske, Sole Traverso, Katharina Hohlbaum, Henning Sprekeler, Christa Thöne-Reineke, Lars Lewejohann, Roman Bhuiyan
Organizations: TU Berlin, Computer Vision & Remote Sensing · TU Berlin, Science of Intelligence Excellence Cluster · TU Berlin, Modeling of Cognitive Processes · German Centre for the Protection of Laboratory Animals (Bf3R), German Federal Institute for Risk Assessment (BfR) · FU Berlin, Institute of Animal Welfare, Animal Behavior and Laboratory Animal Science
Video-based behavior estimation is often implemented end-to-end from image sequences. We advocate a two-stage alternative: reconstruct 3D body and object geometry before classifying behavior. This makes ambiguities of the ill-posed 2D-to-3D inverse problem explicit and potentially correctable. We demonstrate the strategy with simulated interacting mice and extend it to real video.
Figures & tables
Fig. 1: Simulated skeleton and rendered video. Top, front and side view
Figure 2Figure 3
Model
Input source
Interaction F1
Rear0 F1
Rear1 F1
LSTM
Simulated 3D tracks
0.6276
0.8369
0.8316
Reconstructed 3D tracks
0.6147
0.8116
0.8066
xLSTM
Simulated 3D tracks
0.7146
0.7500
0.8866
Reconstructed 3D tracks
0.6702
0.6776
0.8293
Transformer
Simulated 3D tracks
0.7271
0.8611
0.8743
Reconstructed 3D tracks
0.7240
0.8286
0.8432
Table 1: F1 scores for behavior classification on the same three-minute simulated test sequence. The 3D classifiers were trained on simulated 3D body-part tracks and evaluated on simulated or cleaned, completed 3D-reconstructed tracks. The direct Transformer was trained on perfect multiview 2D projections of the same behavior-training trajectory, using the same temporal training/validation split, and evaluated on DLC 2D detections. Its inputs use pair-centered image coordinates, without segmentation, explicit 3D reconstruction, or additional identity correction. Rearing scores are evaluated on ground-truth non-interaction frames. Bold values denote the highest score for each behavior within each of the two 3D input-source groups.
Fig. 7: Mouse lock box experiment. The tail root in the front view is wrongly extracted.
Fig. 8: Camera calibration with the lock-box. The lock-box CAD model ( blue ), projected with the fitted camera parameters, covers the lock-box in the front, side, and top views.
Fig. 9: 3D reconstruction of the mouse. (a)–(c) Keypoints reconstructed in 3D from all three views and projected back into each image: head (nose, ears) ∙ , paws ∙ , tail root ∙ ; white lines connect neighboring keypoints of the skeleton. (d) A keypoint error revealed by geometry: DeepLabCut places the tail root at the red cross (likelihood 0.95) at the left image border. Side and top views agree on a tail-root position whose projection into the front view ( ∙ ) lies on the mouse, about 1,800 pixels away.
Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging. Supervised pose estimation requires extensive manual annotation, while general-purpose 3D reconstruction models trained on generic scene datasets fail on the specialized imagery and sparse-view setting of laboratory experiments. We address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video. BEAST3D uses a vision transformer to predict 3D Gaussian splats that reconstruct held-out views through differentiable rendering, while simultaneously segmenting the animal from the background. BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters--unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings. Through comprehensive evaluation across four species, we demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, which validates the quality of the learned 3D representations; multi-view pose estimation, which provides the sparse keypoint trajectories widely used in behavioral analysis; and neural encoding, which relates 3D behavioral features to simultaneously recorded neural activity. BEAST3D thus establishes a versatile framework for behavioral analysis that leverages 3D structure in modern multi-view laboratory recordings.
Yanchen Wang, Lenny Aharon, Wangshu Zhu +7
Columbia University · Cold Spring Harbor · Stanford University
A deeper understanding of brain function requires a precise, structured characterization of behavior. Yet, extracting behavioral representations from video in a form suitable for scientific analysis remains a fundamental challenge. Many prior studies represent behavior via pose estimation or nonlinear video embeddings. However, pose tracking discards rich information beyond predefined keypoints, while nonlinear video embeddings lack interpretability. We address this limitation with SABLE (Sparse-view Animal Behavior Latent Embeddings), a self-supervised framework that leverages a geometric inductive bias to learn behavior representations. By augmenting a multi-view transformer with priors from monocular depth and pose estimation, SABLE reconstructs 3D animal behavior from extremely sparse views while learning explicit 3D latent structure. Without ground-truth 3D labels, it reliably recovers 3D behavior from two-view videos, whereas state-of-the-art (SOTA) methods fail or yield degenerate solutions. Across the International Brain Lab and Cheese3D datasets, we demonstrate that SABLE learns 3D representations that match or exceed prior SOTA performance in neural encoding and decoding. Once pretrained across animals, SABLE serves as an off-the-shelf model that generalizes zero-shot to unseen animals without animal-specific calibration or retraining. Our method establishes 3D-aware video embeddings that capture complex behavior, opening new avenues for studying brain-behavior relationships.
Xinming Dai, Qihang Jin, Tianshu Tan +8
Columbia University · University of Science and Technology of China · Harvard University +3
Monocular imaging of animals inherently reduces 3D structures to 2D projections. Detection algorithms lead to 2D bounding boxes that lack information about animal's orientation relative to the camera. To build 3D detection methods for RGB animal images, there is a lack of labeled datasets; such labeling processes require 3D input streams along with RGB data. We present a pipeline that utilises Skinned Multi Animal Linear models to estimate 3D bounding boxes and to project them as robust labels into 2D image space using a dedicated camera pose refinement algorithm. To assess which sides of the animal are captured, cuboid face visibility metrics are computed. These 3D bounding boxes and metrics form a crucial step toward developing and benchmarking future monocular 3D animal detection algorithms. We evaluate our method on the Animal3D dataset, demonstrating accurate performance across species and settings.