Photogrammetric 3D Model-Based Animal Behavior Classification
Authors: Olaf Hellwich, Moritz Heine, Patrik Reiske, Sole Traverso, Katharina Hohlbaum, Henning Sprekeler, Christa Thöne-Reineke, Lars Lewejohann, Roman Bhuiyan
Organizations: TU Berlin, Computer Vision & Remote Sensing · TU Berlin, Science of Intelligence Excellence Cluster · TU Berlin, Modeling of Cognitive Processes · German Centre for the Protection of Laboratory Animals (Bf3R), German Federal Institute for Risk Assessment (BfR) · FU Berlin, Institute of Animal Welfare, Animal Behavior and Laboratory Animal Science
Video-based behavior estimation is often implemented end-to-end from image sequences. We advocate a two-stage alternative: reconstruct 3D body and object geometry before classifying behavior. This makes ambiguities of the ill-posed 2D-to-3D inverse problem explicit and potentially correctable. We demonstrate the strategy with simulated interacting mice and extend it to real video.
Figures & tables
Fig. 1: Simulated skeleton and rendered video. Top, front and side view
Figure 2Figure 3
Model
Input source
Interaction F1
Rear0 F1
Rear1 F1
LSTM
Simulated 3D tracks
0.6276
0.8369
0.8316
Reconstructed 3D tracks
0.6147
0.8116
0.8066
xLSTM
Simulated 3D tracks
0.7146
0.7500
0.8866
Reconstructed 3D tracks
0.6702
0.6776
0.8293
Transformer
Simulated 3D tracks
0.7271
0.8611
0.8743
Reconstructed 3D tracks
0.7240
0.8286
0.8432
Table 1: F1 scores for behavior classification on the same three-minute simulated test sequence. The 3D classifiers were trained on simulated 3D body-part tracks and evaluated on simulated or cleaned, completed 3D-reconstructed tracks. The direct Transformer was trained on perfect multiview 2D projections of the same behavior-training trajectory, using the same temporal training/validation split, and evaluated on DLC 2D detections. Its inputs use pair-centered image coordinates, without segmentation, explicit 3D reconstruction, or additional identity correction. Rearing scores are evaluated on ground-truth non-interaction frames. Bold values denote the highest score for each behavior within each of the two 3D input-source groups.
Fig. 7: Mouse lock box experiment. The tail root in the front view is wrongly extracted.
Fig. 8: Camera calibration with the lock-box. The lock-box CAD model ( blue ), projected with the fitted camera parameters, covers the lock-box in the front, side, and top views.
Fig. 9: 3D reconstruction of the mouse. (a)–(c) Keypoints reconstructed in 3D from all three views and projected back into each image: head (nose, ears) ∙ , paws ∙ , tail root ∙ ; white lines connect neighboring keypoints of the skeleton. (d) A keypoint error revealed by geometry: DeepLabCut places the tail root at the red cross (likelihood 0.95) at the left image border. Side and top views agree on a tail-root position whose projection into the front view ( ∙ ) lies on the mouse, about 1,800 pixels away.