Organizations: Department of Electrical Engineering, Indian Institute of Technology Gandhinagar, 382355, Gujarat, India. · National Institute of Mental Health, National Institutes of Health, MD 20892, Bethesda, USA.
Maintaining a consistent identity across video frames while estimating pose is crucial for downstream tasks such as behavior analysis and action recognition, yet existing pipelines largely decouple tracking and pose estimation. Top-down methods rely on per-frame detections followed by identity association, leading to increased computational cost and error propagation, whereas bottom-up approaches often struggle to preserve identity reliably in multi-instance settings. To address these limitations, we introduce DeepSTEP (STEP), a Transformer-based framework for joint tracking and pose estimation across diverse animal species and humans. STEP unifies localization, identity maintenance, and keypoint estimation into a single model, eliminating the need for per-frame detections or post hoc matching. Our approach leverages discriminative model weight prediction to propagate target identity across frames while predicting anatomically meaningful keypoints. Discriminative model prediction typically requires predefined target states. In contrast, STEP assumes only an initial bounding box in the first frame, removing the need for explicit keypoint initialization. This is enabled by two components: Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA), which allow direct keypoint prediction and simplify inference. Starting from the initial bounding box, STEP tracks the subject and estimates its pose in subsequent frames. We train and evaluate STEP on datasets spanning multiple species, demonstrating strong performance and greater efficiency than existing methods. By jointly modeling tracking and pose, STEP provides an effective and scalable solution.
Figures & tables
Figure 1: The proposed STEP framework for simultaneous tracking and pose estimation. (a) the full architecture, (b) the calculation of Target State Encodings, (c) the architecture of the offset map regression adapter (OMRA) module, and (d) Gaussian Soft Map Predictor (GMSP) Module.
Table 1: Comparison with state-of-the-art pose estimation methods, including TransPose [ 40 ] , YOLOv8 [ 17 ] , ViPNAS [ 38 ] , ViTPose [ 39 ] , DeepLabCut [ 19 ] , SuperAnimal [ 42 ] , RTMPose [ 16 ] , HRNet [ 34 ] , and PAVENet [ 46 ] . The best and second-best results are highlighted in green and orange, respectively. For the tracking performance of STEP, please refer to Figure 2 .
Figure 2: AUC and Overlap Precision (OPT) for bounding box tracking in STEP compared with: DiMP [ 2 ] , ToMP [ 28 ] , ATOM [ 8 ] , TAMOS [ 29 ] , and GMOT-Mamba [ 35 ] .
Model
Task
FPS
Detector
T100 (s)
STEP
Track + Pose
63
×
1.63
STEP*
Pose
∼ 63
✓
6.13
ViTPose [ 39 ]
Pose
231
✓
4.97
TransPose [ 40 ]
Pose
43
✓
6.87
YOLOv8-L [ 17 ]
Pose
113
✓
5.43
YOLOv8-M [ 17 ]
Pose
161
✓
5.16
Table 2: Inference speed comparison.
Figure 3: We demonstrate tracking and pose estimation during the occlusion of a target human instance by another human with a similar appearance. Zoom in is recommended for clarity. Panel (a) illustrates memory updates occurring only when at least 50% of keypoints and the bounding box localization are predicted with a probability greater than τm=0.6 . In contrast, panel (b) shows memory updates being performed regardless of the probability value. In both panels, low-confidence frames, which may introduce or propagate errors, are indicated by a red bounding box. In panel (b), STEP wrongly adjusts its confidence from the original target to the new instance based on the memory state. Subsequently, it starts tracking and performing pose estimation on the new object.
Figure 4: Left: Robustness to noisy first-frame initialization. Middle: Influence of tracking quality on pose estimation. Right: Sensitivity to the memory confidence threshold τ . (Please zoom for details.)
Table 4: Initialization using a detector [ 48 ] vs GT boxes.
τ
DiMP-18
ATOM
ToMP-50
GMOT
TaMOs
STEP
0.05
0.997
0.993
0.999
0.998
0.998
0.998
0.15
0.997
0.990
0.999
0.999
0.999
0.998
0.30
0.988
0.981
0.998
0.998
0.992
0.993
0.45
0.971
0.954
0.991
0.989
0.985
0.987
0.60
0.928
0.900
0.967
0.964
0.967
0.962
0.75
0.779
0.753
0.866
0.878
0.914
0.891
Table 5: Tracking. Overlap Precision (OP τ ) at different IoU thresholds for STEP compared with DiMP [ 2 ] , ATOM [ 8 ] , ToMP [ 28 ] , GMOT [ 35 ] , and TaMOs [ 29 ] .
Figure 5: Sensitivity to the memory confidence threshold τ . Effect of τ used for memory updates on tracking performance in terms of IDF1 and AUC.
Figure 6: Comparison of Mean Squared Error (MSE) and Object Keypoint Similarity (OKS) for individual keypoints against various existing methods: VitPose [ 39 ] , YOLOv8 [ 17 ] , TransPose [ 40 ] , ViPNAS [ 38 ] , DeepLabCut [ 19 ] , SuperAnimal [ 42 ] , HRNet [ 34 ] , RTMPose [ 16 ] , and PAVENet [ 46 ] . Notably, the marmoset dataset features top-view images of marmosets in a cage with infrequent occurrences of lower limb keypoints. The inherent imbalance in training data leads to notable inaccuracies across all methods for the four bottom-limb keypoints
Figure 7: (a) Slowest and (b) fastest quartiles of self-movement velocities over two days of recordings. Feeding events cause movement toward food locations, resulting in transitions from black to white clusters, shown in (c). Trajectories are obtained from bounding box centers. A high-occupancy scene with overlaid tracks and poses is shown in (d).
Figure 8: (a) Movement velocity observed on recorded data from AMC (tracks ≥1 sec). Distances are derived from AMC’s perspective on Google Maps. (b) IoU and Overlap Precision (OPT) for bounding box tracking in STEP compared with: DiMP [ 2 ] , ToMP [ 28 ] , ATOM [ 8 ] , TAMOS [ 29 ] , and GMOT-Mamba [ 35 ] .
Figure 9: Frames from supplementary video showcasing tracking and pose estimation. Please refer to the supplementary video for qualitative results on video sequences.
Figure 10: For a video sequence, columns display target score maps K^gm when initiated with bounding boxes, as depicted in (a) and (b). In (c), STEP runs concurrently for each bounding box to track and estimate the pose of respective targets. Notably, observe STEP’s robustness in estimating keypoints, particularly in occluded scenarios. A failure case is showcased in the last frame of (c) for the red bounded target, stemming from ambiguous activations within the red target’s score map, as evident in the last frame of (a).
Primates (APT36K)
Horses (APT36K)
Model
MSE ↓
OKS ↑
Model
MSE ↓
OKS ↑
STEP
437.15
0.783
STEP
503.63
0.784
DLC-Macaque
1948.82
0.650
DLC-Horse
3833.83
0.764
Table 6: Evaluation on the primate and horse subsets.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
Animal pose estimation and tracking is important for wildlife monitoring and conservation research, and with limited expert time for labelling automated approaches are imperative. While human pose estimation and tracking has seen rapid progress thanks to large annotated datasets, animal pose remain challenging, due to large morphological and behavioural differences between species and limited annotated data. Existing approaches either optimise generic keypoint localisation from annotated datasets (such as APTv2) with poor generalisation, or track custom keypoints using visual tracking, at the cost of performance. In this paper, we demonstrate that vision foundation models trained on large datasets can be used effectively to track animal pose with limited labelled data. We propose two models, one unsupervised and the other supervised, to track user-selected keypoints in videos. The supervised approach delivers superior tracking accuracy by employing a keypoint prompt encoder to explicitly inject structural priors from a reference frame into feature matching. In parallel, the unsupervised route provides strong cross-species robustness by leveraging diverse foundation-model features for training-free correspondence matching. Extensive evaluation on challenging animal video benchmarks APTv2 and TigDog demonstrates that our framework achieves strong performance while maintaining an effective balance between accuracy and generalisation, offering a practical solution for real-world animal behaviour analysis and conservation applications.
Single-stage multi-person pose estimation aims to jointly perform human localization and keypoint prediction within a unified framework, offering advantages in inference efficiency and architectural simplicity. Consequently, multi-scale real-time detection architectures, such as YOLO-like models, are widely adopted for real-time pose estimation. However, these approaches typically inherit a box-driven modeling paradigm from object detection, in which pose estimation is implicitly constrained by bounding-box supervision during training. This formulation introduces biases in sample assignment and feature representation, resulting in task misalignment and ultimately limiting pose estimation accuracy. In this work, we revisit box-driven single-stage pose estimation from a keypoint-driven perspective and identify semantic conflicts among parallel objectives as a key source of performance degradation. To address this issue, we propose a keypoint-driven learning paradigm that elevates pose estimation to a primary prediction objective. Specifically, we remove bounding-box prediction and redesign the prediction head to better accommodate the high-dimensional structured representations for pose estimation. We further introduce a keypoint-driven dynamic sample assignment strategy to align training objectives with pose evaluation metrics, enabling dense supervision during training and efficient NMS-free inference. In addition, we propose a smooth OKS-based loss function to stabilize optimization in regression-based pose estimation. Based on these designs, we develop a single-stage multi-person pose estimation framework, termed ER-Pose. On MS COCO and CrowdPose, ER-Pose-n achieves AP improvements of 3.2/6.7 without pre-training and 7.4/4.9 with pre-training respectively compared with the baseline YOLO-Pose. These improvements are achieved with fewer parameters and higher inference efficiency.
Multi-person pose estimation (MPPE), which involves detecting body joint positions (keypoints) for every person in an image, is a fundamental task in computer vision. Despite recent advances, no transformer-based model currently achieves real-time performance. This work addresses the latency challenge by introducing DETRPose, the first family of real-time, end-to-end transformer models for multi-person 2D pose estimation. DETRPose significantly enhances the GroupPose decoder, enabling real-time inference. For training, a novel denoising keypoint technique is proposed to accelerate convergence. The varifocal loss is also extended for keypoints, termed Keypoint Similarity VariFocal loss, to improve query quality. Extensive evaluation demonstrates that DETRPose models achieve accuracy comparable to or exceeding that of leading alternatives while requiring five to ten times fewer training epochs. DETRPose-S matches the accuracy of YOLOv8-Pose-X and YOLO11-Pose-X on the COCO dataset (67.0 vs 67.3 and 67.2 in AP) with 81% fewer parameters (11.5M vs 69.4M and 58.8M) and 52% faster inference speed (2.39ms vs 5.23ms and 4.93ms). On the CrowdPose dataset, DETRPose-X has 13.1× fewer FLOPs (232.3G vs 3048.1G) and only 2 fewer precision (75.1 vs 76.6 in AP) than ED-Pose-SwinL-5S. On the OCHuman dataset, DETRPose-S surpasses all previous models, showing the robustness of DETRPose on out-of-distribution datasets. Code is available at https://github.com/SebastianJanampa/DETRPose