Organizations: Department of Electrical Engineering, Indian Institute of Technology Gandhinagar, 382355, Gujarat, India. · National Institute of Mental Health, National Institutes of Health, MD 20892, Bethesda, USA.
Maintaining a consistent identity across video frames while estimating pose is crucial for downstream tasks such as behavior analysis and action recognition, yet existing pipelines largely decouple tracking and pose estimation. Top-down methods rely on per-frame detections followed by identity association, leading to increased computational cost and error propagation, whereas bottom-up approaches often struggle to preserve identity reliably in multi-instance settings. To address these limitations, we introduce DeepSTEP (STEP), a Transformer-based framework for joint tracking and pose estimation across diverse animal species and humans. STEP unifies localization, identity maintenance, and keypoint estimation into a single model, eliminating the need for per-frame detections or post hoc matching. Our approach leverages discriminative model weight prediction to propagate target identity across frames while predicting anatomically meaningful keypoints. Discriminative model prediction typically requires predefined target states. In contrast, STEP assumes only an initial bounding box in the first frame, removing the need for explicit keypoint initialization. This is enabled by two components: Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA), which allow direct keypoint prediction and simplify inference. Starting from the initial bounding box, STEP tracks the subject and estimates its pose in subsequent frames. We train and evaluate STEP on datasets spanning multiple species, demonstrating strong performance and greater efficiency than existing methods. By jointly modeling tracking and pose, STEP provides an effective and scalable solution.
Figures & tables
Figure 1: The proposed STEP framework for simultaneous tracking and pose estimation. (a) the full architecture, (b) the calculation of Target State Encodings, (c) the architecture of the offset map regression adapter (OMRA) module, and (d) Gaussian Soft Map Predictor (GMSP) Module.
Table 1: Comparison with state-of-the-art pose estimation methods, including TransPose [ 40 ] , YOLOv8 [ 17 ] , ViPNAS [ 38 ] , ViTPose [ 39 ] , DeepLabCut [ 19 ] , SuperAnimal [ 42 ] , RTMPose [ 16 ] , HRNet [ 34 ] , and PAVENet [ 46 ] . The best and second-best results are highlighted in green and orange, respectively. For the tracking performance of STEP, please refer to Figure 2 .
Figure 2: AUC and Overlap Precision (OPT) for bounding box tracking in STEP compared with: DiMP [ 2 ] , ToMP [ 28 ] , ATOM [ 8 ] , TAMOS [ 29 ] , and GMOT-Mamba [ 35 ] .
Model
Task
FPS
Detector
T100 (s)
STEP
Track + Pose
63
×
1.63
STEP*
Pose
∼ 63
✓
6.13
ViTPose [ 39 ]
Pose
231
✓
4.97
TransPose [ 40 ]
Pose
43
✓
6.87
YOLOv8-L [ 17 ]
Pose
113
✓
5.43
YOLOv8-M [ 17 ]
Pose
161
✓
5.16
Table 2: Inference speed comparison.
Figure 3: We demonstrate tracking and pose estimation during the occlusion of a target human instance by another human with a similar appearance. Zoom in is recommended for clarity. Panel (a) illustrates memory updates occurring only when at least 50% of keypoints and the bounding box localization are predicted with a probability greater than τm=0.6 . In contrast, panel (b) shows memory updates being performed regardless of the probability value. In both panels, low-confidence frames, which may introduce or propagate errors, are indicated by a red bounding box. In panel (b), STEP wrongly adjusts its confidence from the original target to the new instance based on the memory state. Subsequently, it starts tracking and performing pose estimation on the new object.
Figure 4: Left: Robustness to noisy first-frame initialization. Middle: Influence of tracking quality on pose estimation. Right: Sensitivity to the memory confidence threshold τ . (Please zoom for details.)
Table 4: Initialization using a detector [ 48 ] vs GT boxes.
τ
DiMP-18
ATOM
ToMP-50
GMOT
TaMOs
STEP
0.05
0.997
0.993
0.999
0.998
0.998
0.998
0.15
0.997
0.990
0.999
0.999
0.999
0.998
0.30
0.988
0.981
0.998
0.998
0.992
0.993
0.45
0.971
0.954
0.991
0.989
0.985
0.987
0.60
0.928
0.900
0.967
0.964
0.967
0.962
0.75
0.779
0.753
0.866
0.878
0.914
0.891
Table 5: Tracking. Overlap Precision (OP τ ) at different IoU thresholds for STEP compared with DiMP [ 2 ] , ATOM [ 8 ] , ToMP [ 28 ] , GMOT [ 35 ] , and TaMOs [ 29 ] .
Figure 5: Sensitivity to the memory confidence threshold τ . Effect of τ used for memory updates on tracking performance in terms of IDF1 and AUC.
Figure 6: Comparison of Mean Squared Error (MSE) and Object Keypoint Similarity (OKS) for individual keypoints against various existing methods: VitPose [ 39 ] , YOLOv8 [ 17 ] , TransPose [ 40 ] , ViPNAS [ 38 ] , DeepLabCut [ 19 ] , SuperAnimal [ 42 ] , HRNet [ 34 ] , RTMPose [ 16 ] , and PAVENet [ 46 ] . Notably, the marmoset dataset features top-view images of marmosets in a cage with infrequent occurrences of lower limb keypoints. The inherent imbalance in training data leads to notable inaccuracies across all methods for the four bottom-limb keypoints
Figure 7: (a) Slowest and (b) fastest quartiles of self-movement velocities over two days of recordings. Feeding events cause movement toward food locations, resulting in transitions from black to white clusters, shown in (c). Trajectories are obtained from bounding box centers. A high-occupancy scene with overlaid tracks and poses is shown in (d).
Figure 8: (a) Movement velocity observed on recorded data from AMC (tracks ≥1 sec). Distances are derived from AMC’s perspective on Google Maps. (b) IoU and Overlap Precision (OPT) for bounding box tracking in STEP compared with: DiMP [ 2 ] , ToMP [ 28 ] , ATOM [ 8 ] , TAMOS [ 29 ] , and GMOT-Mamba [ 35 ] .
Figure 9: Frames from supplementary video showcasing tracking and pose estimation. Please refer to the supplementary video for qualitative results on video sequences.
Figure 10: For a video sequence, columns display target score maps K^gm when initiated with bounding boxes, as depicted in (a) and (b). In (c), STEP runs concurrently for each bounding box to track and estimate the pose of respective targets. Notably, observe STEP’s robustness in estimating keypoints, particularly in occluded scenarios. A failure case is showcased in the last frame of (c) for the red bounded target, stemming from ambiguous activations within the red target’s score map, as evident in the last frame of (a).
Primates (APT36K)
Horses (APT36K)
Model
MSE ↓
OKS ↑
Model
MSE ↓
OKS ↑
STEP
437.15
0.783
STEP
503.63
0.784
DLC-Macaque
1948.82
0.650
DLC-Horse
3833.83
0.764
Table 6: Evaluation on the primate and horse subsets.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.