Vision Transformers (ViTs) have recently achieved state-of-the-art performance in 2D human pose estimation due to their strong global modeling capability. However, existing ViT-based pose estimators are designed for static images and process each frame independently, thereby ignoring the temporal coherence that exists in video sequences. This limitation often results in unstable predictions, especially in challenging scenes involving motion blur, occlusion, or defocus. In this paper, we propose TAR-ViTPose, a novel Temporal Aggregate-and-Restore Vision Transformer tailored for video-based 2D human pose estimation. TAR-ViTPose enhances static ViT representations by aggregating temporal cues across frames in a plug-and-play manner, leading to more robust and accurate pose estimation. To effectively aggregate joint-specific features that are temporally aligned across frames, we introduce a joint-centric temporal aggregation (JTA) that assigns each joint a learnable query token to selectively attend to its corresponding regions from neighboring frames. Furthermore, we develop a global restoring attention (GRA) to restore the aggregated temporal features back into the token sequence of the current frame, enriching its pose representation while fully preserving global context for precise keypoint localization. Extensive experiments demonstrate that TAR-ViTPose substantially improves upon the single-frame baseline ViTPose, achieving a +2.3 mAP gain on the PoseTrack2017 benchmark. Moreover, our approach outperforms existing state-of-the-art video-based methods, while also achieving a noticeably higher runtime frame rate in real-world applications. Project page: https://github.com/zgspose/TARViTPose.
Figures & tables
Figure 1: Comparison between the baseline ViTPose pipeline (a) and our TAR-ViTPose (b). (a) ViTPose adopts a ViT encoder to extract latent features from the input image, which are then fed into a lightweight decoder to regress keypoint heatmaps. (b) Our method enhances the current-frame representation by aggregating temporal cues from adjacent frames, achieving plug-and-play temporal modeling within the original ViTPose architecture.
Figure 2: The pipeline of the proposed Temporal Aggregate-and-Restore Vision Transformer (TAR-ViTPose). The objective is to estimate the human pose of the current frame Xi(t) . Given a video sequence ⟨Xi(t−T),…,Xi(t),…,Xi(t+T)⟩ , each frame is first encoded by the ViT encoder to obtain latent features {Fiout(τ)}τ=t−Tt+T . The JTA precisely aligns and aggregates keypoint features across frames. To achieve this, a query token is assigned to each joint ( Q ), and a mask-aware attention selectively attends to its corresponding joint regions in neighboring frames, guided by masks M(τ)τ=t−Tt+T derived from the decoded keypoint heatmaps H(τ)τ=t−Tt+T . Subsequently, the GRA injects the aggregated temporal features Q back into the current frame’s latent representation, producing an enhanced feature Fiout(t) , which is then fed into the lightweight decoder to generate the final keypoint heatmaps Hi(t) for the current frame.
Method
Head
Should.
Elbow
Wrist
Hip
Knee
Ankle
Mean
Backbone : ViT-S
ViTPose [ 41 ]
86.5
87.2
81.5
75.0
78.9
78.4
70.4
80.1
TAR-ViTPose (Ours)
87.1
88.0
83.2
77.8
80.4
81.0
74.2
81.9
Backbone : ViT-B
ViTPose [ 41 ]
86.7
87.9
83.0
77.7
80.3
81.0
73.9
81.7
TAR-ViTPose (Ours)
88.0
89.2
85.0
80.3
82.9
84.2
77.3
84.0
Table 1: Comparison with the ViTPose baseline (mAP) on PoseTrack2017 val. set.
Figure 3: Qualitative comparison of a) our TAR-ViTPose, b) ViTPose [ 41 ] , c) DCPose [ 24 ] , d) DSTA [ 14 ] , and e) Poseidon [ 27 ] , featuring challenges such as occlusion, motion blur, and defocus. The first two columns are from the PoseTrack datasets, while the last two columns are from in-the-wild videos. Inaccurate predictions are marked with red solid circles. Zoom in for clarity.
Method
Backbone
Head
Should.
Elbow
Wrist
Hip
Knee
Ankle
Mean
Using bounding boxes predicted by the Faster R-CNN detector [ 28 ]
PoseTrack [ 13 ]
ResNet-101
67.5
70.2
62.0
51.7
60.7
58.7
49.8
60.6
PoseFlow [ 40 ]
ResNet-152
66.7
73.3
68.3
61.1
67.5
67.0
61.3
66.5
SimBase. [ 39 ]
ResNet-152
81.7
83.4
80.0
72.4
75.3
74.8
67.1
76.7
HRNet [ 32 ]
HRNet-W48
82.1
83.6
80.4
73.3
75.5
75.3
68.5
77.3
PoseWarp. [ 2 ]
HRNet-W48
81.4
88.3
83.9
78.0
82.4
80.5
73.6
81.2
Table 2: Comparison with the SOTAs on PoseTrack2017 val. set. ‘ † ’ indicates an end-to-end method without bounding box detection, and ‘#’ indicates that the ViT backbone used is not specified. Similar to FAMI-Pose [ 25 ] and DSTA [ 14 ] , our proposed TAR-ViTPose sets the temporal span T to 2, which includes two preceding and two succeeding frames, totaling four auxiliary frames.
Method
Backbone
#Params
FPS(↑)
mAP
PoseWarper [ 2 ]
HRNet-W48
71.1M
52
81.0
DCPose [ 24 ]
HRNet-W48
65.2M
128
82.8
DSTA [ 14 ]
HRNet-W48
63.7M
156
83.4
DSTA [ 14 ]
ViT-H
422.2M
25
84.3
ViT-S
35.6M
413
81.5
ViT-B
101.1M
186
83.4
Table 3: Runtime frame rate (FPS), measured on an A6000. All methods utilize the same two auxiliary frames as in [ 24 ] .
Method
GFLOPs
mAP
(a) Self-attention (All frames)
38.22
82.2
(b) Cross-attention (Cur and Aux)
12.74
82.6
(c) Self-attention (Aux) + (b)
16.99
82.8
(d) Our joint-specific modeling
3.89
84.0
Table 4: Ablation of different temporal modeling strategies.
Method
JTA
GRA
mAP
#Params
GFLOPs
#Params
MFLOPs
14.19M
3.87
2.4M
21.23
(a)
✓
✗
70.3
(b)
✓
✓
84.0
ViTPose baseline [ 41 ]
81.7
Table 5: Effect of JTA and GRA in our temporal modeling.
Figure 4: Visualization of attention heatmaps for joint query tokens with (b) and without (a) mask-aware attention. Given a current frame Xi(t) and a neighboring frame Xi(t−T) , we visualize the attention heatmaps of three different joint query tokens with respect to the features of Xi(t−T) . See Supp. Material for more.
Method
Masking
mAP
(a)
✗
82.6
(b)
✓
84.0
Table 6: Impact of mask-aware attention.
Method
Head
Should.
Elbow
Wrist
Hip
Knee
Ankle
Mean
Backbone : ViT-S
ViTPose [ 41 ]
80.6
84.1
79.9
73.5
77.0
76.2
70.3
78.2
TAR-ViTPose(Ours)
82.8
85.6
81.5
75.2
78.0
78.1
72.1
79.3
Backbone : ViT-B
ViTPose [ 41 ]
83.2
86.3
82.8
77.5
78.2
79.4
74.9
80.5
TAR-ViTPose(Ours)
84.3
87.8
83.6
79.4
79.3
82.8
76.5
82.1
Table 7: Comparison with the ViTPose baseline (mAP) on Pose- Track2018 val. set.
Method
Head
Should.
Elbow
Wrist
Hip
Knee
Ankle
Mean
Backbone : ViT-S
ViTPose [ 41 ]
79.7
84.4
80.4
73.8
77.2
77.3
71.2
77.8
TAR-ViTPose(Ours)
81.6
84.7
80.9
74.2
77.8
77.9
71.7
78.5
Backbone : ViT-B
ViTPose [ 41 ]
81.8
85.2
82.1
76.6
78.5
79.5
74.5
79.9
TAR-ViTPose(Ours)
82.7
85.9
83.5
79.2
79.5
81.7
76.8
81.4
Table 8: Comparison with the ViTPose baseline (mAP) on Pose- Track21 val. set.
Method
Backbone
Head
Should.
Elbow
Wrist
Hip
Knee
Ankle
Mean
Using bounding boxes predicted by the Faster R-CNN detector [ 28 ]
PoseWarper [ 2 ]
HRNet-W48
79.9
86.3
82.4
77.5
79.8
78.8
73.2
79.7
PAVE-Net † [ 44 ]
Swin-L
84.9
87.5
81.7
74.5
77.9
76.9
72.4
79.7
DCPose [ 24 ]
HRNet-W48
84.0
86.6
82.7
78.0
80.4
79.3
73.8
80.9
FAMI-Pose [ 25 ]
HRNet-W48
85.5
87.7
84.2
79.2
81.4
81.1
74.9
82.2
DSTA [ 14 ]
ViT-H
85.9
88.8
85.0
81.1
81.5
83.0
77.4
83.4
Table 9: Comparison with the SOTAs on PoseTrack2018 val. set. ‘#’ indicates that the ViT backbone used is not specified, ‘ † ’ indicates the end-to-end approach. Similar to FAMI-Pose [ 25 ] and DSTA [ 14 ] , our proposed TAR-ViTPose sets the temporal span T to 2, which includes two preceding and two succeeding frames, totaling four auxiliary frames.
Method
Backbone
Head
Should.
Elbow
Wrist
Hip
Knee
Ankle
Mean
Using bounding boxes predicted by the Faster R-CNN detector [ 28 ]
SimBase. [ 39 ]
ResNet-152
80.5
81.2
73.2
64.8
73.9
72.7
67.7
73.9
HRNet [ 32 ]
HRNet-W48
81.5
83.2
81.1
75.4
79.2
77.8
71.9
78.8
PAVE-Net † [ 44 ]
Swin-L
84.7
86.5
81.9
74.7
77.4
76.6
71.4
79.4
PoseWarper [ 2 ]
HRNet-W48
82.3
84.0
82.2
75.5
80.7
78.7
71.6
79.5
DCPose [ 24 ]
HRNet-W48
83.7
84.4
82.6
78.7
80.1
79.8
74.4
80.7
Table 10: Comparison with the SOTAs on PoseTrack21 val. set. ‘#’ indicates that the ViT backbone used is not specified, ‘ † ’ indicates the end-to-end approach. Similar to FAMI-Pose [ 25 ] and DSTA [ 14 ] , our proposed TAR-ViTPose sets the temporal span T to 2, which includes two preceding and two succeeding frames, totaling four auxiliary frames.
Threshold ϕ
0.1
0.15
0.2
0.25
0.3
mAP
83.2
83.7
84.0
83.5
82.9
Table 11: Impact of different mask thresholds.
#Layer
2
4
6
8
10
mAP
83.4
83.8
84.0
83.9
83.9
Table 12: Different number of layers in JTA.
#Layer
1
2
3
4
mAP
84.0
83.9
84.0
84.0
Table 13: Different number of layers in GRA.
Figure 5: Additional qualitative results of our TAR-ViTPose. The first two rows are from the PoseTrack datasets, while the last row is from in-the-wild videos. Across all examples, the model remains robust under occlusion, motion blur, complex poses, and defocus.
#Auxiliary Frame
ViT-S
ViT-B
ViT-H
1 {-1}
81.4
83.2
86.1
2 {-1, +1}
81.5
83.4
86.3
4 {-2, -1, +1, +2}
81.9
84.0
86.8
Table 14: Different number of auxiliary frames. ‘-’ indicates previous frames while ‘+’ indicates subsequent frames.
Figure 6: Visualization of attention heatmaps for joint query tokens with mask-aware attention. For each example, given a current frame Xi(t) and its neighboring frames Xi(t−T) and Xi(t+T) , we visualize the attention heatmaps of nine joint query tokens, including the head, shoulders, elbows, wrists, right knee, and right ankle, over the feature maps of these three frames.
Video-based human pose estimation remains challenged by motion blur, occlusion, and complex spatiotemporal dynamics. Existing methods often rely on heatmaps or implicit spatio-temporal feature aggregation, which limits joint topology expressiveness and weakens cross-frame consistency. To address these problems, we propose a novel node-centric framework that explicitly integrates visual, temporal, and structural reasoning for accurate pose estimation. First, we design a visuo-temporal velocity-based joint embedding that fuses sub-pixel joint cues and inter-frame motion to build appearance- and motion-aware representations. Then, we introduce an attention-driven pose-query encoder, which applies attention over joint-wise heatmaps and frame-wise features to map the joint representations into a pose-aware node space, generating image-conditioned joint-aware node embeddings. Building upon these node embeddings, we propose a dual-branch decoupled spatio-temporal attention graph that models temporal propagation and spatial constraint reasoning in specialized local and global branches. Finally, a node-space expert fusion module is proposed to adaptively fuse the complementary outputs from both branches, integrating local and global cues for final joint predictions. Extensive experiments on three widely used video pose benchmarks demonstrate that our method outperforms state-of-the-art methods. The results highlight the value of explicit node-centric reasoning, offering a new perspective for advancing video-based human pose estimation.
Multi-person pose estimation (MPPE), which involves detecting body joint positions (keypoints) for every person in an image, is a fundamental task in computer vision. Despite recent advances, no transformer-based model currently achieves real-time performance. This work addresses the latency challenge by introducing DETRPose, the first family of real-time, end-to-end transformer models for multi-person 2D pose estimation. DETRPose significantly enhances the GroupPose decoder, enabling real-time inference. For training, a novel denoising keypoint technique is proposed to accelerate convergence. The varifocal loss is also extended for keypoints, termed Keypoint Similarity VariFocal loss, to improve query quality. Extensive evaluation demonstrates that DETRPose models achieve accuracy comparable to or exceeding that of leading alternatives while requiring five to ten times fewer training epochs. DETRPose-S matches the accuracy of YOLOv8-Pose-X and YOLO11-Pose-X on the COCO dataset (67.0 vs 67.3 and 67.2 in AP) with 81% fewer parameters (11.5M vs 69.4M and 58.8M) and 52% faster inference speed (2.39ms vs 5.23ms and 4.93ms). On the CrowdPose dataset, DETRPose-X has 13.1× fewer FLOPs (232.3G vs 3048.1G) and only 2 fewer precision (75.1 vs 76.6 in AP) than ED-Pose-SwinL-5S. On the OCHuman dataset, DETRPose-S surpasses all previous models, showing the robustness of DETRPose on out-of-distribution datasets. Code is available at https://github.com/SebastianJanampa/DETRPose
The rapid advances in deep learning have significantly enhanced the accuracy of multimodal 3D human pose estimation (HPE). However, the state-of-the-art (SOTA) HPE pipelines still rely on Transformers, whose quadratic complexity makes real-time processing for long sequences impractical. Mamba addresses this issue through selective state-space modeling, enabling efficient sequence processing without sacrificing representational power. Nevertheless, it struggles to capture complex spatial dependencies in multimodal settings. To bridge this gap, we propose VIMCAN, a hybrid architecture that combines the efficient sequence modeling of Mamba with the spatial reasoning of Cross-Attention, and performs robust visual-inertial fusion and human pose estimation between RGB keypoints and wearable IMU data. By leveraging Mamba's dynamic parameterization for temporal modeling and Attention for spatial dependency extraction, VIMCAN achieves superior accuracy, with mean per-joint position errors (MPJPE) of 17.2 mm on TotalCapture and 45.3 mm on 3DPW. VIMCAN outperforms prior Transformer-based and other SOTA approaches while supporting real-time inference at over 60 frames per second on consumer-grade hardware. The source code is available at this GitHub repository.
Zepeng Yang, Junxuan Bai, Hao Li +4
Beihang University · Capital University of Physical Education and Sports · Peng Cheng Laboratory +1