Video-based human pose estimation remains challenged by motion blur, occlusion, and complex spatiotemporal dynamics. Existing methods often rely on heatmaps or implicit spatio-temporal feature aggregation, which limits joint topology expressiveness and weakens cross-frame consistency. To address these problems, we propose a novel node-centric framework that explicitly integrates visual, temporal, and structural reasoning for accurate pose estimation. First, we design a visuo-temporal velocity-based joint embedding that fuses sub-pixel joint cues and inter-frame motion to build appearance- and motion-aware representations. Then, we introduce an attention-driven pose-query encoder, which applies attention over joint-wise heatmaps and frame-wise features to map the joint representations into a pose-aware node space, generating image-conditioned joint-aware node embeddings. Building upon these node embeddings, we propose a dual-branch decoupled spatio-temporal attention graph that models temporal propagation and spatial constraint reasoning in specialized local and global branches. Finally, a node-space expert fusion module is proposed to adaptively fuse the complementary outputs from both branches, integrating local and global cues for final joint predictions. Extensive experiments on three widely used video pose benchmarks demonstrate that our method outperforms state-of-the-art methods. The results highlight the value of explicit node-centric reasoning, offering a new perspective for advancing video-based human pose estimation.
Figures & tables
Figure 1: Overview of the proposed node-centric decoupled spatio-temporal attention graph for VHPE. Given three frames, initial pose estimates and backbone features are extracted. A VTVJE encodes sub-pixel joint locations, velocity, peak score, and visibility into a joint descriptor, which the PQE refines into node embeddings. The embeddings are processed by the DSTAG: a local branch enforcing fine-grained structure and a global branch capturing long-range context. The NSEF module then fuses both branches to produce node predictions. Finally, a fixed GNC-Decoding module renders heatmaps from node predictions, supervised by a visibility-aware node loss.
Figure 2: Illustration of the Local and Global masks for one joint. The yellow box denotes the local window, while the blue box indicates the larger global window. Left: past frame t−2 with the heatmap peak and local patch. Middle: past frame t−1 with the peak shifted by the estimated motion between t−2 and t−1 . Right: current frame t , where the velocity-guided node window is centered at the extrapolated position.
Figure 3: Adjacency matrices used in the spatial GAT branches. Each row and column corresponds to a body joint, and nonzero entries indicate neighboring joints. (a) The 1-hop skeleton adjacency matrix used in the local spatial GAT branch. (b) The 2-hop skeleton adjacency matrix used in the global spatial GAT branch.
Method
Frames used
PoseWarper [ 3 ]
(t−3,…,t+3)
DCPose [ 18 ]
(t−1,t,t+1)
SLT-Pose [ 10 ]
(t−2,…,t+2)
FAMI-Pose [ 19 ]
(t−2,…,t+2)
DGNN [ 28 ]
(t−3,t−2,t−1,t)
DSTA [ 11 ]
(t−2,…,t+2)
Table 1: Temporal window configuration for each method. Unlike the majority of offline methods that rely on future frames, our approach is fully online and processes only past and current frames.
Method
Head
Shoulder
Elbow
Wrist
Hip
Knee
Ankle
Mean
DCPose [ 18 ]
83.2
84.7
82.3
78.1
80.3
79.2
73.5
80.5
FAMI-Pose [ 19 ]
83.3
85.4
82.9
78.6
81.3
80.5
75.3
81.2
DiffPose [ 9 ]
84.7
85.6
83.6
80.8
81.4
83.5
80.0
82.9
DSTA [ 11 ]
87.5
87.0
84.2
81.4
82.3
82.5
77.7
83.5
TDMI-ST [ 8 ]
86.8
87.4
85.1
81.4
83.8
82.7
78.0
83.8
JM-Pose [ 26 ]
85.8
88.1
85.7
82.5
84.1
83.1
78.5
84.0
Table 2: PoseTrack21 comparison with per-joint AP and mean.
Method
Head
Shoulder
Elbow
Wrist
Hip
Knee
Ankle
Mean
PoseWarper [ 3 ]
79.9
86.3
82.4
77.5
79.8
78.8
73.2
79.7
DCPose [ 18 ]
84.0
86.6
82.7
78.0
80.4
79.3
73.8
80.9
SLT-Pose [ 10 ]
84.3
87.5
83.5
78.5
80.9
80.2
74.4
81.5
FAMI-Pose [ 19 ]
85.5
87.7
84.2
79.2
81.4
81.1
74.9
82.2
DGNN [ 28 ]
85.1
87.7
85.3
80.0
81.1
81.6
77.2
82.7
DiffPose [ 9 ]
85.0
87.7
84.3
81.5
81.4
82.9
77.6
83.0
Table 3: PoseTrack18 comparison with per-joint AP and mean.
Method
Head
Shoulder
Elbow
Wrist
Hip
Knee
Ankle
Mean
PoseWarper [ 3 ]
81.4
88.3
83.9
78.0
82.4
80.5
73.6
81.2
DCPose [ 18 ]
88.0
88.7
84.1
78.4
83.0
81.4
74.2
82.8
SLT-Pose [ 10 ]
88.9
89.7
85.6
79.5
84.2
83.1
75.8
84.2
FAMI-Pose [ 19 ]
89.6
90.1
86.3
80.0
84.6
83.4
77.0
84.8
DGNN [ 28 ]
90.9
90.7
86.0
79.2
83.8
72.7
78.0
84.9
DSTA [ 11 ]
89.3
90.6
87.3
82.6
84.5
85.1
77.8
85.6
Table 4: PoseTrack17 comparison with per-joint AP and mean.
Method
Params (M)
GFLOPs
GMACs
Ours (ViTPose-H)
643.7
573.3
285.6
Ours (ViTPose-L)
314.9
283.2
141.7
Ours (ViTPose-B)
96.2
94.0
47.1
Ours (ViTPose-S)
30.7
38.6
19.2
TIPose (ViTPose-H) [ 14 ]
666.8
585.0
–
DSTA (ViTPose-H) [ 11 ]
631.0
123.9
–
Table 5: Model complexity comparison.
Figure 4: Comparisons on challenging scenes on PoseTrack17 including heavy occlusion, rapid motion, and atypical pose configuration. (a) Input frames; prediction from (b) Ours, (c) DSTA, and (d) DCPose. False predictions are highlighted with red circles.
Components
Head
Shoulder
Elbow
Wrist
Hip
Knee
Ankle
Mean
Δ Mean
Full Model
90.0
90.7
88.6
83.5
84.3
85.9
80.1
86.4
- 0.0
LOCAL ONLY
87.8
90.2
87.5
82.7
83.6
85.3
79.2
85.4
−1.0
GLOBAL ONLY
88.5
90.2
87.0
82.0
84.1
85.1
77.7
85.2
−1.2
SPATIAL ONLY
87.9
89.7
87.1
82.1
83.2
84.6
78.2
84.9
−1.5
TEMPORAL ONLY
87.4
89.7
86.9
81.7
83.3
84.2
78.1
84.7
−1.7
NO CENTER PREDICT
88.4
90.12
87.4
82.7
84.74
85.1
79.0
85.6
−0.8
Table 6: PoseTrack17 ablation by component (per-joint AP and mean AP). The last column reports the change in mean AP relative to the Full Model (negative means worse).
Vision Transformers (ViTs) have recently achieved state-of-the-art performance in 2D human pose estimation due to their strong global modeling capability. However, existing ViT-based pose estimators are designed for static images and process each frame independently, thereby ignoring the temporal coherence that exists in video sequences. This limitation often results in unstable predictions, especially in challenging scenes involving motion blur, occlusion, or defocus. In this paper, we propose TAR-ViTPose, a novel Temporal Aggregate-and-Restore Vision Transformer tailored for video-based 2D human pose estimation. TAR-ViTPose enhances static ViT representations by aggregating temporal cues across frames in a plug-and-play manner, leading to more robust and accurate pose estimation. To effectively aggregate joint-specific features that are temporally aligned across frames, we introduce a joint-centric temporal aggregation (JTA) that assigns each joint a learnable query token to selectively attend to its corresponding regions from neighboring frames. Furthermore, we develop a global restoring attention (GRA) to restore the aggregated temporal features back into the token sequence of the current frame, enriching its pose representation while fully preserving global context for precise keypoint localization. Extensive experiments demonstrate that TAR-ViTPose substantially improves upon the single-frame baseline ViTPose, achieving a +2.3 mAP gain on the PoseTrack2017 benchmark. Moreover, our approach outperforms existing state-of-the-art video-based methods, while also achieving a noticeably higher runtime frame rate in real-world applications. Project page: https://github.com/zgspose/TARViTPose.
Transformer-based methods have achieved strong performance in monocular 3D human pose estimation, but most existing approaches organise spatial and temporal reasoning as separate stages, which may weaken unified spatial-temporal interdependencies inherent in human motion and compress frame-level structural information before temporal modelling. In this paper, we propose HSTGFormer, a graph-enhanced Transformer framework that reformulates spatial-temporal reasoning as localised coupled graph aggregation over joint-time nodes. Specifically, HSTGFormer introduces a Hyper Spatial-Temporal Graph (HSTG), which decomposes global spatial-temporal reasoning into local spatial-temporal receptive fields around individual joint-time nodes by extending per-frame skeleton graphs into temporal neighbourhoods, thereby enabling structure-aware coupled reasoning while preserving local structural motion information. It further incorporates an Adaptive Dual-Scale Temporal Graph (ADSTG) to capture joint-specific temporal dependencies over complementary short- and long-range windows. A lightweight node-wise fusion module further adaptively integrates the two graph representations for each joint-time node. Experiments on Human3.6M and MPI-INF-3DHP show that HSTGFormer achieves strong accuracy with high computational efficiency.
Ruochen Li, Shuang Chen, Wenke E +2
Department of Computer Science Durham University Durham, UK
Accurate 3D human pose estimation from monocular videos requires effective modelling of complex spatial and temporal dependencies. However, existing methods often face challenges in efficiency and adaptability when modelling spatial and temporal dependencies, particularly under dense attention or fixed modelling schemes. In this work, we propose MASC-Pose, a Motion-Adaptive multi-scale temporal modelling framework with Skeleton-Constrained spatial graphs for efficient 3D human pose estimation. Specifically, it introduces an Adaptive Multi-scale Temporal Modelling (AMTM) module to adaptively capture heterogeneous motion dynamics at different temporal scales, together with a Skeleton-constrained Adaptive GCN (SAGCN) for joint-specific spatial interaction modelling. By jointly enabling adaptive temporal reasoning and efficient spatial aggregation, our method achieves strong accuracy with high computational efficiency. Extensive experiments on Human3.6M and MPI-INF-3DHP datasets demonstrate the effectiveness of our approach.
Ruochen Li, Shuang Chen, Wenke E +2
Department of Computer Science, Durham University, UK