Beyond Static Frames: Temporal Aggregate-and-Restore Vision Transformer for Human Pose Estimation

Hongwei Fang; Jiahang Cai; Xun Wang; Wenwu Yang

cs.CVMar 6, 2026

Beyond Static Frames: Temporal Aggregate-and-Restore Vision Transformer for Human Pose Estimation

Authors: Hongwei Fang, Jiahang Cai, Xun Wang, Wenwu Yang

Organizations: Zhejiang Gongshang University, China

Abstract

Vision Transformers (ViTs) have recently achieved state-of-the-art performance in 2D human pose estimation due to their strong global modeling capability. However, existing ViT-based pose estimators are designed for static images and process each frame independently, thereby ignoring the temporal coherence that exists in video sequences. This limitation often results in unstable predictions, especially in challenging scenes involving motion blur, occlusion, or defocus. In this paper, we propose TAR-ViTPose, a novel Temporal Aggregate-and-Restore Vision Transformer tailored for video-based 2D human pose estimation. TAR-ViTPose enhances static ViT representations by aggregating temporal cues across frames in a plug-and-play manner, leading to more robust and accurate pose estimation. To effectively aggregate joint-specific features that are temporally aligned across frames, we introduce a joint-centric temporal aggregation (JTA) that assigns each joint a learnable query token to selectively attend to its corresponding regions from neighboring frames. Furthermore, we develop a global restoring attention (GRA) to restore the aggregated temporal features back into the token sequence of the current frame, enriching its pose representation while fully preserving global context for precise keypoint localization. Extensive experiments demonstrate that TAR-ViTPose substantially improves upon the single-frame baseline ViTPose, achieving a +2.3 mAP gain on the PoseTrack2017 benchmark. Moreover, our approach outperforms existing state-of-the-art video-based methods, while also achieving a noticeably higher runtime frame rate in real-world applications. Project page: https://github.com/zgspose/TARViTPose.

Figures & tables

Figure 1: Comparison between the baseline ViTPose pipeline (a) and our TAR-ViTPose (b). (a) ViTPose adopts a ViT encoder to extract latent features from the input image, which are then fed into a lightweight decoder to regress keypoint heatmaps. (b) Our method enhances the current-frame representation by aggregating temporal cues from adjacent frames, achieving plug-and-play temporal modeling within the original ViTPose architecture.

Figure 2: The pipeline of the proposed Temporal Aggregate-and-Restore Vision Transformer (TAR-ViTPose). The objective is to estimate the human pose of the current frame $X_{i}(t)$ . Given a video sequence $\langle X_{i}(t\!-\!T),\dots,X_{i}(t),\dots,X_{i}(t\!+\!T)\rangle$ , each frame is first encoded by the ViT encoder to obtain latent features $\{F_{i}^{\text{out}}(\tau)\}_{\tau=t-T}^{t+T}$ . The JTA precisely aligns and aggregates keypoint features across frames. To achieve this, a query token is assigned to each joint ( $Q$ ), and a mask-aware attention selectively attends to its corresponding joint regions in neighboring frames, guided by masks $M(\tau)_{\tau=t-T}^{t+T}$ derived from the decoded keypoint heatmaps $\overline{H}(\tau)_{\tau=t-T}^{t+T}$ . Subsequently, the GRA injects the aggregated temporal features $\widetilde{Q}$ back into the current frame’s latent representation, producing an enhanced feature $\widehat{F}_{i}^{\text{out}}(t)$ , which is then fed into the lightweight decoder to generate the final keypoint heatmaps $H_{i}(t)$ for the current frame.

Method	Head	Should.	Elbow	Wrist	Hip	Knee	Ankle	Mean
Backbone : ViT-S
ViTPose [ 41 ]	86.5	87.2	81.5	75.0	78.9	78.4	70.4	80.1
TAR-ViTPose (Ours)	87.1	88.0	83.2	77.8	80.4	81.0	74.2	81.9
Backbone : ViT-B
ViTPose [ 41 ]	86.7	87.9	83.0	77.7	80.3	81.0	73.9	81.7
TAR-ViTPose (Ours)	88.0	89.2	85.0	80.3	82.9	84.2	77.3	84.0

Table 1: Comparison with the ViTPose baseline (mAP) on PoseTrack2017 val. set.

Figure 3: Qualitative comparison of a) our TAR-ViTPose, b) ViTPose [ 41 ] , c) DCPose [ 24 ] , d) DSTA [ 14 ] , and e) Poseidon [ 27 ] , featuring challenges such as occlusion, motion blur, and defocus. The first two columns are from the PoseTrack datasets, while the last two columns are from in-the-wild videos. Inaccurate predictions are marked with red solid circles. Zoom in for clarity.

Method	Backbone	Head	Should.	Elbow	Wrist	Hip	Knee	Ankle	Mean
Using bounding boxes predicted by the Faster R-CNN detector [ 28 ]
PoseTrack [ 13 ]	ResNet-101	67.5	70.2	62.0	51.7	60.7	58.7	49.8	60.6
PoseFlow [ 40 ]	ResNet-152	66.7	73.3	68.3	61.1	67.5	67.0	61.3	66.5
SimBase. [ 39 ]	ResNet-152	81.7	83.4	80.0	72.4	75.3	74.8	67.1	76.7
HRNet [ 32 ]	HRNet-W48	82.1	83.6	80.4	73.3	75.5	75.3	68.5	77.3
PoseWarp. [ 2 ]	HRNet-W48	81.4	88.3	83.9	78.0	82.4	80.5	73.6	81.2

Table 2: Comparison with the SOTAs on PoseTrack2017 val. set. ‘

\dagger

’ indicates an end-to-end method without bounding box detection, and ‘#’ indicates that the ViT backbone used is not specified. Similar to FAMI-Pose [ 25 ] and DSTA [ 14 ] , our proposed TAR-ViTPose sets the temporal span

T

to 2, which includes two preceding and two succeeding frames, totaling four auxiliary frames.

Method	Backbone	#Params	FPS(↑)	mAP
PoseWarper [ 2 ]	HRNet-W48	71.1M	52	81.0
DCPose [ 24 ]	HRNet-W48	65.2M	128	82.8
DSTA [ 14 ]	HRNet-W48	63.7M	156	83.4
DSTA [ 14 ]	ViT-H	422.2M	25	84.3
	ViT-S	35.6M	413	81.5
	ViT-B	101.1M	186	83.4

Table 3: Runtime frame rate (FPS), measured on an A6000. All methods utilize the same two auxiliary frames as in [ 24 ] .

Method	GFLOPs	mAP
(a) Self-attention (All frames)	38.22	82.2
(b) Cross-attention (Cur and Aux)	12.74	82.6
(c) Self-attention (Aux) + (b)	16.99	82.8
(d) Our joint-specific modeling	3.89	84.0

Table 4: Ablation of different temporal modeling strategies.

Method	JTA		GRA		mAP
	#Params	GFLOPs	#Params	MFLOPs
	14.19M	3.87	2.4M	21.23
(a)	✓		✗		70.3
(b)	✓		✓		84.0
ViTPose baseline [ 41 ]					81.7

Table 5: Effect of JTA and GRA in our temporal modeling.

Figure 4: Visualization of attention heatmaps for joint query tokens with (b) and without (a) mask-aware attention. Given a current frame $X_{i}(t)$ and a neighboring frame $X_{i}(t-T)$ , we visualize the attention heatmaps of three different joint query tokens with respect to the features of $X_{i}(t-T)$ . See Supp. Material for more.

Method	Masking	mAP
(a)	✗	82.6
(b)	✓	84.0

Table 6: Impact of mask-aware attention.

Method	Head	Should.	Elbow	Wrist	Hip	Knee	Ankle	Mean
Backbone : ViT-S
ViTPose [ 41 ]	80.6	84.1	79.9	73.5	77.0	76.2	70.3	78.2
TAR-ViTPose(Ours)	82.8	85.6	81.5	75.2	78.0	78.1	72.1	79.3
Backbone : ViT-B
ViTPose [ 41 ]	83.2	86.3	82.8	77.5	78.2	79.4	74.9	80.5
TAR-ViTPose(Ours)	84.3	87.8	83.6	79.4	79.3	82.8	76.5	82.1

Table 7: Comparison with the ViTPose baseline (mAP) on Pose- Track2018 val. set.

Method	Head	Should.	Elbow	Wrist	Hip	Knee	Ankle	Mean
Backbone : ViT-S
ViTPose [ 41 ]	79.7	84.4	80.4	73.8	77.2	77.3	71.2	77.8
TAR-ViTPose(Ours)	81.6	84.7	80.9	74.2	77.8	77.9	71.7	78.5
Backbone : ViT-B
ViTPose [ 41 ]	81.8	85.2	82.1	76.6	78.5	79.5	74.5	79.9
TAR-ViTPose(Ours)	82.7	85.9	83.5	79.2	79.5	81.7	76.8	81.4

Table 8: Comparison with the ViTPose baseline (mAP) on Pose- Track21 val. set.

Method	Backbone	Head	Should.	Elbow	Wrist	Hip	Knee	Ankle	Mean
Using bounding boxes predicted by the Faster R-CNN detector [ 28 ]
PoseWarper [ 2 ]	HRNet-W48	79.9	86.3	82.4	77.5	79.8	78.8	73.2	79.7
PAVE-Net $\dagger$ [ 44 ]	Swin-L	84.9	87.5	81.7	74.5	77.9	76.9	72.4	79.7
DCPose [ 24 ]	HRNet-W48	84.0	86.6	82.7	78.0	80.4	79.3	73.8	80.9
FAMI-Pose [ 25 ]	HRNet-W48	85.5	87.7	84.2	79.2	81.4	81.1	74.9	82.2
DSTA [ 14 ]	ViT-H	85.9	88.8	85.0	81.1	81.5	83.0	77.4	83.4

Table 9: Comparison with the SOTAs on PoseTrack2018 val. set. ‘#’ indicates that the ViT backbone used is not specified, ‘

\dagger

’ indicates the end-to-end approach. Similar to FAMI-Pose [ 25 ] and DSTA [ 14 ] , our proposed TAR-ViTPose sets the temporal span

T

to 2, which includes two preceding and two succeeding frames, totaling four auxiliary frames.

Method	Backbone	Head	Should.	Elbow	Wrist	Hip	Knee	Ankle	Mean
Using bounding boxes predicted by the Faster R-CNN detector [ 28 ]
SimBase. [ 39 ]	ResNet-152	80.5	81.2	73.2	64.8	73.9	72.7	67.7	73.9
HRNet [ 32 ]	HRNet-W48	81.5	83.2	81.1	75.4	79.2	77.8	71.9	78.8
PAVE-Net $\dagger$ [ 44 ]	Swin-L	84.7	86.5	81.9	74.7	77.4	76.6	71.4	79.4
PoseWarper [ 2 ]	HRNet-W48	82.3	84.0	82.2	75.5	80.7	78.7	71.6	79.5
DCPose [ 24 ]	HRNet-W48	83.7	84.4	82.6	78.7	80.1	79.8	74.4	80.7

Table 10: Comparison with the SOTAs on PoseTrack21 val. set. ‘#’ indicates that the ViT backbone used is not specified, ‘

\dagger

’ indicates the end-to-end approach. Similar to FAMI-Pose [ 25 ] and DSTA [ 14 ] , our proposed TAR-ViTPose sets the temporal span

T

to 2, which includes two preceding and two succeeding frames, totaling four auxiliary frames.

Threshold $\phi$	0.1	0.15	0.2	0.25	0.3
mAP	83.2	83.7	84.0	83.5	82.9

Table 11: Impact of different mask thresholds.

#Layer	2	4	6	8	10
mAP	83.4	83.8	84.0	83.9	83.9

Table 12: Different number of layers in JTA.

#Layer	1	2	3	4
mAP	84.0	83.9	84.0	84.0

Table 13: Different number of layers in GRA.

Figure 5: Additional qualitative results of our TAR-ViTPose. The first two rows are from the PoseTrack datasets, while the last row is from in-the-wild videos. Across all examples, the model remains robust under occlusion, motion blur, complex poses, and defocus.

#Auxiliary Frame	ViT-S	ViT-B	ViT-H
1 {-1}	81.4	83.2	86.1
2 {-1, +1}	81.5	83.4	86.3
4 {-2, -1, +1, +2}	81.9	84.0	86.8

Table 14: Different number of auxiliary frames. ‘-’ indicates previous frames while ‘+’ indicates subsequent frames.

Figure 6: Visualization of attention heatmaps for joint query tokens with mask-aware attention. For each example, given a current frame $X_{i}(t)$ and its neighboring frames $X_{i}(t-T)$ and $X_{i}(t+T)$ , we visualize the attention heatmaps of nine joint query tokens, including the head, shoulders, elbows, wrists, right knee, and right ankle, over the feature maps of these three frames.

Explore similar work

NCSTR: Node-Centric Decoupled Spatio-Temporal Reasoning for Video-based Human Pose Estimation
Mar 20, 2026Quang Dang Huynh, Xuefei Yin, Andrew Busch +4Human Pose Estimation Spatiotemporal GNNs
Griffith University
DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints
Jun 16, 2025Sebastian Janampa, Marios PattichisDetection Transformer Efficient ViTs
The University of New Mexico
VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network
May 8, 2026Zepeng Yang, Junxuan Bai, Hao Li +43D Human Pose Estimation Human Pose Estimation
Beihang University · Capital University of Physical Education and Sports · Peng Cheng Laboratory +1

Mar 20, 2026J/K move · Enter open · S save

NCSTR: Node-Centric Decoupled Spatio-Temporal Reasoning for Video-based Human Pose Estimation

Quang Dang Huynh, Xuefei Yin, Andrew Busch, Hugo G. Espinosa+3

Griffith University

Human Pose Estimation Spatiotemporal GNNs Graph Attention Networks

Open paper AskPDF arXiv

Video-based human pose estimation remains challenged by motion blur, occlusion, and complex spatiotemporal dynamics. Existing methods often rely on heatmaps or implicit spatio-temporal feature aggregation, which limits joint topology expressiveness and weakens cross-frame consistency. To address these problems, we propose a novel node-centric framework that explicitly integrates visual, temporal, and structural reasoning for accurate pose estimation. First, we design a visuo-temporal velocity-based joint embedding that fuses sub-pixel joint cues and inter-frame motion to build appearance- and motion-aware representations. Then, we introduce an attention-driven pose-query encoder, which applies attention over joint-wise heatmaps and frame-wise features to map the joint representations into a pose-aware node space, generating image-conditioned joint-aware node embeddings. Building upon these node embeddings, we propose a dual-branch decoupled spatio-temporal attention graph that models temporal propagation and spatial constraint reasoning in specialized local and global branches. Finally, a node-space expert fusion module is proposed to adaptively fuse the complementary outputs from both branches, integrating local and global cues for final joint predictions. Extensive experiments on three widely used video pose benchmarks demonstrate that our method outperforms state-of-the-art methods. The results highlight the value of explicit node-centric reasoning, offering a new perspective for advancing video-based human pose estimation.

Abstract

Explore similar work

NCSTR: Node-Centric Decoupled Spatio-Temporal Reasoning for Video-based Human Pose Estimation

DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints

VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network