Efficient video human pose estimation must balance localisation accuracy, motion quality, and energy cost. We present SpikePose, a lightweight video pose estimator that pairs a frame-reset, single-step spiking encoder with motion-aligned heatmap memory (MAM). The encoder supplies per-frame spatial evidence, while MAM estimates joint displacement, aligns historical heatmaps, and adaptively fuses them with current predictions. By confining temporal reasoning to a compact, lightweight module outside the spiking backbone, SpikePose helps bridge the gap between spiking and classical Artificial Neural Networks (ANN) on large-scale, non-event-driven video tasks. We evaluate spatial accuracy, motion errors, and estimated arithmetic energy on MPII and a fixed NTU60 Cross-Subject subset. Against the architecture-matched ANN with MAM, SpikePose cuts estimated arithmetic energy by 91.4% at a cost of just 1.12 points in head-bone-normalised PCK at 0.5 threshold (PCK_HB(0.5)). Within SpikePose, adding MAM reduces velocity and acceleration errors by 28.13% and 31.46% while slightly improving localisation, at a cost of only 0.003M parameters and 0.0023 analytical GFLOPs per frame. Code will be released on GitHub upon acceptance.
Figures & tables
Figure 1: Overview of the proposed SpikePose pipeline. The frame-reset spiking encoder extracts per-frame spatial evidence, while MAM performs compact cross-frame motion estimation, heatmap alignment, memory update, and residual decoding.
Method
Type
PCKHB(0.5)↑
MPJVE ↓
MPJAccE ↓
GFLOPs/frame ↓
Eproxy (mJ/frame) ↓
Params (M) ↓
ResNet-50
ANN
85.81
3.79
6.06
14.555
33.277
34.000
HRNet-W32
ANN
86.35
3.67
5.87
20.528
46.889
28.540
SpikePose–ANN w/o MAM
ANN
84.92
3.11
4.81
8.443
19.309
7.518
SpikePose–ANN
ANN
84.93
3.03
4.69
8.445
19.310
7.523
SpikeYOLO–Frame
SNN
84.73
4.88
8.06
17.097
6.628
16.739
SpikePose w/o MAM (Ours)
SNN
83.63
4.87
8.02
8.459
1.627
7.520
Table 1: Results on the frozen NTU60 Cross-Subject test subset. All rows use a complete video and MAM state reset only at video boundaries, as described in Sec. 3 . MPJVE and MPJAccE average videos equally. GFLOPs and Eproxy are normalized per processed frame; parameters are in millions. Bold marks the best valid ANN and SNN result separately.
Method
Model
PCKh ↑
GFLOPs ↓
Eproxy↓
Params ↓
SpikePose–ANN w/o MAM
ANN
87.05
8.44
19.309
7.518
ResNet-50 [ 2 ]
ANN
88.54
14.56
33.277
33.999
HRNet-W32 [ 3 ]
ANN
90.32
20.53
46.889
28.536
SpikeYOLO [ 16 ]
SNN
87.79
17.10
6.579
16.739
SpikePose w/o MAM ( Tsnn=1 )
SNN
85.84
8.46
1.820
7.518
SpikePose w/o MAM ( Tsnn=2 )
SNN
86.16
16.92
2.984
7.518
Table 2: MPII validation comparison. PCKh uses flip testing. GFLOPs and Eproxy are per 256×256 person crop, including all Tsnn updates and excluding flip testing. All arithmetic-energy estimates use the same MAC/AC accounting convention. PCK: PCKh (0.5) ; Parameters are in millions.
Schedule
PCKHB(0.5)↑
MPJVE ↓
MPJAccE ↓
Eproxy↓
V1U1
85.08
5.30
8.81
1.788
V1U2
85.09
5.20
8.60
2.982
V1U4
84.75
4.94
8.13
5.616
V2U2
83.03
5.32
8.73
2.969
V2U4
83.14
4.89
7.95
5.564
V4U4
80.66
6.22
9.74
5.602
Table 3: Frame/update controls for the SpikePose spatial encoder on NTU60. VmUn uses m distinct physical frames in n SNN updates. Frames are repeated uniformly when n>m .
MAM components
NTU60 test subset
Motion alignment
Adaptive gate
Residual fusion
PCKHB(0.5)↑
MPJAccE ↓
×
×
×
83.63
8.02
×
✓
✓
83.77
5.86
✓
×
✓
83.75
5.66
✓
✓
×
83.85
5.53
✓
✓
✓
83.81
5.50
Table 4: Core MAM-component ablation on the frozen NTU60 test subset with state reset only at source-video boundaries.
Overall
AE by speed
Method
PCK ↑
VE ↓
AE ↓
L ↓
M ↓
H ↓
Base
83.63
4.87
8.02
4.81
6.77
12.48
+ EMA
83.69
4.15
6.60
3.70
5.51
10.59
+ One Euro
83.78
3.61
5.64
2.55
4.46
9.90
+ SG
83.60
4.47
6.91
3.91
5.77
11.03
SpikePose
83.81
3.50
5.50
2.68
4.48
9.33
Table 5: Filtering comparison on NTU60. Base is SpikePose without MAM; + rows apply coordinate filters to Base. PCK: PCKHB(0.5) ; VE: MPJVE; AE: MPJAccE. L/M/H: low/medium/high motion-speed strata (1,521 videos each). EMA: exponential moving average [ 22 ] ; One Euro: One Euro filter [ 23 ] ; SG: Savitzky–Golay filter [ 24 ] .
Video-based human pose estimation remains challenged by motion blur, occlusion, and complex spatiotemporal dynamics. Existing methods often rely on heatmaps or implicit spatio-temporal feature aggregation, which limits joint topology expressiveness and weakens cross-frame consistency. To address these problems, we propose a novel node-centric framework that explicitly integrates visual, temporal, and structural reasoning for accurate pose estimation. First, we design a visuo-temporal velocity-based joint embedding that fuses sub-pixel joint cues and inter-frame motion to build appearance- and motion-aware representations. Then, we introduce an attention-driven pose-query encoder, which applies attention over joint-wise heatmaps and frame-wise features to map the joint representations into a pose-aware node space, generating image-conditioned joint-aware node embeddings. Building upon these node embeddings, we propose a dual-branch decoupled spatio-temporal attention graph that models temporal propagation and spatial constraint reasoning in specialized local and global branches. Finally, a node-space expert fusion module is proposed to adaptively fuse the complementary outputs from both branches, integrating local and global cues for final joint predictions. Extensive experiments on three widely used video pose benchmarks demonstrate that our method outperforms state-of-the-art methods. The results highlight the value of explicit node-centric reasoning, offering a new perspective for advancing video-based human pose estimation.
Vision Transformers (ViTs) have recently achieved state-of-the-art performance in 2D human pose estimation due to their strong global modeling capability. However, existing ViT-based pose estimators are designed for static images and process each frame independently, thereby ignoring the temporal coherence that exists in video sequences. This limitation often results in unstable predictions, especially in challenging scenes involving motion blur, occlusion, or defocus. In this paper, we propose TAR-ViTPose, a novel Temporal Aggregate-and-Restore Vision Transformer tailored for video-based 2D human pose estimation. TAR-ViTPose enhances static ViT representations by aggregating temporal cues across frames in a plug-and-play manner, leading to more robust and accurate pose estimation. To effectively aggregate joint-specific features that are temporally aligned across frames, we introduce a joint-centric temporal aggregation (JTA) that assigns each joint a learnable query token to selectively attend to its corresponding regions from neighboring frames. Furthermore, we develop a global restoring attention (GRA) to restore the aggregated temporal features back into the token sequence of the current frame, enriching its pose representation while fully preserving global context for precise keypoint localization. Extensive experiments demonstrate that TAR-ViTPose substantially improves upon the single-frame baseline ViTPose, achieving a +2.3 mAP gain on the PoseTrack2017 benchmark. Moreover, our approach outperforms existing state-of-the-art video-based methods, while also achieving a noticeably higher runtime frame rate in real-world applications. Project page: https://github.com/zgspose/TARViTPose.
Camera pose matters. The position and orientation of each viewpoint define a shared spatial coordinate frame that relates observations across video frames. Yet this signal is largely absent from multimodal LLMs (MLLMs) for video understanding, which process frames as isolated 2D snapshots, instead of the persistent scene humans perceive. We revisit pose as a lightweight supervisory signal and introduce Cambrian-P, a video MLLM augmented with per-frame learnable camera tokens and a pose regression head. With a carefully designed sampling scheme, the model achieves substantial gains of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, generalizes across eight additional spatial and general video QA benchmarks, and, as a byproduct, achieves state of the art streaming pose estimation on ScanNet. Surprisingly, training on pseudo-annotated poses from in-the-wild video further improves general video QA benchmarks, showing pose helps beyond spatial reasoning. Together, these results position camera pose as a fundamental signal for video models that reason about the physical world.