ER-Pose: Rethinking Keypoint-Driven Representation Learning for Real-Time Human Pose Estimation

Nanjun Li; Pinqi Cheng; Zean Liu; Minghe Tian; Xuanyin Wang

cs.CVMar 9, 2026

ER-Pose: Rethinking Keypoint-Driven Representation Learning for Real-Time Human Pose Estimation

Authors: Nanjun Li, Pinqi Cheng, Zean Liu, Minghe Tian, Xuanyin Wang

Abstract

Single-stage multi-person pose estimation aims to jointly perform human localization and keypoint prediction within a unified framework, offering advantages in inference efficiency and architectural simplicity. Consequently, multi-scale real-time detection architectures, such as YOLO-like models, are widely adopted for real-time pose estimation. However, these approaches typically inherit a box-driven modeling paradigm from object detection, in which pose estimation is implicitly constrained by bounding-box supervision during training. This formulation introduces biases in sample assignment and feature representation, resulting in task misalignment and ultimately limiting pose estimation accuracy. In this work, we revisit box-driven single-stage pose estimation from a keypoint-driven perspective and identify semantic conflicts among parallel objectives as a key source of performance degradation. To address this issue, we propose a keypoint-driven learning paradigm that elevates pose estimation to a primary prediction objective. Specifically, we remove bounding-box prediction and redesign the prediction head to better accommodate the high-dimensional structured representations for pose estimation. We further introduce a keypoint-driven dynamic sample assignment strategy to align training objectives with pose evaluation metrics, enabling dense supervision during training and efficient NMS-free inference. In addition, we propose a smooth OKS-based loss function to stabilize optimization in regression-based pose estimation. Based on these designs, we develop a single-stage multi-person pose estimation framework, termed ER-Pose. On MS COCO and CrowdPose, ER-Pose-n achieves AP improvements of 3.2/6.7 without pre-training and 7.4/4.9 with pre-training respectively compared with the baseline YOLO-Pose. These improvements are achieved with fewer parameters and higher inference efficiency.

Figures & tables

Panel of Figure 1 — Fig. 1 : Performance comparison between ER-Pose and representative human pose estimation methods on the COCO test-dev set. (a) Accuracy vs. inference time. (b) Accuracy vs. number of parameters. ER-Pose demonstrates superior inference efficiency while maintaining competitive accuracy.

Fig. 2 : Illustration of the typical box-driven pipeline: the bounding-box-centered single-stage pose estimation paradigm.

Fig. 3 : Overview of the ER-Pose framework. ER-Pose adopts keypoint-driven single-stage architecture. Feature extraction design follows YOLOv8, with CSPDarkNet [ 57 , 7 ] used as backbone and PAN [ 58 ] employed as neck to produce multi-scale features. Head consists of human confidence branch and pose branch. Confidence branch adopts DSConv [ 59 ] to reduce computational cost, while keypoint regression employs standard convolution to preserve sufficient representational capacity. Network is trained under MAH-SAH scheme.

Fig. 4 : Suboptimal keypoint feature selection induced by box-driven modeling. The upper part shows positive samples selected by the network, while the lower part presents samples from the same network that yield more accurate keypoint estimates but are suppressed during inference.

Panel of Figure 6 — Fig. 6 : Comparison of Gaussian, Laplace, and SOKS distributions.

Method	Backbone	Input Size	Params (M)	Latency (ms)	AP	$AP_{50}$	$AP_{75}$
HigherHRNet [ 21 ]	HRNet-W32 [ 17 ]	512 $\times$ 512	28.6	34.398	66.4	87.5	72.8
CenterNet [ 45 ]	Hourglass-3x [ 23 ]	512 $\times$ 512	194.9	45.855	63.0	86.8	69.6
DEKR [ 51 ]	HRNet-W32 [ 17 ]	512 $\times$ 512	29.6	33.935	68.0	86.7	74.5
DEKR [ 51 ]	HRNet-W48 [ 17 ]	640 $\times$ 512	65.7	36.585	71.0	88.3	77.4
CID [ 47 ]	HRNet-W32 [ 17 ]	512 $\times$ 512	29.4	45.678	68.9	88.5	76.6
ED-Pose [ 42 ]	ResNet-50 [ 63 ]	800 $\times$ 1333	50.6	171.971	69.8	90.2	77.2

TABLE I : Comparison results of ER-Pose with advanced methods on the COCO test-dev set. Inference latency was measured on one Tesla V100 GPU under identical hardware conditions, with evaluation performed using onnxruntime.

Input size	Latency(ms)	AP	$\mathrm{AP}_{50}$	$\mathrm{AP}_{75}$	AR
640	10.587	57.1	84.6	62.0	62.4
800	11.844	59.5	86.2	65.0	64.5
960	13.412	60.9	86.8	68.8	65.9

TABLE II : Comparison of model performance under different input Resolutions

Box branch	Keypoint-driven Assignment	AP	$AP_{50}$	$AP_{75}$	AR
$\checkmark$	$\times$	50.4	79.5	54.2	57.8
$\checkmark$	$\checkmark$	54.7	83.2	58.8	60.2
$\times$	$\checkmark$	57.1	84.6	62.0	62.4

TABLE III : Semantic ablation experiments on the box-based branch

Panel of Figure 10 — Fig. 7 : Gradient heatmaps of confidence prediction branches for models of tiny and small variants. From left to right, results correspond to models using one, two, and three feature scales. (a) Grad-CAM visualization of ER-Pose-n. (b) Grad-CAM visualization of ER-Pose-s.

scale	features	AP	$AP_{m}$	$AP_{l}$	AR	$AR_{m}$	$AR_{l}$
n	{P5}	55.9	46.7	69.1	60.4	51.2	73.2
	{P4, P5}	57.1	49.1	68.8	62.4	54.9	73.0
	{P3, P4, P5}	56.4	47.5	68.9	61.6	53.4	73.0
s	{P5}	64.0	56.1	75.9	68.4	60.5	79.7
s	{P4, P5}	64.0	58.9	72.1	68.6	63.4	75.9
	{P3, P4, P5}	65.7	58.7	75.9	70.5	63.9	79.7

TABLE IV : Branch ablation experiments

Panel of Figure 12 — Fig. 8 : Gradient heatmaps of confidence prediction branches for models of tiny and small variants. From left to right, results correspond to models using one, two, and three feature scales. (a) Grad-CAM visualization of ER-Pose-n. (b) Grad-CAM visualization of ER-Pose-s.

scale	features
n	{P4, P5}
s	{P3, P4, P5}
m	{P3, P4, P5}
l	{P3, P4, P5}

TABLE V : Features maps protocol

IoU thr.	ER-Pose(OKS-IoU)			Box-driven(IoU)
IoU thr.	AP	AR	NMS(ms)	AP	AR	NMS(ms)
0.4	57.1	61.5	7.434	49.9	56.3	0.978
0.5	57.0	61.7	8.769	50.1	56.7	1.085
0.6	56.9	61.8	9.746	50.3	57.1	1.034
0.7	56.6	61.9	12.199	50.4	57.4	1.040
0.8	56.2	62.2	13.791	50.4	57.8	0.951

TABLE VI : Model performance comparison under different positive sample assignment centers and NMS strategies

model	AP	$AP_{50}$	$AP_{75}$	AR	latency(ms)
SAH	53.1	81.8	56.7	58.8	10.587
MAH+NMS(OKS)	56.6	83.4	61.4	60.8	13.544
MAH+SAH	57.1	84.6	62.0	62.4	10.721

TABLE VII : Ablation results of multi-head structures. SAH denotes models trained using single-head assignment, where inference relies exclusively on SAH. Models using only MAH adopt OKS-based NMS during inference to suppress redundant predictions.

Fig. 9 : Grad-CAM visualizations of dual-head structures.

Fig. 10 : Visualization of human target confidence maps.

$\alpha$	$\beta$	AP	$AP_{50}$	$AP_{75}$	AR
0.5	0	50.7	78.9	54.6	56.7
0.5	2	46.2	80.6	46.4	52.0
0.5	4	54.8	82.9	58.8	60.4
0.5	6	57.1	84.4	62.4	62.2
0.5	8	56.7	84.4	61.2	62.1
1.0	0	48.7	77.7	52.2	54.1

TABLE VIII : Comparison experiments under different

\alpha

and

\beta

settings.

Fig. 11 : Task alignment deviation under different

\alpha

and

\beta

settings. Red bars denote alignment deviation of box-driven models, where positive samples are assigned based on box localization. Blue bars represent alignment deviation of ER-Pose-n architecture.

Fig. 12 : Scatter Plot of TAE vs. AP

Fig. 13 : Comparison of task alignment, illustrating effects of keypoint-driven assignment on sample selection.

Loss	ER-Pose-n		ER-Pose-s		ER-Pose-m
Loss	AP	AR	AP	AR	AP	AR
Lap	55.7	60.6	62.2	67.2	68.9	73.0
Gau(OKS)	56.1	61.4	65.3	70.1	69.1	73.3
SOKS	57.1	61.6	65.8	70.2	69.7	73.8

TABLE IX : Performance evaluation under different loss functions.

model	Backbone	Input size	Params(M)	AP	$\mathrm{AP_{50}}$	$\mathrm{AP_{75}}$	$\mathrm{AP_{E}}$	$\mathrm{AP_{M}}$	$\mathrm{AP_{H}}$
SimpleBaseline [ 22 ]	ResNet-101	256 $\times$ 192	34.0	60.8	81.4	65.7	71.4	61.2	51.2
DEKR [ 22 ]	HRNet-W32	512 $\times$ 512	65.7	67.3	86.4	72.2	74.6	68.1	58.7
CID [ 47 ]	HRNet-W32	512 $\times$ 512	29.4	71.2	89.8	76.7	77.9	71.9	63.8
HigherHRNet [ 21 ]	HRNet-W32	512 $\times$ 512	63.8	65.9	86.4	70.6	73.3	66.5	57.9
YOLO-Pose-s [ 6 ]	CSPDarknet	640 $\times$ 640	11.6	61.9	87.6	67.8	69.3	62.5	53.9
YOLO-Pose-m [ 6 ]	CSPDarknet	640 $\times$ 640	26.4	66.6	89.4	73.7	73.8	67.3	58.8

TABLE X : Evaluation on Crowdpose

Panel of Figure 24 — Fig. 14 : Visualization comparison of ER-Pose and advanced regression methods on partial images of CrowdPose. (a) YOLO-Pose-s test results. (b) KAPAO-s test results. (c) ER-Pose-s test results. ER-Pose can better localize humans in complex and occluded scenes, and can achieve better accuracy.

Setting	Dataset	YOLO-Pose-n	ER-POSE-n	$\Delta AP$
From scratch	COCO	47.5	50.7	+3.2
Pretrained	COCO	50.4	57.1	+6.7
From scratch	CrowdPose	45.5	52.9	+7.4
Pretrained	CrowdPose	53.4	58.3	+4.9
Pretrained(COCO)	OCHuman	30.7	37.1	+6.4

TABLE XI : Comparison of AP with and without pretraining on COCO, CrowdPose, and OCHuman, relative to baseline YOLO-Pose.

Fig. 15 : Visualization examples of ER-Pose-n predictions across diverse scenarios.

Human Pose Estimation

Human pose estimation recovers people’s body keypoints, 3D skeletons, or meshes from images, video, and other sensor data.

See all 79 papers

DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints
Jun 16, 2025Sebastian Janampa, Marios PattichisDetection Transformer Efficient ViTs
The University of New Mexico
Yolo-Key-6D: Single Stage Monocular 6D Pose Estimation with Keypoint Enhancements
Mar 4, 2026Kemal Alperen Çetiner, Hazım Kemal EkenelObject Pose Estimation
ASELSAN, Aerospace Technologies, Türkiye · Istanbul Technical University, Department of Computer Engineering, Türkiye · New York University Abu Dhabi, Division of Engineering, UAE
TinyDETR-Pose: Towards End-to-End Real-Time Single-Stage 6DoF Object Pose Estimation with Lightweight Transformers
Aug 15, 2026Paul Julius Kühn, Duc Anh Nguyen, Saptarshi Neil Sinha +2Object Pose Estimation Edge Inference
Fraunhofer IGD 64283 Darmstadt, Germany · TU Darmstadt, 64289 Darmstadt, Germany · TU Delft 2628 CD Delft, Netherlands

Jun 16, 2025J/K move · Enter open · S save

DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints

Sebastian Janampa, Marios Pattichis

The University of New Mexico

Detection Transformer Efficient ViTs Human Pose Estimation Keypoint Detection

Open paper AskPDF arXiv

Multi-person pose estimation (MPPE), which involves detecting body joint positions (keypoints) for every person in an image, is a fundamental task in computer vision. Despite recent advances, no transformer-based model currently achieves real-time performance. This work addresses the latency challenge by introducing DETRPose, the first family of real-time, end-to-end transformer models for multi-person 2D pose estimation. DETRPose significantly enhances the GroupPose decoder, enabling real-time inference. For training, a novel denoising keypoint technique is proposed to accelerate convergence. The varifocal loss is also extended for keypoints, termed Keypoint Similarity VariFocal loss, to improve query quality. Extensive evaluation demonstrates that DETRPose models achieve accuracy comparable to or exceeding that of leading alternatives while requiring five to ten times fewer training epochs. DETRPose-S matches the accuracy of YOLOv8-Pose-X and YOLO11-Pose-X on the COCO dataset (67.0 vs 67.3 and 67.2 in AP) with 81% fewer parameters (11.5M vs 69.4M and 58.8M) and 52% faster inference speed (2.39ms vs 5.23ms and 4.93ms). On the CrowdPose dataset, DETRPose-X has

13.1\times

fewer FLOPs (232.3G vs 3048.1G) and only

2%

fewer precision (75.1 vs 76.6 in AP) than ED-Pose-SwinL-5S. On the OCHuman dataset, DETRPose-S surpasses all previous models, showing the robustness of DETRPose on out-of-distribution datasets. Code is available at https://github.com/SebastianJanampa/DETRPose

Abstract

Human Pose Estimation

Explore similar work

DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints

Yolo-Key-6D: Single Stage Monocular 6D Pose Estimation with Keypoint Enhancements

TinyDETR-Pose: Towards End-to-End Real-Time Single-Stage 6DoF Object Pose Estimation with Lightweight Transformers