ExMR: Extremity Motion Refinement for Monocular Human Motion Capture

Tom Wehrbein; Bodo Rosenhahn

cs.CVOct 19, 2026

ExMR: Extremity Motion Refinement for Monocular Human Motion Capture

Authors: Tom Wehrbein, Bodo Rosenhahn

Organizations: Institute of Information Processing, Leibniz University Hannover, 30167 Hannover, Germany

Abstract

Detailed hand and foot articulation is important for animation and motion analysis, yet remains challenging to recover from monocular video. Recent expressive video methods infer hand articulation from shared whole-body representations. Our conference work suggests that this underexploits local evidence: the Foot Motion Refinement (FootMR) module refines ankle rotations from foot keypoints, improving reconstruction even when the initial estimator receives the same keypoints. We extend this principle to hands with the Hand Motion Refinement (HandMR) module. Because 2D finger keypoints are often missing or unreliable, HandMR combines image features of a pretrained hand reconstruction model with initial arm and hand rotations to refine elbows, wrists, and fingers. We integrate FootMR and HandMR into ExMR (Extremity Motion Refinement), a system for reconstructing 3D body, hand, and foot motion from monocular video. An initial temporal estimator predicts whole-body motion, after which both refiners revisit local evidence to predict residual joint rotations. Experiments on ARCTIC, MOYO, RICH, and MOOF show that ExMR outperforms state-of-the-art methods in hand and foot reconstruction, reducing hand per-vertex error and ankle joint angle error on MOYO by 19% and 18%, respectively, relative to the strongest competitor. Code will be made available for research purposes. Project page: https://twehrbein.github.io/exmr-website/

Figures & tables

Fig. 1: Monocular motion reconstruction with ExMR. Our work focuses on the articulation of the extremities (the hands and feet) within full-body motion capture.

Fig. 2: Overview of ExMR at inference. Given a monocular video, the initial estimator combines per-frame body and hand image features with 2D body-and-foot keypoints to predict a SMPL-X motion sequence. The HandMR and FootMR modules reuse the corresponding visual evidence and predict residual rotations conditioned on part-specific initial rotations: elbows, wrists, and fingers for HandMR, and knees and ankles for FootMR. HandMR refines elbows, wrists, and fingers, while FootMR refines ankles. The circled plus denotes addition of 6D rotation residuals, with conversion to parent-relative SMPL-X rotations where needed. The initial estimator and both refinement modules are jointly trained temporal transformers. Snowflakes denote frozen pretrained models. Additional inputs are omitted for clarity: the person bounding-box encoding for all three temporal networks and the relative camera rotation for the initial estimator. Insets compare the initial and refined hand and foot estimates.

		ARCTIC				MOYO				RICH
		PA-PVE		PVE		PA-PVE		PVE		PA-PVE		PVE
	Method	All	Hands	All	Hands	All	Hands	All	Hands	All	Hands	All	Hands
per-frame	AiOS [ 36 ]	30.2	19.2	47.1	38.3	$-$	$-$	$-$	$-$	$-$	$-$	$-$	$-$
	Hand4Whole++ [ 12 ]	$-$	$-$	46.0	25.0	$-$	$-$	$-$	$-$	$-$	$-$	$-$	$-$
	SMPLer-X-H [ 7 ]	27.6	18.7	44.6	36.9	$-$	$-$	$-$	$-$	$-$	$-$	$-$	$-$
	released checkpoint *	25.4	18.4	43.8	39.2	53.6	16.8	96.5	64.8	31.5	16.0	69.0	44.7

TABLE I: Comparison of expressive whole-body and hand reconstruction on ARCTIC [ 29 ] , MOYO [ 30 ] , and RICH [ 31 ] . All errors are per-vertex errors in

\mathrm{mm}

(lower is better). DanceHMR-style fusion denotes our locally implemented and trained baseline, not a reproduction of the complete DanceHMR method. ExMR is best on every hand metric and competitive on full-body metrics.

Alternative ways of exploiting HaMeR [ 27 ] :
	ARCTIC				MOYO				RICH
	PA-PVE		PVE		PA-PVE		PVE		PA-PVE		PVE
Method	All	Hands	All	Hands	All	Hands	All	Hands	All	Hands	All	Hands
early fusion (without HandMR)	21.8	7.2	37.4	16.4	37.5	10.3	63.9	39.9	25.4	8.0	63.4	25.4
deeper variant (18 layers)	21.5	7.2	35.8	16.1	37.8	10.5	64.4	40.9	25.4	8.0	63.1	25.7
wider variant (608 dimensions)	21.3	7.1	36.3	16.3	37.7	10.1	64.2	40.6	25.5	8.0	63.8	25.8

TABLE II: Comparison of hand evidence and integration strategies. Early fusion is an otherwise identical ExMR variant with HandMR removed. Its deeper and wider variants approximately match ExMR’s trainable parameter count. ExMR + HaMeR copy-paste replaces ExMR’s final wrist and finger rotations with HaMeR predictions. Per-vertex errors are in

\mathrm{mm}

(lower is better).

Fig. 3: Where HandMR refines the initial estimate. Both panels compare initial and HandMR-refined outputs from one ExMR checkpoint on the same evaluated frames. (a) Severity of high-error predictions. The 95th percentile (P95) of wrist-aligned hand PVE after averaging the two hands’ errors per frame. Percentiles are computed separately for initial and refined outputs. (b) Error reduction versus wrist articulation. Mean per-hand PVE reduction (initial minus refined error) by within-dataset quintile of ground-truth local wrist articulation (Q1: lowest, Q5: highest). Articulation is the

\mathrm{SO}(3)

geodesic distance from the elbow-relative wrist rotation to identity. Each hand’s articulation is paired with its PVE reduction. Positive values indicate improvement.

	ARCTIC			MOYO
Method	MPJVE	Accel	Jitter	MPJVE	Accel	Jitter
SMPLer-X-H * [ 7 ]	204.5	8.4	43.2	196.9	8.3	45.1
SMPLest-X-H * [ 8 ]	206.5	8.5	43.6	187.4	7.8	42.2
SAM3D-Body * [ 9 ]	129.4	6.0	32.4	208.6	10.5	53.4
ExMR + HaMeR copy-paste [ 27 ]	179.2	7.7	40.9	260.3	12.4	68.1
DanceHMR-style fusion [ 20 ]	65.2	1.6	4.6	58.8	1.2	3.5

TABLE III: Temporal accuracy and stability of hand reconstruction. Metrics use 21 wrist-centered joints per hand. MPJVE (

\mathrm{mm/s}

) and Accel (

\mathrm{m/s^{2}}

) are ground-truth-referenced; Jitter (

10\ \mathrm{m/s^{3}}

) is prediction-only. ExMR + HaMeR copy-paste replaces ExMR’s wrist and finger rotations with independent per-frame HaMeR predictions. The final two rows are temporal methods. DanceHMR-style fusion denotes our locally implemented and trained baseline.

Fig. 4: Qualitative examples of extremity refinement in ExMR. Rows show one example each from MOYO (top), RICH (middle), and ARCTIC (bottom). Initial and refined estimates are outputs of the same ExMR checkpoint. Amber circles in the MOYO row highlight the correction of the initial foot orientation.

	Input Joint Rotations				ARCTIC		MOYO
	Finger	Wrist	Elbow	All	PA-PVE	PVE	PA-PVE	PVE
	$\times$	$\times$	$\times$	$\times$	6.6	16.6	9.0	31.5
	$\surd$	$\times$	$\times$	$\times$	6.3	14.8	8.8	27.5
	$\times$	$\surd$	$\times$	$\times$	7.4	15.1	9.8	28.5
	$\surd$	$\surd$	$\times$	$\times$	6.0	14.7	8.6	26.7
$\blacktriangleright$	$\surd$	$\surd$	$\surd$	$\times$	6.1	14.7	8.6	26.5

TABLE IV: Design ablations of the HandMR module. All configurations are retrained from scratch. Top: joint-rotation inputs. The All column adds all remaining body joints. Finger rotations are parent-relative and all others global. Fingers and wrists are predicted residually given initial rotations and directly otherwise. Elbows are refined residually only when supplied. Bottom: variants of the final ExMR configuration (

\blacktriangleright

). Hand PA-PVE and PVE are in

\mathrm{mm}

(lower is better).

	ARCTIC				MOYO				RICH
	Elbow		Wrist		Elbow		Wrist		Elbow		Wrist
Method	G-RE	L-RE	G-RE	L-RE	G-RE	L-RE	G-RE	L-RE	G-RE	L-RE	G-RE	L-RE
without elbow refinement	8.3	9.0	10.9	11.4	18.4	17.5	19.4	21.2	18.4	16.3	17.3	20.0
with elbow refinement (ExMR)	7.4	8.4	11.0	10.4	16.9	15.9	19.4	19.6	16.5	15.4	17.1	18.4

TABLE V: Effect of elbow refinement on elbow and wrist rotation accuracy. Both rows jointly retrain the initial estimator and the FootMR and HandMR modules from scratch. They differ only in whether HandMR predicts residual updates to the elbows. G-RE and L-RE denote global camera-frame and parent-relative geodesic rotation errors, respectively, in degrees.

		MOYO		RICH		MOOF
	Method	AJAE $\downarrow$	N-MPJPE F $\downarrow$	AJAE $\downarrow$	N-MPJPE F $\downarrow$	PCK F 0.05 $\uparrow$	N-FKE 2d $\downarrow$
per-frame	HMR2.0 * [ 35 ]	42.0	55.7	$-$	36.9	76.1	1.95
	ReFit * [ 64 ]	41.8	47.6	$-$	30.3	81.8	1.68
	TokenHMR * [ 83 ]	38.0	48.1	$-$	27.7	74.5	1.85
	CameraHMR * [ 84 ]	31.0	39.1	$-$	24.8	88.3	1.42
	SAM3D-Body * [ 9 ]	24.4	32.6	17.5	21.9	96.6	0.89

TABLE VI: Comparison of foot-pose reconstruction on MOYO [ 30 ] , RICH [ 31 ] , and MOOF [ 21 ] . FootMR (conference) and FootMR (revised) denote complete body-and-foot systems. Both use the same FootMR ankle-refinement module design. Their corresponding GVHMR 23j baselines use matched training and inference configurations without the FootMR module. ExMR adds the HandMR module to the revised system. AJAE is in degrees, N-MPJPE F in

\mathrm{mm}

, PCK F in

\%

, and N-FKE 2d in normalized pixel space.

Configuration	AJAE $\downarrow$	AJAE $\downarrow$	PCK F 0.05 $\uparrow$
	MOYO	RICH	MOOF
FootMR (conference) [ 21 ]	25.9	15.8	92.6
$+$ SAM3D-Body feature & training settings	22.5	14.3	95.0
$+$ revised training data	22.1	13.7	96.2
$+$ revised losses & reweighting	21.4	13.7	97.3
$+$ EMA	21.0	13.1	97.5

TABLE VII: Cumulative changes from FootMR (conference) to FootMR (revised). The FootMR module retains its design and is jointly retrained with the initial estimator for each training configuration. The detector change affects only test-time inputs. Final row: FootMR (revised) in Table VI . EMA: exponential moving average of model parameters. AJAE in degrees and PCK F in

\%

		3DPW (14)				EMDB-1 (24)				RICH (24)
	Method	PA-MPJPE	MPJPE	PVE	Accel	PA-MPJPE	MPJPE	PVE	Accel	PA-MPJPE	MPJPE	PVE	Accel
per-frame	HMR2.0 [ 35 ]	44.4	69.8	82.2	18.1	60.6	98.0	120.3	19.8	48.1	96.0	110.9	18.8
	ReFit [ 64 ]	40.5	65.3	75.1	18.5	58.6	88.0	104.5	20.7	47.9	80.7	92.9	17.1
	TokenHMR [ 83 ]	44.3	71.0	84.6	$-$	55.6	91.7	109.4	$-$	$-$	$-$	$-$	$-$
	CameraHMR [ 84 ]	35.1	56.0	65.9	$-$	43.3	70.2	81.7	$-$	34.0	55.7	64.4	$-$
	NLF-L +fit [ 85 ]	33.6	54.9	63.7	$-$	40.9	68.4	80.6	$-$	$-$	$-$	$-$	$-$

TABLE VIII: Comparison of body-only reconstruction on 3DPW [ 77 ] , EMDB-1 [ 78 ] , and RICH [ 31 ] . Evaluation uses the default hand pose for ExMR and the locally evaluated SAM3D-Body checkpoint. PA-MPJPE, MPJPE, and PVE are in

\mathrm{mm}

, and Accel is in

\mathrm{m/s^{2}}

. Parentheses give the number of joints used for PA-MPJPE, MPJPE, and Accel.

Stage	Time ( $\mathrm{ms/frame}$ )
Person detection and tracking	3.02
SAM3D-Body feature extraction	25.33
HaMeR feature extraction	6.34
2D keypoint detection: Sapiens2-1B	118.62
2D keypoint detection: RTMW-x	2.11
Initial estimator	0.0119

TABLE S1: Runtime breakdown of ExMR with alternative 2D detectors. Times are in milliseconds per video frame for one reconstructed person, measured over complete passes through the video after warm-up. Sapiens2-1B uses the default

1024\times 768

-pixel input resolution. Totals are sums of stage medians, not measured end-to-end times.

	Time ( $\mathrm{ms/frame}$ )		ARCTIC	MOYO			RICH			MOOF	3DPW	EMDB-1
2D detector	Detector	Summed stages	Hand PVE	All PVE	Hand PVE	AJAE	All PVE	Hand PVE	AJAE	N-FKE 2d	MPJPE	MPJPE
Sapiens2-1B ( $1024\times 768$ )	118.62	153.33	14.7	64.2	26.5	20.0	58.4	21.9	12.8	0.67	51.8	64.1
Sapiens2-1B ( $576\times 432$ )	33.06	67.77	14.7	64.2	26.5	19.8	58.5	21.9	12.9	0.67	51.7	64.1
Sapiens2-1B ( $448\times 336$ )	19.77	54.47	14.7	64.1	26.5	19.9	58.6	21.9	13.1	0.68	51.7	64.1
RTMW-x	2.11	36.81	14.7	65.1	26.6	21.8	58.9	22.0	13.7	0.69	51.8	64.8

TABLE S2: Effect of the 2D keypoint detector on reconstruction accuracy and runtime. All configurations use the same trained ExMR checkpoint. Accuracy is evaluated on the benchmark datasets with person and hand boxes held fixed across configurations, following the dataset-specific protocol in Section IV-A . Only the 23 body-and-foot keypoint inputs change. Detector and summed stage costs are measured on the demonstration video following the timing protocol in Table S1 . PVE and MPJPE are in millimeters, AJAE is in degrees, and N-FKE 2d is in normalized pixel space.

Stage	Input	Inference precision	Batch size	Workers	Implementation notes
Detection/tracking	$384\times 640$	FP16	8	—	YOLO11m, ByteTrack; frame prefetching
SAM3D-Body	$512\times 512$	BF16 backbone	32	8	—
HaMeR	$256\times 192$	FP16 autocast	32 hand crops	8	—
Sapiens2-1B	$1024\times 768$	FP16	1	2	FP32 preprocessing; GPU heatmap decoding
Sapiens2-1B	$576\times 432$	FP16	2	2	FP32 preprocessing; GPU heatmap decoding
Sapiens2-1B	$448\times 336$	FP16	4	2	FP32 preprocessing; GPU heatmap decoding

TABLE S3: Implementation settings for inference-time measurements. Spatial input sizes are height

\times

width in pixels. Batch sizes count person crops except where stated. Worker counts refer to data loader workers. A dash denotes a worker count that is not applicable or no additional implementation notes. Frame prefetching supplies decoded video frames from a background worker.

	Input Joint Rotations				MOYO	RICH	MOOF
	Ankle	Knee	Hip	Pelvis	AJAE	AJAE	N-FKE 2d
	$\times$	$\times$	$\times$	$\times$	29.2	19.4	0.63
	$\surd$	$\times$	$\times$	$\times$	20.2	14.5	0.61
	$\times$	$\surd$	$\times$	$\times$	20.9	13.5	0.73
$\blacktriangleright$	$\surd$	$\surd$	$\times$	$\times$	20.0	12.8	0.67
	$\surd$	$\surd$	$\surd$	$\times$	20.1	12.8	0.67

TABLE S4: Ablation of the FootMR module within ExMR. All ExMR configurations are retrained from scratch with otherwise identical settings. Top: initial joint-rotation inputs to FootMR, all expressed in the global camera frame. When no initial ankle rotation is supplied, FootMR directly regresses the global ankle rotation rather than predicting a residual update. Bottom: variants of the

\blacktriangleright

-marked final ExMR configuration. AJAE is in degrees, N-FKE 2d in normalized pixel space (lower is better).

Fig. S1: Qualitative comparison with SAM3D-Body. Each example shows, from left to right, the input image, the SAM3D-Body prediction (gray), and the ExMR prediction (blue). Panels (a), (c), and (e) show object-manipulation examples; (b) and (d) show dance poses; (f) shows a seated pose. Within each example, corresponding insets use identical image regions and magnification. SAM3D-Body overlays show its native MHR meshes, while ExMR overlays show SMPL-X meshes.

3D Human Reconstruction

3D Human Reconstruction recovers human shape, pose, appearance, and motion from images, video, or other sensor data.

See all 103 papers

Mar 10, 2026

Improving 3D Foot Motion Reconstruction in Markerless Monocular Human Motion Capture

State-of-the-art methods can recover accurate overall 3D human body motion from in-the-wild videos. However, they often fail to capture fine-grained articulations, especially in the feet, which are critical for applications such as gait analysis and animation. This limitation results from training datasets with inaccurate foot annotations and limited foot motion diversity. We address this gap with FootMR, a Foot Motion Refinement method that refines foot motion estimated by an existing human recovery model through lifting 2D foot keypoint sequences to 3D. By avoiding direct image input, FootMR circumvents inaccurate image-3D annotation pairs and can instead leverage large-scale motion capture data. To resolve ambiguities of 2D-to-3D lifting, FootMR incorporates knee and foot motion as context and predicts only residual foot motion. Generalization to extreme foot poses is further improved by representing joints in global rather than parent-relative rotations and applying extensive data augmentation. To support evaluation of foot motion reconstruction, we introduce MOOF, a 2D dataset of complex foot movements. Experiments on MOOF, MOYO, and RICH show that FootMR outperforms state-of-the-art methods, reducing ankle joint angle error on MOYO by up to 30% over the best video-based approach.

Tom Wehrbein, Bodo Rosenhahn

L3S - Leibniz University Hannover, Germany

Markerless Motion Capture 3D Human Reconstruction 3D Human Pose Estimation

Ask PDF

May 18, 2026

DanceHMR: Hand-Aware Whole-Body Human Mesh Recovery from Monocular Videos

Monocular video human mesh recovery is essential for digital humans, avatar animation, and embodied simulation, where both temporal stability and expressive whole-body motion are required. Existing video HMR methods produce coherent body motion but often overlook detailed hand articulation, while image-based whole-body methods recover SMPL-X meshes independently per frame, often leading to jittery and inaccurate hand motion. We present a temporally coherent whole-body HMR framework for challenging in-the-wild monocular videos. Our model unifies body context and part-specific hand observations through residual body-hand fusion, enabling stable body motion and detailed hand recovery within a single temporal architecture. We further introduce close-up-aware augmentation to improve robustness under upper-body framing. Experiments on whole-body and body-only benchmarks demonstrate improved hand reconstruction and competitive body accuracy. Our method also produces temporally stable and 2D-consistent SMPL-X motion in challenging real-world videos.

Wenhao Shen, Ming Zhou, Hengyuan Zhang +3

ByteDance Intelligent Creation

3D Hand Pose Estimation Human Mesh Recovery 3D Mesh Reconstruction 3D Human Reconstruction 3D Human Motion Reconstruction

Ask PDF

May 10, 2026

Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Models

Accurate hand and finger tracking from video has significant clinical applications for monitoring activities of daily living and measuring range of motion, yet monocular video approaches for obtaining hand biomechanics remain under-developed. We present a method that combines the SAM 3D Body foundation model with inverse kinematics optimization in a full-body biomechanical model to extract anatomically-constrained finger joint angles from single-view video. We port SAM 3D Body from PyTorch to JAX for integration with MuJoCo-MJX, enabling GPU-accelerated optimization, and develop a novel mapping between the Momentum Human Rig (MHR) outputs and biomechanical model markers. Validation against 8-camera multiview reconstruction on 4,590 frames from 7 participants performing a variety of hand poses and object manipulation tasks shows finger joint angle errors of approximately 10 degrees and hand position errors of approximately 6 mm, after Procrustes alignment. Results were consistent across camera viewpoints and robust to different methods for producing reference values from multiview video. This work extends monocular biomechanical analysis to detailed finger tracking, expanding access to quantitative characterization of hand movement from readily available video.

R. James Cotton, Pouyan Firouzabadi, Wendy Murray

1Shirley Ryan AbilityLab · Department of Physical Medicine and Rehabilitation, Northwestern University · Department of Biomedical Engineering, Northwestern University

Markerless Motion Capture 3D Hand Pose Estimation

Ask PDF

Abstract

3D Human Reconstruction

Explore similar work

Improving 3D Foot Motion Reconstruction in Markerless Monocular Human Motion Capture

DanceHMR: Hand-Aware Whole-Body Human Mesh Recovery from Monocular Videos

Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Models