VGGT-HPE: Reframing Head Pose Estimation as Relative Pose Prediction
Authors: Vasiliki Vasileiou, Panagiotis P. Filntisis, Petros Maragos, Kostas Daniilidis
Organizations: Archimedes, Athena Research Center, Marousi, Greece · HERON – Hellenic Robotics Center of Excellence, Athens, Greece · School of ECE, National Technical University of Athens, Greece · Robotics Institute, Athena Research Center, Marousi, Greece · University of Pennsylvania
Monocular head pose estimation is traditionally formulated as direct regression from a single image to an absolute pose. This paradigm forces the network to implicitly internalize a dataset-specific canonical reference frame. In this work, we argue that predicting the relative rigid transformation between two observed head configurations is a fundamentally easier and more robust formulation. We introduce VGGT-HPE, a relative head pose estimator built upon a general-purpose geometry foundation model. Finetuned exclusively on synthetic facial renderings, our method sidesteps the need for an implicit anchor by reducing the problem to estimating a geometric displacement from an explicitly provided anchor with a known pose. As a practical benefit, the relative formulation also allows the anchor to be chosen at test time - for instance, a near-neutral frame or a temporally adjacent one - so that the prediction difficulty can be controlled by the application. Despite zero real-world training data, VGGT-HPE achieves state-of-the-art results on the BIWI benchmark, outperforming established absolute regression methods trained on mixed and real datasets. Through controlled easy- and hard-pair benchmarks, we also systematically validate our core hypothesis: relative prediction is intrinsically more accurate than absolute regression, with the advantage scaling alongside the difficulty of the target pose. Project page and code: https://vasilikivas.github.io/VGGT-HPE
Figures & tables
Figure 1: Absolute methods (top) regress pose from a single image relative to an implicit canonical frame learned during training, making them brittle under domain shift and extreme poses. VGGT-HPE (bottom) predicts the relative transformation between an anchor with known pose and a query, recovering the target pose by composition. This sidesteps the need for an implicit reference frame, transfers better from synthetic to real data, and allows the anchor to be chosen at test time to control prediction difficulty.
Figure 2: Overview of the VGGT-HPE architecture. The model takes an anchor image ( Ia ) and a query image ( Iq ) to predict the relative rigid transformation ( Tq←a ) between them. We use a frozen, pre-trained VGGT backbone efficiently fine-tuned with LoRA for the facial domain. The final absolute query pose ( T^q ) is recovered by composing the predicted relative displacement with the known anchor pose.
Figure 3: Samples from our synthetic training set. Each row shows a single identity rendered under varying viewpoints, expressions, and HDRI environment maps.
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Data
Reported numbers
Dlib [ 19 ]
11.86
13.00
19.56
14.81
R
3DDFA [ 45 ]
5.50
41.90
13.22
19.07
M
EVA-GCN [ 38 ]
4.01
4.78
2.98
3.92
M
HopeNet [ 29 ]
4.81
6.61
3.27
4.89
M
QuatNet [ 14 ]
4.01
5.49
2.94
4.15
M
Table 1: Cross-domain evaluation on BIWI. Rotation errors in degrees (lower is better). Best in bold , second-best underlined . Top: numbers reported in original papers (from [ 6 ] ). Bottom: reproduced under our shared MTCNN detection protocol. “Data” indicates the training-data regime: synthetic-only (S), real-only (R), or mixed (M; real images combined with synthetic augmentation / synthetic pose expansion, e.g., 300W-LP).
Figure 4: Qualitative results on BIWI. Each row shows a different subject. From left to right: the query frame, the anchor frame with its known pose overlay, the ground-truth pose, our prediction (VGGT-HPE), and three baselines (6DRepNet, TokenHPE, TRG). Our method produces pose estimates that are visually closer to the ground truth across a range of head orientations, including challenging near-profile and upward-looking poses where absolute methods exhibit larger deviations.
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Data
VGGT-HPE-Abs
40.74
18.65
33.66
31.02
S
TokenHPE [ 40 ]
21.85
26.35
19.34
22.51
M
TRG [ 6 ]
8.95
33.88
8.87
17.23
M
6DRepNet [ 13 ]
14.27
18.91
6.81
13.33
M
VGGT-HPE (Rel., ours)
3.81
15.87
6.93
8.87
S
Table 2: BIWI hard benchmark (360 neutral-anchor / extreme-query pairs). Rotation errors in degrees (lower is better). Best in bold , second-best underlined . “Data” indicates the training regime (S: synthetic-only, M: mixed) as defined in Table 1 .
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Data
VGGT-HPE-Abs
5.62
2.26
4.06
3.98
S
TokenHPE [ 40 ]
3.41
5.99
1.43
3.61
M
TRG [ 6 ]
3.59
4.24
2.52
3.45
M
6DRepNet [ 13 ]
2.31
4.93
1.14
2.80
M
VGGT-HPE (Rel., ours)
1.17
0.74
0.97
0.96
S
Table 3: BIWI easy neutral-anchor benchmark (360 pairs; pair delta mean: 3.82∘ ). Rotation errors in degrees (lower is better). Best in bold , second-best underlined . “Data” indicates the training regime (S: synthetic-only, M: mixed).
Synthetic
BIWI
Variant
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Adaptation strategy
Full finetune
38.00
33.76
31.12
34.29
23.07
17.90
10.05
17.00
From scratch
9.21
15.07
14.05
12.78
7.71
8.12
6.94
7.59
Head-only
3.82
6.50
5.89
5.40
18.08
15.17
8.25
13.83
LoRA (ours)
2.46
4.65
4.51
3.87
2.24
3.04
3.17
2.82
Table 4: Ablation study on synthetic validation data and BIWI (cross-domain). Rotation errors in degrees, sorted by decreasing MAE on BIWI. Best in bold , second-best underlined . The absolute single-image variant (Abs. Single) fits the synthetic distribution best but transfers worst to BIWI, while the full relative model (VGGT-HPE) reverses this ranking — evidence that the relative formulation transfers better across domains.
Figure 5: BIWI neutral-anchor evaluation as a function of anchor-query rotation gap. The upper plot reports rotation MAE, while the lower band shows the number of sampled pairs per bin. VGGT-HPE remains the strongest method across the full range.
Figure 6: BIWI query-pose evaluation as a function of absolute query pose. For VGGT-HPE, each query is paired with a same-subject anchor whose anchor-query geodesic gap is below 5∘ . The upper plot reports rotation MAE, while the lower band shows the number of sampled pairs per bin.
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
VGGT-HPE (Rel., GT anchor)
2.24
3.04
3.17
2.82
VGGT-HPE (Rel., VGGT-HPE-Abs anchor)
4.56
5.28
3.42
4.42
VGGT-HPE (Rel., 6DRepNet anchor)
2.89
6.05
3.29
4.08
VGGT-HPE (Rel., TokenHPE anchor)
3.35
7.30
3.91
4.85
VGGT-HPE (Rel., TRG anchor)
4.02
12.03
7.13
7.73
Table 5: Relaxing the ground-truth anchor assumption on the full BIWI set. The anchor pose is provided by an external absolute model instead of ground truth. Rotation errors in degrees (lower is better).
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
VGGT-HPE (Rel., GT anchor)
3.81
15.87
6.93
8.87
VGGT-HPE (Rel., VGGT-HPE-Abs anchor)
6.93
17.36
8.68
10.99
VGGT-HPE (Rel., 6DRepNet anchor)
4.81
19.36
6.62
10.26
VGGT-HPE (Rel., TokenHPE anchor)
5.92
20.31
7.44
11.22
VGGT-HPE (Rel., TRG anchor)
7.07
21.82
17.00
15.30
Table 6: Relaxing the ground-truth anchor assumption on the hard BIWI subset (360 neutral-anchor / extreme-query pairs). The anchor pose is provided by an external absolute model instead of ground truth. Rotation errors in degrees (lower is better).
Relative pose estimation is fundamental for SLAM, visual localization, and 3D reconstruction. Existing Relative Pose Regression (RPR) methods face a key trade-off: feature-matching pipelines achieve high accuracy but block gradient flow via non-differentiable RANSAC, while ViT-based regressors are end-to-end trainable but prohibitively expensive for real-time deployment. We identify the core bottlenecks as the coupling between rotation and translation estimation and insufficient cross-view feature alignment. We propose IUP-Pose, a geometry-driven decoupled iterative framework with implicit dense alignment. A lightweight Multi-Head Bi-Cross Attention (MHBC) module aligns cross-view features without explicit matching supervision. The aligned features are processed by a decoupled rotation-translation pipeline: two shared-parameter rotation stages iteratively refine rotation with uncertainty, and feature maps are realigned via rotational homography H_inf before translation prediction. IUP-Pose achieves 73.3% AUC@20deg on MegaDepth1500 with full end-to-end differentiability, 70 FPS throughput, and only 37M parameters, demonstrating a favorable accuracy-efficiency trade-off for real-time edge deployment.
Head-mounted stereo fisheye cameras provide a compact view of the wearer's body, but strong distortion, partial visibility and a short baseline make 3D pose estimation difficult. Existing refinement decoders sample image features independently in each view, without requiring the samples to correspond to a common 3D hypothesis. We introduce EgoRayFormer, a causal pose estimator that couples stereo feature sampling and geometric refinement in ray space. For each joint, the decoder projects shared 3D hypotheses along and around its viewing ray into both images. A geometric branch then corrects the projected joint locations and intersects the resulting rays; a learned gate blends this geometric estimate with a learned pose update. This formulation introduces an explicit stereo constraint while retaining flexibility when the rays provide weak depth constraints. In local evaluations on the official device-relative test splits, EgoRayFormer achieves three-seed mean MPJPEs of 17.64 mm on UnrealEgo2 and 49.55 mm on UnrealEgo-RW, improving to 47.57 mm with UnrealEgo2 pre-training. In paired validation ablations, removing the geometric branch increases MPJPE in all three seeds at both input resolutions. With a common ResNet-18 encoder and input resolution, our system reduces RW test MPJPE by 9.09 mm relative to the device-relative EgoPoseFormer baseline. A separate protocol analysis quantifies the difference between device-relative and oracle-root evaluation.
Object 6D pose estimation formulations have progressively reduced reliance on object-specific priors, evolving from explicit 3D models to multi-view object captures to single reference images. We take this progression to its extreme by introducing prior-free relative 6D pose estimation, which lifts the assumption of knowing which object is to be posed within the scene. This novel setting aims to estimate the relative poses of multiple instances of an unknown object within the same image, without requiring CAD models, templates, or reference images. We solve this by formulating a novel method (PROSE) that finds coarse correspondences between object instances using multimodal foundation features, thus requiring no training. We refine these correspondences by imposing cycle consistency across tuples of instances, and leverage the resulting globally consistent correspondences to estimate the relative 6D pose between any pair of instances. To enable systematic evaluation, we design a novel benchmark (PRENCH) built from three multi-instance BOP datasets and enriched with task-specific metadata. PROSE consistently outperforms baselines obtained by adapting state-of-the-art single-image methods to the proposed setting, while requiring neither task-specific supervision nor additional learned components. Project website: https://tev-fbk.github.io/PROSE/
Behdad Khodabandehloo, Andrea Caraffa, Davide Boscaini +1
Fondazione Bruno Kessler, Trento, Italy · University of Trento, Trento, Italy