VGGT-HPE: Reframing Head Pose Estimation as Relative Pose Prediction
Authors: Vasiliki Vasileiou, Panagiotis P. Filntisis, Petros Maragos, Kostas Daniilidis
Organizations: Archimedes, Athena Research Center, Marousi, Greece · HERON – Hellenic Robotics Center of Excellence, Athens, Greece · School of ECE, National Technical University of Athens, Greece · Robotics Institute, Athena Research Center, Marousi, Greece · University of Pennsylvania
Monocular head pose estimation is traditionally formulated as direct regression from a single image to an absolute pose. This paradigm forces the network to implicitly internalize a dataset-specific canonical reference frame. In this work, we argue that predicting the relative rigid transformation between two observed head configurations is a fundamentally easier and more robust formulation. We introduce VGGT-HPE, a relative head pose estimator built upon a general-purpose geometry foundation model. Finetuned exclusively on synthetic facial renderings, our method sidesteps the need for an implicit anchor by reducing the problem to estimating a geometric displacement from an explicitly provided anchor with a known pose. As a practical benefit, the relative formulation also allows the anchor to be chosen at test time - for instance, a near-neutral frame or a temporally adjacent one - so that the prediction difficulty can be controlled by the application. Despite zero real-world training data, VGGT-HPE achieves state-of-the-art results on the BIWI benchmark, outperforming established absolute regression methods trained on mixed and real datasets. Through controlled easy- and hard-pair benchmarks, we also systematically validate our core hypothesis: relative prediction is intrinsically more accurate than absolute regression, with the advantage scaling alongside the difficulty of the target pose. Project page and code: https://vasilikivas.github.io/VGGT-HPE
Figures & tables
Figure 1: Absolute methods (top) regress pose from a single image relative to an implicit canonical frame learned during training, making them brittle under domain shift and extreme poses. VGGT-HPE (bottom) predicts the relative transformation between an anchor with known pose and a query, recovering the target pose by composition. This sidesteps the need for an implicit reference frame, transfers better from synthetic to real data, and allows the anchor to be chosen at test time to control prediction difficulty.
Figure 2: Overview of the VGGT-HPE architecture. The model takes an anchor image ( Ia ) and a query image ( Iq ) to predict the relative rigid transformation ( Tq←a ) between them. We use a frozen, pre-trained VGGT backbone efficiently fine-tuned with LoRA for the facial domain. The final absolute query pose ( T^q ) is recovered by composing the predicted relative displacement with the known anchor pose.
Figure 3: Samples from our synthetic training set. Each row shows a single identity rendered under varying viewpoints, expressions, and HDRI environment maps.
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Data
Reported numbers
Dlib [ 19 ]
11.86
13.00
19.56
14.81
R
3DDFA [ 45 ]
5.50
41.90
13.22
19.07
M
EVA-GCN [ 38 ]
4.01
4.78
2.98
3.92
M
HopeNet [ 29 ]
4.81
6.61
3.27
4.89
M
QuatNet [ 14 ]
4.01
5.49
2.94
4.15
M
Table 1: Cross-domain evaluation on BIWI. Rotation errors in degrees (lower is better). Best in bold , second-best underlined . Top: numbers reported in original papers (from [ 6 ] ). Bottom: reproduced under our shared MTCNN detection protocol. “Data” indicates the training-data regime: synthetic-only (S), real-only (R), or mixed (M; real images combined with synthetic augmentation / synthetic pose expansion, e.g., 300W-LP).
Figure 4: Qualitative results on BIWI. Each row shows a different subject. From left to right: the query frame, the anchor frame with its known pose overlay, the ground-truth pose, our prediction (VGGT-HPE), and three baselines (6DRepNet, TokenHPE, TRG). Our method produces pose estimates that are visually closer to the ground truth across a range of head orientations, including challenging near-profile and upward-looking poses where absolute methods exhibit larger deviations.
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Data
VGGT-HPE-Abs
40.74
18.65
33.66
31.02
S
TokenHPE [ 40 ]
21.85
26.35
19.34
22.51
M
TRG [ 6 ]
8.95
33.88
8.87
17.23
M
6DRepNet [ 13 ]
14.27
18.91
6.81
13.33
M
VGGT-HPE (Rel., ours)
3.81
15.87
6.93
8.87
S
Table 2: BIWI hard benchmark (360 neutral-anchor / extreme-query pairs). Rotation errors in degrees (lower is better). Best in bold , second-best underlined . “Data” indicates the training regime (S: synthetic-only, M: mixed) as defined in Table 1 .
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Data
VGGT-HPE-Abs
5.62
2.26
4.06
3.98
S
TokenHPE [ 40 ]
3.41
5.99
1.43
3.61
M
TRG [ 6 ]
3.59
4.24
2.52
3.45
M
6DRepNet [ 13 ]
2.31
4.93
1.14
2.80
M
VGGT-HPE (Rel., ours)
1.17
0.74
0.97
0.96
S
Table 3: BIWI easy neutral-anchor benchmark (360 pairs; pair delta mean: 3.82∘ ). Rotation errors in degrees (lower is better). Best in bold , second-best underlined . “Data” indicates the training regime (S: synthetic-only, M: mixed).
Synthetic
BIWI
Variant
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
Adaptation strategy
Full finetune
38.00
33.76
31.12
34.29
23.07
17.90
10.05
17.00
From scratch
9.21
15.07
14.05
12.78
7.71
8.12
6.94
7.59
Head-only
3.82
6.50
5.89
5.40
18.08
15.17
8.25
13.83
LoRA (ours)
2.46
4.65
4.51
3.87
2.24
3.04
3.17
2.82
Table 4: Ablation study on synthetic validation data and BIWI (cross-domain). Rotation errors in degrees, sorted by decreasing MAE on BIWI. Best in bold , second-best underlined . The absolute single-image variant (Abs. Single) fits the synthetic distribution best but transfers worst to BIWI, while the full relative model (VGGT-HPE) reverses this ranking — evidence that the relative formulation transfers better across domains.
Figure 5: BIWI neutral-anchor evaluation as a function of anchor-query rotation gap. The upper plot reports rotation MAE, while the lower band shows the number of sampled pairs per bin. VGGT-HPE remains the strongest method across the full range.
Figure 6: BIWI query-pose evaluation as a function of absolute query pose. For VGGT-HPE, each query is paired with a same-subject anchor whose anchor-query geodesic gap is below 5∘ . The upper plot reports rotation MAE, while the lower band shows the number of sampled pairs per bin.
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
VGGT-HPE (Rel., GT anchor)
2.24
3.04
3.17
2.82
VGGT-HPE (Rel., VGGT-HPE-Abs anchor)
4.56
5.28
3.42
4.42
VGGT-HPE (Rel., 6DRepNet anchor)
2.89
6.05
3.29
4.08
VGGT-HPE (Rel., TokenHPE anchor)
3.35
7.30
3.91
4.85
VGGT-HPE (Rel., TRG anchor)
4.02
12.03
7.13
7.73
Table 5: Relaxing the ground-truth anchor assumption on the full BIWI set. The anchor pose is provided by an external absolute model instead of ground truth. Rotation errors in degrees (lower is better).
Method
Yaw ↓
Pitch ↓
Roll ↓
MAE ↓
VGGT-HPE (Rel., GT anchor)
3.81
15.87
6.93
8.87
VGGT-HPE (Rel., VGGT-HPE-Abs anchor)
6.93
17.36
8.68
10.99
VGGT-HPE (Rel., 6DRepNet anchor)
4.81
19.36
6.62
10.26
VGGT-HPE (Rel., TokenHPE anchor)
5.92
20.31
7.44
11.22
VGGT-HPE (Rel., TRG anchor)
7.07
21.82
17.00
15.30
Table 6: Relaxing the ground-truth anchor assumption on the hard BIWI subset (360 neutral-anchor / extreme-query pairs). The anchor pose is provided by an external absolute model instead of ground truth. Rotation errors in degrees (lower is better).