Does VGGT Know What It Does Not Know?

Adityan Thirumani Srinivasan; David Nordström

cs.CVOct 26, 2026

Does VGGT Know What It Does Not Know?

Authors: Adityan Thirumani Srinivasan, David Nordström

Organizations: University of Wisconsin–Madison · Chalmers University of Technology

Abstract

Feed-forward 3D reconstruction models such as VGGT regress cameras and geometry in a single forward pass, but they report no uncertainty for the camera poses they predict: an incorrect relative pose looks no different from a correct one. We ask whether the frozen model nonetheless knows when its pose is wrong. On HardMatch, a benchmark of 1,000 deliberately difficult image pairs on which VGGT fails on 35% of pairs, a linear probe on VGGT's frozen features predicts pose failure on held-out categories with ROC-AUC 93.5%, compared with 79.7% for VGGT's own point-map confidence and 77.7% for a classifier trained on all of VGGT's outputs. The probe catches failures VGGT is confident about, lets a downstream system abstain on the pairs it distrusts, and transfers without retraining to rotation failures on PhotoTourism and ScanNet-1500, beyond what co-visibility explains. This knowledge is absent from the per-image encoder: it is computed partway through the layers that fuse the two views, and the same holds for a second model, $π^3$ . In essence, we find that VGGT knows more about its own failures than it reports.

Figures & tables

Dataset	Pairs	GT	Failure	Rate
HardMatch [ 18 ]	903	corresp.	epi. $>50$ px	$34.9\%$
PhotoTourism [ 10 ]	1,262	SfM	$e_{R}>15^{\circ}$	$1.7\%$
ScanNet-1500 [ 22 ]	1,500	RGB-D	$e_{\text{pose}}>15^{\circ}$	$7.7\%$

Table 1: Datasets. Ground truth (hand-annotated correspondences, about

11

per pair, or poses), the failure definition used for each, and VGGT’s failure rate on image pairs.

Figure 2: The probe catches failures VGGT is confident about. Top: first image with hand-annotated points. Bottom: second image with the epipolar lines implied by VGGT’s own predicted cameras (same colour as the point); for a correct pose each point lies on its line. “conf.”: percentile of VGGT’s point-map confidence among all pairs (higher = more confident); “probe”: out-of-fold failure probability. (a)–(c) VGGT is confident but wrong (aerial vs . ground view, large viewpoint change, photograph vs . drawing) and the probe flags it. (d) VGGT is unconfident but correct, and the probe does not flag it. (e) A miss: extreme zoom on a confidently wrong pair.

VGGT’s own outputs
Predictor	ROC-AUC (%)	AP (%)
Point-map conf. (min of pair)	$79.7$ [76.6, 82.4]	$60.4$
Depth conf. (min of pair)	$66.5$ [62.8, 69.9]	$47.5$
Image sharpness (min)	$57.7$ [53.2, 62.1]	$44.0$
Predicted rel. rotation	$62.9$ [58.5, 67.1]	$43.1$
Predicted focal ratio	$51.1$ [46.6, 55.4]	$37.1$

Table 2: VGGT’s features know more about its failures than anything it outputs. Predicting VGGT’s pose failures on HardMatch (failure: median epipolar error

>50

px). ROC-AUC (%) with

95\%

category-bootstrap CI, and average precision (AP, %). All learned rows use the same category-grouped cross-validation.

Threshold	Fail rate	Probe	Point conf.	All outputs
$10$ px	$64\%$	$\mathbf{91.1}$	$72.8$	$73.8$
$20$ px	$51\%$	$\mathbf{92.9}$	$76.0$	$76.6$
$50$ px	$35\%$	$\mathbf{93.5}$	$79.7$	$77.7$
$100$ px	$25\%$	$\mathbf{92.4}$	$77.8$	$76.9$
Controls and probe variants				Probe
Shuffled labels (control)				$51.6$

Table 3: The result is robust to the failure threshold and to every control we tried (HardMatch, ROC-AUC, %). Top: failure threshold. Bottom: controls and probe variants ( global_last unless noted).

Figure 3: Failure awareness emerges abruptly in the middle of the network, in both models. ROC-AUC for predicting VGGT’s pose failures on HardMatch from every block of (a) VGGT and (b)

\pi^{3}

, with a linear probe (blue) and with one training-free number, the cosine similarity between the two views’ pooled features (orange). Dashed: the model’s own confidence. Shaded: the per-image encoder. The probe is flat through the encoder and the first half of the cross-view layers, then rises within a few blocks; only in VGGT does the training-free similarity follow.

	VGGT	$\pi^{3}$
Failure rate ( $>50$ px)	$34.9\%$	$34.6\%$
mAA@10 px (test split)	$19.9$	$18.4$
Own confidence (min of pair)	$79.7$	$62.2$
All outputs (learned)	$77.7$	$76.9$
Probe, best of encoder blocks 6/12/23	$65.0$	$66.3$
Probe, last attention block	$\mathbf{93.5}$	$\mathbf{91.7}$

Table 4: A second model,

\pi^{3}

, shows the same gap between its features and its confidence. HardMatch, same pairs, failure definition and protocol. ROC-AUC (%) unless stated; “most confident half”: the half of the pairs on which the model’s own confidence is highest.

Figure 4: Within each level of confidence, VGGT’s confidence is near chance; the probe is not. HardMatch pairs split into quartiles of VGGT’s point-map confidence. Bars: VGGT’s failure rate. Lines: ROC-AUC for predicting failure within each quartile.

Figure 5: The probe beats VGGT’s confidence in every HardMatch category but one. Bars: VGGT’s failure rate. Markers: AUC of the probe (filled) and of point-map confidence (open).

Figure 6: Abstaining on the pairs the probe distrusts removes most of VGGT’s failures. Keeping the pairs ranked most reliable by each score: failure rate (left) and mAA@10 px (right) of the kept pairs. Oracle: ranked by true error.

Co-visibility	$<0.02$	$0.02$ – $0.1$	$0.1$ – $0.3$	$>0.3$
Pairs	302	320	320	320
$e_{R}>15^{\circ}$	$6.6\%$	$0.3\%$	$0.0\%$	$0.3\%$
	all pairs		low covis.
	$e_{R}>10^{\circ}$	$e_{R}>15^{\circ}$	$e_{R}>10^{\circ}$	$e_{R}>15^{\circ}$
#failures	25	22	21	20
HardMatch probe	$\mathbf{94.6}$	$\mathbf{97.5}$	$\mathbf{86.9}$	$\mathbf{89.5}$

Table 5: The HardMatch probe transfers to rotation failures on PhotoTourism, beyond co-visibility. Top: VGGT’s rotation failures by co-visibility. Bottom: ROC-AUC (%) for rotation failures of the HardMatch probe (no retraining), VGGT’s confidence, ground-truth co-visibility (not available at test time) and a probe trained on PhotoTourism itself (scene-grouped CV). “low covis.”: pairs with co-visibility

<0.02

only.

Views per forward pass	$2$	$4$	$8$	$16$
Same pairs: $e_{R}>5^{\circ}$	$3.6\%$	$1.4\%$	$0.5\%$	$0.3\%$
Same pairs: $e_{R}>15^{\circ}$	$0.9\%$	$0.3\%$	$0.0\%$	$0.0\%$
All pairs	2,304	2,304	5,376	11,520
failures ( $e_{R}>5^{\circ}$ )	83	33	28	35
HardMatch probe (2-view)	$81.5$	$82.6$	$74.2$	$86.3$
VGGT confidence	$91.2$	$90.4$	$92.9$	$82.1$

Table 6: More views remove most failures; the few left are hard to rank for both the probe and confidence (PhotoTourism,

96

sets of

16

images). Top: failure rate of the same

2{,}304

pairs when VGGT sees

2

4

8

16

views per forward pass. Bottom: all pairs of each setting; ROC-AUC (%) for rotation errors above

5^{\circ}

of the HardMatch probe (trained on two-view pairs, no retraining) and of VGGT’s confidence. With

28

–

83

failures per setting, the

95\%

confidence intervals span more than

40

points.

True baseline	$<10$ cm	$10$ – $20$ cm	$20$ – $40$ cm	$>40$ cm
Pairs	83	173	292	952
$e_{t}>15^{\circ}$	$47.0\%$	$27.2\%$	$6.8\%$	$0.9\%$
$e_{R}>5^{\circ}$	$0.0\%$	$0.0\%$	$0.3\%$	$3.3\%$
Failure	#fail	HM probe	Confidence	In-domain
$e_{R}>5^{\circ}$	32	$79.5$	$58.7$	$\mathbf{82.3}$
$e_{\text{pose}}>10^{\circ}$	202	$47.6$	$61.0$	$\mathbf{90.5}$

Table 7: On ScanNet, the probe transfers to rotation failures but not to short-baseline translation failures. Top: VGGT’s errors by true camera baseline; translation-direction errors dominate at short baselines. Bottom: ROC-AUC (%) of the HardMatch probe (no retraining), VGGT’s confidence and a probe trained on ScanNet itself (scene-grouped CV), by failure type.

e_{\text{pose}}=\max(e_{R},e_{t})

UQ for 3D Reconstruction

See all 22 papers

GeoCond: A Conditioning-Aware Reliability Adapter for Feed-Forward 3D Reconstruction
Sep 16, 2026David Ahmedt-Aristizabal, Mohammad Ali Armin, Russell Tsuchida +1UQ for 3D Reconstruction 3D Reconstruction
CSIRO, Australia · Monash University, Australia
Uncertainty Quality of VGGT: An Analysis on the DTU Benchmark Dataset
Jun 15, 2026Markus Hillemann, Robert Langendörfer, Steven Landgraf +13D ViTs 3D Reconstruction
Institute of Photogrammetry and Remote Sensing, Karlsruhe Institute of Technology, Germany
Revar3r: gauge-aware perturbation uncertainty for feed-forward 3d reconstruction
Oct 6, 2026Sammam Mahdi, Fariha Binta Salim, Rakin Bin Rabbani +1Uncertainty Quantification 3D Reconstruction
BRAC University · Mahidol University

Sep 16, 2026J/K move · Enter open · S save

GeoCond: A Conditioning-Aware Reliability Adapter for Feed-Forward 3D Reconstruction

David Ahmedt-Aristizabal, Mohammad Ali Armin, Russell Tsuchida, Lars Petersson

CSIRO, Australia · Monash University, Australia

UQ for 3D Reconstruction 3D Reconstruction Feed-Forward 3D Reconstruction 3D Foundation Models Camera Pose Estimation

Open paper AskPDF arXiv

Feed-forward 3D foundation models such as VGGT predict cameras, depth, and point maps in a single pass, but can fail silently under low overlap, low parallax, and extreme relative rotation. Stratified analyses over these factors show that these failures are governed by geometric conditioning and are poorly captured by native aleatoric confidence. We introduce GeoCond, a lightweight reliability adapter for frozen feed-forward 3D backbones. GeoCond reads geometric conditioning cues and the backbone's own stability signals and outputs pose-level uncertainty and a refinement gate. During training, it can be supervised by frame-permutation orbit variance, ground-truth pose error when labels are available, or cycle residuals from unlabelled independent pose graphs. At inference, the default head requires only one backbone pass and a small MLP. On VGGT, GeoCond improves out-of-distribution (OOD) AUSE (area under the sparsification-error curve; lower is better) from

0.32

0.20

over native confidence on held-out outdoor scenes (and to

0.21

when trained on indoor data only) and avoids the collapse caused by applying bundle adjustment uniformly, without improving on feed-forward poses there. For three of six backbones, cycle-distilled variants provide a ground-truth-free adaptation route, including cases where permutation variance vanishes on equivariant models. The same reliability signal supports gated refinement, pose-graph weighting, calibration, curation, and capture decisions. Reliable feed-forward 3D reconstruction requires not only predicting geometry, but also knowing when that geometry should be trusted.

Abstract

UQ for 3D Reconstruction

Explore similar work

GeoCond: A Conditioning-Aware Reliability Adapter for Feed-Forward 3D Reconstruction

Uncertainty Quality of VGGT: An Analysis on the DTU Benchmark Dataset

Revar3r: gauge-aware perturbation uncertainty for feed-forward 3d reconstruction