Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement

Yakov Pyotr Shkolnikov

cs.CVMar 6, 2026

Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement

Authors: Yakov Pyotr Shkolnikov

Abstract

Vision-language models encode continuous geometry that their text pathway fails to express: a 6,000-parameter linear probe extracts hand joint angles at 6.1 degrees MAE from frozen features, while the best text output achieves only 20.0 degrees -- a 3.3x bottleneck. LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees, providing evidence for a pathway-training deficit rather than a representational one. Training objective determines accuracy more than architecture: five encoders spanning self-supervised, contrastive, and hybrid paradigms converge to statistically equivalent accuracy (R^2 approximately 0.55, TOST-equivalent at delta=0.03) despite sharing as little as CKA=0.41 representational similarity -- functional convergence without representational convergence. Autoregressive generation damages geometric fidelity, but the damage originates in the generation process, not in language alignment: Qwen2.5-VL's LLM layers actually improve probe accuracy over its raw vision encoder. Layer-wise analysis reveals a universal mid-network accuracy peak across all architectures, with attention heads in layers 18-22 carrying disproportionate geometric signal. These findings enable a single frozen backbone to function as a multi-task geometric sensor through lightweight probes, without fine-tuning or text generation.

Figures & tables

Panel of Figure 1 — Figure 1: Overview. Frozen foundation model features encode continuous geometry (joint angles) with 6.1 ∘ MAE via a linear probe, while the text pathway achieves only 20.0 ∘ , a 3.3 $\times$ bottleneck. Adding LoRA fine-tuning (r = 16) partially recovers probe-level accuracy (6.5 ∘ ) through the text pathway.

Regime	Method	MAE ( ∘ )	R 2	Parse
Task-specific	MediaPipe Hands ∗ [ 43 ]	16.3	$-$ 2.44	N/A
Text generation	Direct prompt (Qwen-3B)	39.3	—	varies
	Chain-of-thought (Qwen-3B)	139.3	—	varies
	Few-shot 3-ex. (Qwen-3B)	20.0	—	varies
LoRA text	LoRA Qwen-3B	7.45	0.299	100%
LoRA text	LoRA Gemma 3 4B	6.51	0.400	100%

Table 1: Four readout regimes for geometric information on FreiHAND. MAE in degrees; R 2 is uniform mean across 5 finger targets. LoRA: r = 16, 2,000 images, 2 epochs. Frozen probes: RRR, 6,400 images. ∗ MediaPipe uses monocular 3D world landmarks evaluated zero-shot (see Discussion for caveats).

Panel of Figure 2 — Figure 2: Bootstrap 95% CIs for 13 models on FreiHAND (ConvNeXt-L omitted; see Table 2 ). Five models form a TOST equivalence cluster (shaded) at R 2 $\,{\approx}\,$ 0.55. DINOv2 falls outside despite being the same architecture family as DINOv3.

Model	Training	Layer	R 2	CV R 2 †
SigLIP 2 ViT-L	Hybrid VL	L16	0.559	0.563
DINOv3 ViT-L	Self-supervised	L20	0.556	0.550
CLIP ViT-L	Contrastive VL	L20	0.551	0.554
SigLIP ViT-L	Contrastive VL	L16	0.550	0.549
InternViT-300M	Hybrid VL	L20	0.547	0.549
DINOv2 ViT-L	Self-supervised	L20	0.523	0.494

Table 2: Cross-architecture comparison on FreiHAND (8,000 images). R 2 is uniform mean across 5 finger targets. † Nested 10-fold CV (Friedman

\chi^{2}

= 94.3,

p

{<}

10 -15 ; Nemenyi CD = 4.45). Cluster models within 0.012 of test R 2 . — = not included in CV (added for ablation/scaling analysis).

Panel of Figure 3 — Figure 3: CKA similarity vs. probing accuracy difference for all 28 pairwise comparisons among eight models (six ViT-L + two ViT-B) on FreiHAND. Spearman $\rho$ = 0.03 ( $p$ = 0.88): no detectable correlation between representational similarity and geometric probing accuracy ( $n$ = 28). The most similar pair (DINOv2–DINOv3, CKA = 0.88) differs by 0.033 R 2 ; the least similar pair (SigLIP 2–CLIP, CKA = 0.41) differs by only 0.008.

Figure 4: Layer-wise R 2 on FreiHAND for ten models (solid: vision encoders; dashed: LLM decoders). X-axis is normalized layer depth (0 = first, 1 = last). Vision encoders rise monotonically; LLM decoders peak at early layers and decline, consistent with autoregressive processing discarding fine-grained geometry.

Approach	Params	Train data	Tasks	MAE ( ∘ )
MediaPipe Hands [ 43 ] 1 1 1 3.8M = palm detector (1.76M) + hand landmark model (2.01M).	3.8M	proprietary	1	16.3 ∗
HRNet-W48 (hand) [ 53 ]	63.6M	150K+	1	—
6DRepNet (head) [ 52 ]	${\sim}$ 41M	300K	1	2.66 †
Frozen probe (ours)	6K / task	6,400	any	6.14
+ shared backbone	304M (shared)	—	—	—
LoRA readout (ours)	${\sim}$ 1M / task	2,000	any	6.51

Table 3: Cost comparison: frozen probe vs. task-specific models. Frozen probes reuse an existing backbone; per-task overhead is minimal. ∗ MediaPipe MAE evaluated zero-shot on our FreiHAND test set via 3D world landmarks (monocular depth; see Discussion for caveats). † Published MAE on BIWI 70/30 split [ 52 ] ; our evaluation on the same 4-subject test split yields 6.10 ∘ (different face detector and split).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Thumb	Index	Middle	Ring	Pinky	Mean	MAE ( ∘ )
SigLIP 2 ViT-L	0.195	0.580	0.693	0.701	0.625	0.559	6.14
DINOv3 ViT-L	0.149	0.590	0.709	0.708	0.624	0.556	6.11
CLIP ViT-L	0.177	0.575	0.686	0.699	0.620	0.551	6.19
SigLIP ViT-L	0.175	0.569	0.688	0.706	0.614	0.550	6.16
InternViT-300M	0.174	0.571	0.681	0.699	0.607	0.547	6.12
DINOv2 ViT-L	0.143	0.551	0.667	0.662	0.594	0.523	6.39

Appendix

Table 4: Per-finger R 2 on FreiHAND (8,000 images). Best overall in bold .

Model	Type	Thumb	Index	Middle	Ring	Pinky	Mean R 2	MAE ( ∘ )
DeiT3-L (IN-1K)	ViT	0.047	0.421	0.512	0.495	0.420	0.379	7.65
ConvNeXt-L (IN-1K)	CNN	0.102	0.451	0.516	0.502	0.455	0.405	7.49
ConvNeXt-L (IN-22K)	CNN	0.100	0.495	0.581	0.581	0.515	0.455	7.05

Appendix

Table 5: Controlled architecture ablation: per-finger R 2 on FreiHAND.

	Ridge				AttentionPool
Model	Yaw	Pitch	Roll	Mean	Yaw	Pitch	Roll	Mean
DINOv3	0.705	0.948	0.168	0.607	—	—	—	0.838
DINOv2	0.668	0.874	0.052	0.532	0.958	0.940	0.779	0.892
SigLIP 2	0.385	0.902	0.078	0.455	—	—	—	0.787

Appendix

Table 6: BIWI head pose R 2 by component (Ridge / AttentionPool).

Bone Type	Ridge R 2	AttnPool R 2	Count
MCP (proximal)	0.544	0.602	5
PIP (middle)	0.559	0.616	5
DIP (distal)	0.271	0.312	5
Mean (15 joints)	0.458	0.510	15

Appendix

Table 7: Per-bone R 2 for DINOv3 on FreiHAND (Ridge / AttentionPool).

Model	R 2 ( $f_{x}$ )	Type
QwenVIT	0.939	Vision encoder
DINOv3	0.924	Self-supervised
DINOv2	0.913	Self-supervised
SigLIP 2	0.902	Hybrid VL
Gemma 3 L0	0.887	Generative VLM
InternViT	0.883	Hybrid VL

Appendix

Table 8: Camera intrinsics probing (R 2 for

f_{x}

on FreiHAND).

Layer	DINOv2	DINOv2+reg	$\Delta$
L4	0.114	0.207	+0.093
L8	0.160	0.269	+0.110
L12	0.300	0.368	+0.067
L16	0.432	0.441	+0.010
L20	0.523	0.541	+0.019
L23	0.510	0.522	+0.012

Appendix

Table 9: DINOv2 vs. DINOv2+registers: layer-wise R 2 on FreiHAND.

Model	Test R 2	CV R 2	$\Delta$
SigLIP 2	0.559	0.563	$+$ 0.004
DINOv3	0.556	0.550	$-$ 0.006
CLIP	0.551	0.554	$+$ 0.003
SigLIP	0.550	0.549	$-$ 0.001
InternViT	0.547	0.549	$+$ 0.002
DINOv2	0.523	0.494	$-$ 0.029

Appendix

Table 10: Nested 10-fold CV vs. test-set R 2 on FreiHAND.

Ablation	BIWI $\Delta$ R 2	BIWI (post)	YCB $\Delta$ R 2	YCB (post)
Top-norm patches	$-$ 0.126	0.481	$-$ 0.003	0.706
Random patches	$-$ 0.107	0.500	$-$ 0.040	0.669

Appendix

Table 11: Patch ablation: R 2 change from removing 100 patches (DINOv3).

	DINOv2	DINOv3	SigLIP	SigLIP2	CLIP	InternViT	DINOv2-B	SigLIP-B
DINOv2	1.000	0.881	0.703	0.523	0.565	0.708	0.861	0.659
DINOv3		1.000	0.691	0.522	0.558	0.681	0.800	0.643
SigLIP			1.000	0.526	0.524	0.744	0.659	0.771
SigLIP2				1.000	0.412	0.631	0.427	0.493
CLIP					1.000	0.554	0.470	0.510
InternViT						1.000	0.594	0.736

Appendix

Table 12: Linear CKA similarity on FreiHAND (8 models at best layers). Six ViT-L models plus DINOv2-B (L12) and SigLIP-B (L12).

Layer	DINOv2	DINOv3	SigLIP	SigLIP 2	CLIP	InternViT
L0	—	0.114	0.149	0.135	0.128	0.103
L4	0.114	0.204	0.276	0.265	0.238	0.246
L8	0.160	0.262	0.381	0.365	0.315	0.331
L12	0.302	0.340	0.519	0.507	0.435	0.432
L16	0.434	0.425	0.550	0.559	0.541	0.536
L20	0.523	0.556	0.549	0.557	0.551	0.547

Appendix

Table 13: Layer-wise R 2 on FreiHAND for 6 ViT-L models.

Model	Yaw	Pitch	Roll	$t_{x}$	$t_{y}$	$t_{z}$	Rot	Trans
DINOv2	0.828	0.639	0.716	0.609	0.678	0.887	0.728	0.725
DINOv3	0.806	0.637	0.683	0.607	0.653	0.908	0.709	0.723
SigLIP 2	0.781	0.617	0.711	0.580	0.680	0.871	0.703	0.710
SigLIP	0.778	0.626	0.683	0.585	0.680	0.879	0.696	0.715
Qwen-7B	0.810	0.616	0.683	0.572	0.673	0.887	0.703	0.711

Appendix

Table 14: YCB-Video per-component R 2 (Ridge probe).

Head	R 2	max $\|\rho\|$	Best joint
6	0.479	0.167	middle PIP
14	0.478	0.035	ring MCP
4	0.475	0.273	index PIP
1	0.473	0.093	index MCP
11	0.473	0.280	middle PIP
12	0.469	0.042	ring MCP

Appendix

Table 15: Per-head probing R 2 for DINOv2-L layer 20 (top 10 of 16 heads).

Model	R 2 yaw	R 2 pitch	R 2 mean	MAE ( ∘ )
DINOv3	0.855	0.719	0.787	3.14
DINOv2	0.803	0.360	0.582	4.53
CLIP	0.854	0.259	0.557	4.70
SigLIP 2	0.844	0.248	0.546	4.74
ConvNeXt-L	0.709	0.249	0.479	5.11

Appendix

Table 16: Gaze probing on MPIIFaceGaze (45,000 images, RRR probe).

Layer	LoRA R 2	Frozen R 2	$\Delta$
L0	0.530	0.505	$+$ 0.025
L2	0.532	0.504	$+$ 0.028
L4	0.524	0.492	$+$ 0.032
L8	0.529	0.483	$+$ 0.046
L12	0.491	0.438	$+$ 0.052
L16	0.403	0.361	$+$ 0.041

Appendix

Table 17: Layer-wise R 2 for Gemma 3 4B (LoRA vs. frozen base).

Representation Geometry in VLMs

See all 21 papers

GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models
Oct 13, 2026Rong Li, Shijie Li, Jiangyong Huang +7Geospatial QA Visual Spatial Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Institute for General Artificial Intelligence · Agency for Science, A*STAR +3
GeoWorld-VLM: Geometry from World Models for Vision-Language Models
May 15, 2026Renjie Gu, Kaichen Zhou, Yan Luo +1Visual Spatial Reasoning VLM Adaptation
Harvard AI and Robotics Lab · Kempner Institute for the Study of Natural and Artificial Intelligence · Harvard University
GeoPID: Decomposing and Steering Visual Information in Vision-Language Models
Oct 6, 2026Seulgi Kim, Zhixiong Zhang, Xinwei Zhang +2Vision-Language Grounding Representation Geometry in VLMs
Georgia Institute of Technology · Work done as an intern at Amazon. · Amazon

Oct 13, 2026J/K move · Enter open · S save

GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models

Rong Li, Shijie Li, Jiangyong Huang, Lingdong Kong+6

The Hong Kong University of Science and Technology (Guangzhou) · Beijing Institute for General Artificial Intelligence · Agency for Science, A*STAR+3

Geospatial QA Visual Spatial Reasoning VLM Reasoning Representation Geometry in VLMs Geometric Representation Learning

Open paper AskPDF arXiv

Vision-language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models recover this structure from RGB, but existing methods either fuse the estimated features with visual tokens and learn to use them from question answering alone, or align VLM latents with geometry features. Capturing geometry, however, does not ensure its use: zeroing the aligned latents of a released feature-alignment model leaves its accuracy unchanged. We introduce GeoReader, a framework that learns geometric representations for the VLM that reads them. First, we train lightweight Geometric Readers to produce continuous input tokens, supervising them with geometry descriptions decoded by a frozen VLM, so that the Readers express geometry in a form the target model can already read. We then freeze the Readers to preserve this interface while fine-tuning the VLM on spatial QA, with geometry injection conditioned on the question. Geometry descriptions serve only as training targets: at inference, the VLM answers directly from the continuous geometric tokens without generating a geometry chain of thought. With geometry predicted from RGB images, our approach reaches 94.3 on MindCube-Tiny with Qwen3-VL-4B and, with a multi-source QA mixture, an average of 72.5 across six additional spatial benchmarks with Qwen3.5-9B. These results suggest that a useful geometric representation is one the receiving model can use, not merely one from which geometry can be reconstructed.

Abstract

Representation Geometry in VLMs

Explore similar work

GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models

GeoWorld-VLM: Geometry from World Models for Vision-Language Models

GeoPID: Decomposing and Steering Visual Information in Vision-Language Models