Beyond the Training Cameras: Coverage-Controlled Probes of Viewing Direction in Frozen Vision Models
Organizations: Columbia University
Abstract
Vision models are trained on 2D images of a 3D world. Do their hidden features encode the direction from which the camera views a scene, and if so, in what form? We study the relative viewing direction of two objects in rendered scenes where both object centres stay at fixed pixels while the camera moves, so that perspective, shading and shadows change but the projected layout does not. In eight pretrained models, from diffusion models to self-supervised and image-text encoders, is encoded in a strikingly simple form: a single two-dimensional plane in the hidden space, shared across scenes, in which the difference between the two objects' features traces . The plane keeps its orientation when lighting, colours, floors and object shapes change. This encoding is more than a summary of a few visible measurements: readouts fitted on one set of camera paths stay accurate at camera positions outside their range in seven of the eight models, whereas decoders of the measured object and shadow geometry, equally accurate within that range, fail there. In the diffusion transformer PixArt-, the plane forms abruptly at a single block and carries direction information to later blocks, yet editing it changes the objects rather than the generated viewpoint. We also examine how this encoding carries over to scanned real objects.
Figures & tables
| Plane | Single-image error ( ∘ ) | |||||||
| Model | Training | Block, | [95% CI] | Block, | Familiar | Within | Outside | Input, within |
| PixArt- | Diffusion, DiT | 6, 200 | 0.71 [0.69, 0.73] | 4, 50 | 4.6 | 4.5 | 6.5 [6, 7] | 30.6 |
| SD3-medium | Diffusion, MM-DiT | 1, 400 | 0.87 [0.83, 0.90] | 12, 200 | 6.9 | 6.5 | 22.3 [17, 29] | 40.1 |
| DiT-XL/2 | Diffusion, DiT | 6, 100 | 0.68 [0.61, 0.75] | 14, 100 | 4.2 | 4.4 | 8.3 [6, 11] | 20.6 |
| SD 1.5 | Diffusion, UNet | 0, 800 | 0.52 [0.48, 0.56] | 5, 200 | 2.1 | 2.1 | 7.7 [7, 9] | 28.6 |
| DINOv2-L | Self-distillation | 22 | 0.69 [0.65, 0.72] | 16 | 2.6 | 2.6 | 6.8 [6, 8] | 20.0 |
| Dataset | Own | Sph. set | Set sph. | Overlap | Rot. ( ∘ ) |
|---|---|---|---|---|---|
| A: spheres | 0.63 | 0.63 | 0.63 | 1.00 | 0 |
| A: cylinders | 0.35 | 0.22 | 0.49 | 0.69 | -9 |
| A: cubes (pair) | 0.34 | 0.21 | 0.48 | 0.64 | -5 |
| A: cubes (world) | 0.32 | 0.18 | 0.45 | 0.64 | -11 |
| B1: light | 0.62 | 0.62 | 0.64 | 0.90 | -2 |
| B2: colour | 0.52 | 0.51 | 0.56 | 0.48 | 3 |
| Error at block ( ∘ ) | |||||||
| Intervention | Dim. | Var. | 7 | 12 | 21 | 27 | |
| None | – | – | 6.7 | 6.2 | 6.1 | 7.1 | 0.00 |
| Ablate probe plane | 2 | 31 | 36.6 | 15.4 | 15.2 | 13.1 | 3.75 |
| Ablate top-2 PCs | 2 | 16 | 8.3 | 6.1 | 6.0 | 8.0 | 8.91 |
| Ablate variance-matched PCs | 6 | 34 | 8.8 | 5.7 | 5.9 | 8.2 | 14.28 |
| Ablate random plane | 2 | 0.2 | 7.0 | 6.3 | 6.0 | 7.0 | 0.68 |
| Block | Main | Empty | Descriptive |
|---|---|---|---|
| input | 0.297 | 0.297 | 0.297 |
| 3 | 0.374 | 0.375 | 0.373 |
| 4 | 0.472 | 0.472 | 0.475 |
| 5 | 0.416 | 0.420 | 0.417 |
| 6 | 0.713 | 0.710 | 0.718 |
| 7 | 0.524 | 0.514 | 0.526 |
| Features | Rep. | 95% CI | Plane | Error | |
|---|---|---|---|---|---|
| Input | Concat. | 0.178 | [0.130, 0.223] | 52.8 | 17.8 |
| Input | Diff. | 0.297 | [0.246, 0.346] | 68.9 | 24.0 |
| Block 6 | Concat. | 0.472 | [0.455, 0.490] | 55.7 | 5.9 |
| Block 6 | Diff. | 0.713 | [0.693, 0.733] | 76.4 | 6.8 |
| Condition | Concat. | Difference |
|---|---|---|
| Matched noise (main) | 0.472 | 0.713 |
| Independent noise per view | 0.205 | 0.339 |
| Shuffled labels (mean of 100) | 0.000 | 0.000 |
| Shuffle -value | 0.0099 | 0.0099 |
| Decoder | Error | 95% CI |
|---|---|---|
| DiT block 4, (linear) | 4.6 | [3.9, 5.5] |
| DiT input (linear) | 28.9 | [25.4, 31.8] |
| Spheres (linear) | 11.1 | [9.0, 13.2] |
| Spheres + shadows (linear) | 10.4 | [8.7, 12.5] |
| Spheres + shadows (MLP) | 3.6 | [2.7, 4.7] |
| Plane | Single-image error on held-out trajectories ( ∘ ) | ||||||
| Protocol | Seen | Held-out [95% CI] | Readout | Input | Spheres | +Shadows | +Shadows (MLP) |
| Small large radii | 0.74 | 0.63 [0.60, 0.66] | 8.4 | 31.8 | 15.3 | 15.0 | 5.6 |
| Large small radii | 0.69 | 0.71 [0.67, 0.75] | 8.9 | 35.5 | 20.4 | 18.5 | 13.5 |
| Circles ellipses | 0.71 | 0.56 [0.49, 0.61] | 5.6 | 62.2 | 52.7 | 48.9 | 30.8 |
| Readout | Input | |||||||
| Data | Model | Block, | Within | Near | Outside | Within | Near | Outside |
| A: spheres | PixArt- | 4, 50 | 4.5 | 7.0 | 6.5 | 30.6 | 67.6 | 92.1 |
| SD3-medium | 12, 200 | 6.5 | 16.1 | 22.3 | 40.1 | 61.3 | 81.9 | |
| DiT-XL/2 | 14, 100 | 4.4 | 6.8 | 8.3 | 20.6 | 48.1 | 73.0 | |
| SD 1.5 | 5, 200 | 2.1 | 3.9 | 7.7 | 28.6 | 63.4 | 78.7 | |
| DINOv2-L | 16 | 2.6 | 5.5 | 6.8 | 20.0 | 52.0 | 96.2 | |
| Retrieved shift | Median log ratio | |||||
|---|---|---|---|---|---|---|
| Plane | Mean | Max | Toward | Output | True | |
| Probe plane | 1.9 | 7.2 | 38 | 0.07 | 0.95 | |
| Probe plane | 1.0 | 4.5 | 50 | 0.04 | 1.14 | |
| Random plane | 0.0 | 0.0 | 12 | 0.00 | 0.95 | |
| Random plane | 0.0 | 0.0 | 12 | 0.00 | 1.14 | |
| Readout | Geometry MLP | Input | ||||||
| Data | Model | Block, | Val. | Familiar | Ellipses | Familiar | Ellipses | Ellipses |
| B5 | PixArt- | 23, 200 | 16.0 | 16.6 | 47.7 | 9.4 | 42.7 | 80.8 |
| B4 | PixArt- | 14, 200 | 13.6 | 21.3 | 35.9 | 8.3 | 39.9 | 85.7 |
| PixArt- | 10, 50 | 11.7 | 21.2 | 37.5 | 8.3 | 39.9 | 76.6 | |
| SD3-medium | 13, 200 | 11.7 | 20.0 | 51.9 | 8.3 | 39.9 | 65.7 | |
| SD3-medium | 16, 50 | 13.9 | 27.8 | 51.3 | 8.3 | 39.9 | 58.2 | |
| PixArt- | Geom. linear | Geom. MLP | |||||
|---|---|---|---|---|---|---|---|
| Scenes | Block | Fam. | Ell. | Fam. | Ell. | Fam. | Ell. |
| 33 | 14 | 21.3 | 35.9 | 19.8 | 55.3 | 8.3 | 39.9 |
| 66 | 14 | 13.8 | 34.0 | 19.1 | 54.7 | 8.0 | 42.5 |
| 132 | 15 | 10.6 | 32.0 | 18.0 | 53.4 | 5.1 | 35.5 |
| Block 4 | Block 6 | Block 14 | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Variant | Overlap | Rot. | Overlap | Rot. | Overlap | Rot. | |||
| A: spheres | 0.38 | 1.00 | 0 | 0.63 | 1.00 | 0 | 0.47 | 1.00 | 0 |
| A: cylinders | 0.28 | 0.72 | -5 | 0.35 | 0.69 | -9 | 0.32 | 0.43 | 1 |
| A: cubes (pair) | 0.25 | 0.63 | -1 | 0.34 | 0.64 | -5 | 0.32 | 0.43 | -1 |
| A: cubes (world) | 0.23 | 0.62 | -3 | 0.32 | 0.64 | -11 | 0.28 | 0.45 | 2 |
| B1: light | 0.37 | 0.90 | -2 | 0.62 | 0.90 | -2 | 0.49 | 0.89 | 0 |
| PixArt- | DINOv2-L | Geometry MLP | ||||||||
| Dataset | Fam. | In | Out | Fam. | In | Out | Fam. | In | Out | |
| A: spheres | 0.63 | 4.6 | 4.5 | 6.5 | 2.6 | 2.6 | 6.8 | 3.6 | 2.4 | 63.4 |
| A: cylinders | 0.35 | 2.6 | 3.0 | 67.8 | 1.5 | 1.2 | 54.6 | 3.0 | 2.6 | 86.5 |
| A: cubes, pair-aligned | 0.34 | 3.3 | 3.5 | 34.3 | 2.0 | 2.3 | 69.5 | 3.7 | 3.6 | 67.6 |
| A: cubes, world-aligned | 0.32 | 3.7 | 4.1 | 9.8 | 2.6 | 2.5 | 31.7 | 4.6 | 4.3 | 79.1 |
| B1: light | 0.62 | 5.6 | 6.1 | 8.1 | 2.8 | 2.4 | 6.9 | 4.5 | 3.7 | 67.5 |
| Within | Outside | |||||
|---|---|---|---|---|---|---|
| Model | Plane | 8 | 24 | 8 | 16 | 24 |
| SD 1.5 (UNet) | 0.14 | 6.6 | 6.4 | 14.2 | 14.2 | 14.5 [11, 19] |
| MAE-L | 0.06 | 11.9 | 12.8 | 21.8 | 20.9 | 18.4 [14, 23] |
| DINOv2-L | 0.06 | 10.4 | 11.3 | 47.5 | 40.4 | 28.2 [22, 35] |
| CLIP-L | 0.05 | 24.5 | 20.1 | 56.1 | 48.5 | 35.1 [24, 46] |
| SigLIP | 0.06 | 21.4 | 19.8 | 46.5 | 36.8 | 37.7 [25, 53] |
Visual Representation Learning
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.
See all 245 papers