Vision models are trained on 2D images of a 3D world. Do their hidden features encode the direction from which the camera views a scene, and if so, in what form? We study the relative viewing direction α of two objects in rendered scenes where both object centres stay at fixed pixels while the camera moves, so that perspective, shading and shadows change but the projected layout does not. In eight pretrained models, from diffusion models to self-supervised and image-text encoders, α is encoded in a strikingly simple form: a single two-dimensional plane in the hidden space, shared across scenes, in which the difference between the two objects' features traces (cosα,sinα). The plane keeps its orientation when lighting, colours, floors and object shapes change. This encoding is more than a summary of a few visible measurements: readouts fitted on one set of camera paths stay accurate at camera positions outside their range in seven of the eight models, whereas decoders of the measured object and shadow geometry, equally accurate within that range, fail there. In the diffusion transformer PixArt-α, the plane forms abruptly at a single block and carries direction information to later blocks, yet editing it changes the objects rather than the generated viewpoint. We also examine how this encoding carries over to scanned real objects.
Figures & tables
Figure 2: Fixed-projection camera trajectories and coverage. (a) Top view of the six circular camera paths around spheres A and B in one scene ( r : radius). At pose 4, the arrow is the ground projection of the camera’s optical axis, the dashed line the direction A → B and the dotted line the normal of the pair; β is the angle from A → B to the axis and α=90∘−β the angle between the axis and the normal. (b) Rendered views at the numbered poses; white crosses mark the projected object centres, which are the same pixels in every image. (c) Camera positions of all scenes in the object-pair frame; red and blue discs: objects A and B; grey: training cameras; blue: test views within coverage (200 of 448); orange: test views outside coverage (203). (d) Mean single-image error on test ellipse views of spheres within (blue) and outside (orange) coverage, for the readout of each model and for the geometric decoders.
Plane
Single-image error ( ∘ )
Model
Training
Block, t
R2 [95% CI]
Block, t
Familiar
Within
Outside
Input, within
PixArt- α
Diffusion, DiT
6, 200
0.71 [0.69, 0.73]
4, 50
4.6
4.5
6.5 [6, 7]
30.6
SD3-medium
Diffusion, MM-DiT
1, 400
0.87 [0.83, 0.90]
12, 200
6.9
6.5
22.3 [17, 29]
40.1
DiT-XL/2
Diffusion, DiT
6, 100
0.68 [0.61, 0.75]
14, 100
4.2
4.4
8.3 [6, 11]
20.6
SD 1.5
Diffusion, UNet
0, 800
0.52 [0.48, 0.56]
5, 200
2.1
2.1
7.7 [7, 9]
28.6
DINOv2-L
Self-distillation
22
0.69 [0.65, 0.72]
16
2.6
2.6
6.8 [6, 8]
20.0
Table 1: Eight frozen models (Tier A spheres, 8 test scenes). Plane: shared-plane model on difference features at the validation-selected block and timestep, held-out R2 . Single image: linear readout at the validation-selected block and timestep; mean angular error ( ∘ ) on the training paths (familiar) and on test ellipse views within and outside the coverage of training cameras (95% scene-bootstrap interval), and for the same readout at the model input within the coverage.
Figure 3: Where the plane forms. Held-out R2 of the shared-plane probe of PixArt- α (difference features, test scenes) at the input (in) and at every block, for each diffusion timestep t . Outlined: the block and timestep selected on validation scenes.
Figure 4: Appearance datasets. The same test scene, camera and direction in the Tier A shape datasets, in Tiers B1–B4 and in Tier C (scanned objects, path-traced).
Figure 5: Shape and appearance (Tier A shapes, Tiers B1–B4 and Tier C; 8 test scenes; numbers in Sec. S9). (a) Held-out R2 of the dataset’s own shared plane in PixArt- α (difference features, block 6, t=50 ). (b–d) Single-image angular error of the validation-selected readout of PixArt- α and DINOv2 and of the geometric MLP, on training paths (grey rings) and on test ellipse views within (blue) and outside (orange) the coverage; dashed line: chance. Tier C uses 8-pixel pooling windows (other windows and models in Sec. S9). C, per scene: plane and decoders fitted within each test scene on its own circles ( R2 and familiar error on a held-out circle; coverage relative to the scene’s own cameras).
Dataset
Own
Sph. → set
Set → sph.
Overlap
Rot. ( ∘ )
A: spheres
0.63
0.63
0.63
1.00
0
A: cylinders
0.35
0.22
0.49
0.69
-9
A: cubes (pair)
0.34
0.21
0.48
0.64
-5
A: cubes (world)
0.32
0.18
0.45
0.64
-11
B1: light
0.62
0.62
0.64
0.90
-2
B2: colour
0.52
0.51
0.56
0.48
3
Table 2: Plane sharing (PixArt- α , block 6, t=50 ). Held-out R2 of each dataset’s own plane, of the sphere plane on the dataset, and of the dataset’s plane on spheres; subspace overlap with the sphere plane (mean squared cosine of the two principal angles; 1: identical) and rotation of the (cosα,sinα) frame relative to it.
Error at block ( ∘ )
Intervention
Dim.
Var.
7
12
21
27
∥Δϵ∥
None
–
–
6.7
6.2
6.1
7.1
0.00
Ablate probe plane
2
31
36.6
15.4
15.2
13.1
3.75
Ablate top-2 PCs
2
16
8.3
6.1
6.0
8.0
8.91
Ablate variance-matched PCs
6
34
8.8
5.7
5.9
8.2
14.28
Ablate random plane
2
0.2
7.0
6.3
6.0
7.0
0.68
Table 3: Interventions at block 6 , t=200 , on 672 held-out views, applied to the tokens of each object’s pooling square. Ablate : the tokens’ coordinates in a subspace are replaced by their training mean, removing all variation within it. Rotate : the tokens are shifted by Wo[u(α+180∘)−u(α)] , which moves their probe-plane coordinates to those of α+180∘ ; for the top two PCs and the random plane, Wo is replaced by those directions with the probe’s column norms, giving a shift of the same size. PCs: principal components of the features orthogonal to the plane. Dim.: dimension of the edited subspace per object; Var.: share (%) of the object features’ training variance that lies in it. Error at block: mean error ( ∘ ) of α decoded at a later block (27 is the last) by decoders fitted on unedited activations. ∥Δϵ∥ : mean change in the model’s noise prediction ϵ ; a larger value means the edit disturbs the model’s output more, whatever its effect on direction.
Figure S1 : Main dataset. Four scenes (two training, one validation, one test) along the source circle at six viewing directions.
Figure S2 : Tier B4. The same scenes and views with shapes, colours, floor and light drawn per scene.
Figure S3 : Tier C. The same scenes and views with scanned household objects, HDR environment lighting and textured floors.
Figure S4 : Tier A shapes and Tiers B1–B3. Two test scenes at three viewing directions; each row changes one property of the main dataset (Tier A spheres).
Block
Main
Empty
Descriptive
input
0.297
0.297
0.297
3
0.374
0.375
0.373
4
0.472
0.472
0.475
5
0.416
0.420
0.417
6
0.713
0.710
0.718
7
0.524
0.514
0.526
Table S1 : Prompt robustness. Held-out R2 at t=200 for the main prompt (“a scene containing two geometric objects”, three seeds), an empty prompt, and “a red sphere and a blue sphere on a checkered floor” (one seed each).
Figure S5 : Where the plane forms ( t=200 , test scenes). Grey bar: block 6.
Features
Rep.
R2
95% CI
Plane
Error
Input
Concat.
0.178
[0.130, 0.223]
52.8
17.8
Input
Diff.
0.297
[0.246, 0.346]
68.9
24.0
Block 6
Concat.
0.472
[0.455, 0.490]
55.7
5.9
Block 6
Diff.
0.713
[0.693, 0.733]
76.4
6.8
Table S2 : Shared-plane probe of PixArt- α on held-out scenes (block 6 and input, t=200 ; 8 test scenes). Plane: fraction of centred feature energy in the learned plane (%). Error: mean angular error of the trajectory-centred decoder ( ∘ ).
Table S4 : Single-image decoding on held-out scenes along the training trajectories (PixArt- α ; mean angular error in degrees, 95% scene-bootstrap interval).
Plane R2
Single-image error on held-out trajectories ( ∘ )
Protocol
Seen
Held-out [95% CI]
Readout
Input
Spheres
+Shadows
+Shadows (MLP)
Small → large radii
0.74
0.63 [0.60, 0.66]
8.4
31.8
15.3
15.0
5.6
Large → small radii
0.69
0.71 [0.67, 0.75]
8.9
35.5
20.4
18.5
13.5
Circles → ellipses
0.71
0.56 [0.49, 0.61]
5.6
62.2
52.7
48.9
30.8
Table S5 : Transfer to held-out camera trajectories (test scenes; plane at block 6, t=200 ; single-image readout at block 4, t=50 ). The plane and all decoders are fitted on training scenes and training trajectories only. R2 : trajectory-centred shared-plane model (difference features, block 6) on training (seen) and held-out trajectories. Error: single-image angular decoding error on held-out trajectories ( ∘ ; best in bold) for linear decoders on DiT features at the validation-selected readout (block 4, t=50 ) and at the input, and for decoders on measured sphere and shadow geometry.
Readout
Input
Data
Model
Block, t
Within
Near
Outside
Within
Near
Outside
A: spheres
PixArt- α
4, 50
4.5
7.0
6.5
30.6
67.6
92.1
SD3-medium
12, 200
6.5
16.1
22.3
40.1
61.3
81.9
DiT-XL/2
14, 100
4.4
6.8
8.3
20.6
48.1
73.0
SD 1.5
5, 200
2.1
3.9
7.7
28.6
63.4
78.7
DINOv2-L
16
2.6
5.5
6.8
20.0
52.0
96.2
Table S6 : Single-image error by coverage (test ellipse views, 8 test scenes, readouts trained on all noise seeds; degrees). Within: distance to the nearest training camera <0.25 object separations (200 views); near: 0.25 – 0.5 (45); outside: ≥0.5 (203). Readout: validation-selected block and timestep; input: the same procedure at the model input. The geometric decoders use measured object and shadow geometry. Tier C rows use the 8-pixel window; Tab. S12 compares windows.
Figure S6 : Mediation. Error of α decoded at each later block after mean ablation at block 6 ( t=200 , 672 test views). Only ablation of the probe plane affects downstream direction information.
Retrieved shift
Median Δ log ratio
Plane
Δ
Mean
Max
Toward
Output
True
Probe plane
+90∘
1.9
7.2
38
0.07
− 0.95
Probe plane
+180∘
1.0
4.5
50
0.04
− 1.14
Random plane
+90∘
0.0
0.0
12
0.00
− 0.95
Random plane
+180∘
0.0
0.0
12
0.00
− 1.14
Table S7 : Clamping during generation (8 faithful sources). Retrieved shift: angular difference between the renders retrieved for edited and unedited outputs (degrees; mean and maximum). Toward: outputs whose change in log(areaB/areaA) has the sign of the true change (%). Median change of this log ratio for edited outputs and for the true renders.
Figure S7 : Clamping during generation. Faithfully inverted sources. Clamping the probe plane alters material, shadows and ground texture but does not produce the viewpoint of the true renders (right); a random plane driven by the same procedure changes the image little.
Figure S8 : Gradient guidance toward +90∘ (one source, inverted to t=600 ). (a) Source, output without guidance, outputs with plane guidance at strengths s=0.25 and 1 and with a random readout, and the true view at the target. (b) Magnitude of the guidance gradient over the latent at three denoising steps (green: pooling windows), and the share of gradient mass inside the windows. (c) Readout minus source angle over the denoising steps; jumps occur where the readout switches to the next cached timestep, and the unguided readout also drifts at high noise.
Readout
Geometry MLP
Input
Data
Model
Block, t
Val.
Familiar
Ellipses
Familiar
Ellipses
Ellipses
B5
PixArt- α
23, 200
16.0
16.6
47.7
9.4
42.7
80.8
B4
PixArt- α
14, 200
13.6
21.3
35.9
8.3
39.9
85.7
PixArt- α
10, 50
11.7
21.2
37.5
8.3
39.9
76.6
SD3-medium
13, 200
11.7
20.0
51.9
8.3
39.9
65.7
SD3-medium
16, 50
13.9
27.8
51.3
8.3
39.9
58.2
Table S8 : All appearance runs (8 test scenes). Readout layer and timestep chosen on validation scenes (validation error shown); single-image angular error (degrees) on familiar and elliptical trajectories for the readout, the geometric MLP, and the readout applied at the model input.
Table S10 : Plane sharing across blocks (PixArt- α , t=50 ). For each block, the held-out R2 of the variant’s own plane, its subspace overlap with the sphere plane, and the rotation of its (cosα,sinα) frame relative to spheres (degrees).
PixArt- α
DINOv2-L
Geometry MLP
Dataset
R2
Fam.
In
Out
Fam.
In
Out
Fam.
In
Out
A: spheres
0.63
4.6
4.5
6.5
2.6
2.6
6.8
3.6
2.4
63.4
A: cylinders
0.35
2.6
3.0
67.8
1.5
1.2
54.6
3.0
2.6
86.5
A: cubes, pair-aligned
0.34
3.3
3.5
34.3
2.0
2.3
69.5
3.7
3.6
67.6
A: cubes, world-aligned
0.32
3.7
4.1
9.8
2.6
2.5
31.7
4.6
4.3
79.1
B1: light
0.62
5.6
6.1
8.1
2.8
2.4
6.9
4.5
3.7
67.5
Table S11 : Shape and appearance (Tier A shapes, Tiers B1–B4 and Tier C; 8 test scenes; plotted in Fig. 5 of the main text). R2 : held-out R2 of the dataset’s own shared plane in PixArt- α (difference features, block 6, t=50 ). Single-image angular error ( ∘ ) of the validation-selected readout on training paths (Fam.) and on test ellipse views within (In) and outside (Out) the coverage. Tier C uses 8-pixel pooling windows. C, per scene: plane and decoders fitted within each test scene on its own circles ( R2 and Fam. on a held-out circle; coverage relative to the scene’s own cameras).
Within
Outside
Model
Plane
8
24
8
16
24
SD 1.5 (UNet)
0.14
6.6
6.4
14.2
14.2
14.5 [11, 19]
MAE-L
0.06
11.9
12.8
21.8
20.9
18.4 [14, 23]
DINOv2-L
0.06
10.4
11.3
47.5
40.4
28.2 [22, 35]
CLIP-L
0.05
24.5
20.1
56.1
48.5
35.1 [24, 46]
SigLIP
0.06
21.4
19.8
46.5
36.8
37.7 [25, 53]
Table S12 : Tier C: scanned objects at matched pooling windows (8 test scenes; one noise seed, t=50 for diffusion models). Plane: best held-out R2 over windows. Single-image error ( ∘ ) within the coverage (8- and 24-pixel windows) and outside it (8, 16, 24 pixels; 95% interval at 24).
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.