Existing video multimodal large language models (MLLMs) organize observations primarily in time, but embodied spatial reasoning requires organizing them in space. Consequently, their visual representations can remain locally consistent across nearby frames while lacking a persistent spatial context that is stable across viewpoints and time. We exploit depth and camera pose, which are naturally available to embodied agents, to explicitly establish this spatial correspondence. We introduce Cambrian-BEV, which extends Cambrian-S with a parameter-free geometric tokenization scheme that back-projects visual patch features from each frame into a shared bird's-eye-view (BEV) coordinate frame and averages features that fall within the same spatial cell. Thus, observations of the same physical location, regardless of viewpoint or time step, are aggregated into a common spatial token. In addition to providing a persistent scene representation, this formulation acts as an efficient compression mechanism: 128 frames containing approximately 100K DINOv3 patch features are consolidated into a fixed 27×27 grid of only 729 BEV tokens, cutting LLM prefill time by 2.9× and peak memory by 3.6×. Cambrian-BEV requires no architectural modification beyond BEV tokenization and uses a frozen DINOv3 vision encoder with a trainable multimodal projector and LLM. On VSI-Bench, our model reaches 63.48, +8.5 points over an identically fine-tuned frame-token control with 12.6× fewer visual tokens, with particularly large gains on room size (+24.5) and relative direction (+34.1). On ReVSI, it reaches 44.68, +4.5 over the corresponding control.
Figures & tables
Figure 1: From frame-indexed tokens to BEV visual tokens. Top: A conventional video MLLM represents each frame independently, producing repeated evidence for regions observed from multiple viewpoints. Bottom: We use depth and camera pose to back-project patch features into a shared metric coordinate frame and aggregate them on a fixed 27×27 bird’s-eye-view grid. Repeated observations of the same spatial region are fused into common BEV cells, replacing approximately 9K frame tokens with 729 spatially indexed tokens at the LLM interface.
Figure 2: Cambrian-BEV architecture. Patch tokens leave the frozen encoder and the vision-to-language projector already in the language embedding space, so the parameter-free geometric module (steps 1–6) can pool them directly into a metric BEV grid whose cells become the BEV tokens that replace the frame tokens in the LLM context. The BEV projection (green) is introduced only in the final fine-tuning stage (stage 5, Tab. S3 ); the encoder, projector, and LLM are inherited from the unmodified four-stage Cambrian-S recipe, which trains on frame tokens.
Method
Encoder
#Vis. tok.
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
App. Order
Avg
Cambrian-S-7B [ 33 ] (full fp32)
SigLIP2
9,216
72.81
49.90
74.72
69.62
68.45
74.38
42.27
81.39
66.69
Cambrian-S-0.5B [ 33 ] (full fp32)
SigLIP2
9,216
68.37
33.51
53.92
52.92
55.35
48.98
29.90
70.87
51.73
Frames
SigLIP2
9,216
61.31
32.10
43.28
27.85
42.39
47.70
29.90
50.97
41.94
Frames
DINOv3
9,216
63.01
29.53
47.72
26.04
43.38
46.66
30.41
49.03
41.97
Frames parent + BEV tokens, zero-shot
DINOv3
729
61.91
32.10
46.99
25.90
45.49
46.43
26.80
47.25
41.61
Frames + spatial FT (control)
DINOv3
9,216
69.79
37.49
69.97
60.49
49.86
48.58
26.80
76.86
54.98
Table 1: VSI-Bench results (%). The first four categories are scored by MRA, the last four by accuracy; Avg is their unweighted mean. All models see the same ≤ 128 frames per scene. Rows below the rule share our recipe and data (0.5B) and are trained under bf16 autocast, which accounts for their gap to the official full-fp32 releases ( Appendix G ); the grey row feeds BEV tokens to the frame-trained parent without fine-tuning. Bold marks the best among our rows.
VSI-Bench
ReVSI
Variant
Avg
Room
Rel. Dir.
Avg
(a) Grid frame: aggregation × orientation
hard × world
55.69
84.44
46.87
41.79
hard × PCA-rectified
52.09
82.92
47.52
41.16
hard × FP (ours)
63.48
85.00
82.67
44.68
bilinear × world
60.32
81.81
78.27
45.20
Table 2: Ablations . Single-variable changes to the recipe of Tab. 1 , on VSI-Bench (overall average, room size, relative direction) and ReVSI (overall average, all_frame , Sec. 4.4 ). “FP” is first-pose orientation. Bold marks the best in each column. We adopt hard × FP as the default because it ties bilinear × FP on VSI-Bench with a 3.4× cheaper projection ( Sec. 4.3 ); bilinear × FP is the stronger variant on ReVSI. Blocks (d) and (e) are discussed in Appendix C .
Method
all_frame
64_frame
Cambrian-S-0.5B (official, full fp32)
37.68
37.59
Frames (SigLIP2)
32.10
31.92
Frames (DINOv3)
33.41
33.10
Frames + spatial FT (control)
40.17
40.02
BEV, SigLIP2 (hard × FP)
40.56
40.34
BEV, hard × world
41.79
42.27
Table 3: ReVSI results . Avg is the unweighted mean of the seven categories; both subsets cover the same 380 scenes. all_frame : 6,808 questions, 128 frames per scene; 64_frame : 6,616 questions, 64 frames per scene. Per-category scores are in Appendix I . All rows are 0.5B; ours are trained under bf16 autocast ( Appendix G ). Bold marks the best among our rows.
Figure 3: Patch-level text similarity, SigLIP2 vs. DINOv3. SigLIP2 distributes evidence for the queried object (row label) across the image, whereas DINOv3 more closely follows the object’s extent. Warm colors indicate higher similarity; color scales are normalized per panel. Details in Appendix H .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Range
Cell (m)
ScanNet
ScanNet++
ARKitScenes
All
±3 m
0.22
93.79
86.98
91.75
91.29
±4 m
0.30
98.43
94.60
97.47
97.14
±5 m
0.37
99.55
97.40
99.19
98.94
±6 m
0.44
99.85
98.59
99.72
99.54
±7.5 m
0.56
99.97
99.30
99.94
99.83
±10 m
0.74
100.00
99.61
99.99
99.92
Appendix
Table S1: Grid range vs. coverage. Fraction (%) of valid back-projected points inside a centered ±R window over the 4,955 scenes of VSI-288K-Depth, and cell size at 27×27 . Bold marks the range used by the final model.
Frames
BEV
Ratio
Visual tokens
9,216
729
12.6×
Per scene (ms)
Vision encoder + projector
366.0
364.8
1.0×
Frame downsample / BEV build
0.4
1.9
–
Per question (ms)
LLM prefill
45.9
15.7
2.9×
Appendix
Table S2 : Inference cost of Cambrian-BEV and the frames-only control, median GPU ms on one H100 SXM (bf16, FlashAttention-3, 128-frame scenes, 15 decode steps). Scene encoding is paid once per scene and cached across its questions; the remaining rows are paid per question. Peak memory is measured over the whole pipeline.
Table S3: Training recipe. Stages 1–4 are the Cambrian-S recipe as run in our reproduction; stage 5 is the fine-tuning stage of Sec. 4.1 , run once with frame tokens (control) and once with BEV tokens (Cambrian-BEV). The vision encoder is frozen throughout. Parenthesized sizes are subsets of the named corpus.
Query
Encoder
min
mean
max
p2
p98
I
“tv”
SigLIP2
−0.105
−0.001
0.255
−0.084
0.216
0.33
“tv”
DINOv3
−0.058
0.041
0.214
−0.021
0.167
0.75
“bicycle”
SigLIP2
−0.167
0.001
0.219
−0.088
0.176
0.33
“bicycle”
DINOv3
−0.058
0.024
0.200
−0.033
0.125
0.74
Appendix
Table S4: Statistics of each panel of Fig. 3 : cosine similarity over all patch tokens, with p2 and p98 the endpoints of that panel’s colorbar, and Moran’s I (4-neighbour spatial autocorrelation). Grids are 27×27 (SigLIP2) and 28×28 (DINOv3).
Method
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route
Avg
Cambrian-S-0.5B (official, full fp32)
40.77
32.92
46.30
37.72
34.20
35.54
36.33
37.68
Frames (SigLIP2)
32.14
38.36
32.22
25.36
30.05
34.75
31.83
32.10
Frames (DINOv3)
32.44
36.90
34.98
31.39
30.35
36.63
31.19
33.41
Frames + spatial FT (control)
38.91
38.12
47.34
45.30
33.01
36.41
42.12
40.17
BEV, SigLIP2 (hard × FP)
37.34
36.00
46.83
54.37
31.31
36.27
41.80
40.56
BEV, hard × world
39.44
39.16
46.44
58.54
31.53
34.68
42.77
41.79
Appendix
Table S5 : ReVSI all_frame per category (6,808 questions, 380 scenes, 128 frames per scene). Avg is the unweighted mean of the seven categories. All rows are 0.5B. Bold marks the best among our rows.
Method
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route
Avg
Cambrian-S-0.5B (official, full fp32)
44.57
32.09
44.04
32.52
35.34
35.14
39.43
37.59
Frames (SigLIP2)
32.45
36.91
32.17
24.93
31.20
33.86
31.90
31.92
Frames (DINOv3)
32.09
36.02
35.60
31.36
29.63
36.56
30.47
33.10
Frames + spatial FT (control)
36.59
36.07
48.13
45.03
32.53
37.69
44.09
40.02
BEV, SigLIP2 (hard × FP)
37.71
37.79
46.36
48.97
30.65
36.49
44.44
40.34
BEV, hard × world
38.61
39.58
46.89
60.86
31.20
35.06
43.73
42.27
Appendix
Table S6 : ReVSI 64_frame per category (6,616 questions, 380 scenes, 64 frames per scene). Avg is the unweighted mean of the seven categories. All rows are 0.5B. Bold marks the best among our rows.
Figure S1 : BEV occupancy on the floor plan and questions the frame baseline gets wrong. Three VSI-Bench scenes (ARKitScenes living room and kitchen, ScanNet++ office). Top: for each object named in the direction question, the video frame in which it is most visible, with the dataset’s annotated 3D box projected into the image. Middle: top-down rendering of the RGB-D reconstruction with the model’s 27×27 grid ( 0.37 m cells, ±5 m) in the grid’s own frame; grey cells receive at least one patch token, the dark line is the camera path, and the arrow marks the first camera pose whose heading defines the grid’s forward axis. Coloured cells and outlines are the same annotated boxes seen from above (orange: standing by, blue: facing, pink: asked about). Bottom: the room-size and relative-direction questions from these scenes; the frames-only control (“frames”, 9,216 frame tokens) answers them incorrectly and Cambrian-BEV (“ours”, 729 BEV tokens) correctly. The occupied area and the egocentric frame of the grid are the two quantities behind the largest gains in Tab. 1 (room size +24.5 , relative direction +34.1 ).
Figure S2 : Failure cases. (a) Appearance order (ScanNet scene0164_02 ): the frames in which each object first becomes visible (annotated boxes projected into the video: counter, refrigerator, backpack, door) are all pooled into one grid, so this order is not represented; the control answers all nine appearance-order questions of this scene, ours one. (b) Thin structures: the token of a cell averages every surface inside it. Measured with the datasets’ object annotations, a door or a whiteboard contributes 71% and 53% of the points pooled into its cells, a 38 cm television 20%, a telephone 10% and a 31 cm coat hanger 1%; three of the four desk monitors in the office share a single cell. Size and count questions about such objects fail for both models. (c) Corrupted geometry: in ScanNet++ d755b3d9d8 the shipped poses drift kilometres from the first frame, so the centroid and the floor anchor are meaningless, no point falls inside the ±5 m grid, every BEV token is empty, and the model answers from its prior (room size 2.8 vs. 16.9 m 2 ). In ARKitScenes 45261150 spurious depth below the floor (5.6% of points) pulls the 2nd-percentile floor anchor 0.30 m below the true floor; this affects only the floor-relative height filter of Appendix B , which the final model does not use.
Spatial understanding is fundamental to embodied intelligence, underpinning applications such as robotic manipulation, embodied navigation, and autonomous driving. Although recent vision-language models (VLMs) have achieved impressive performance on spatial reasoning benchmarks, state-of-the-art approaches typically rely on additional spatial encoders or architectural modifications during inference, increasing computational cost. We introduce Space Tokens, a lightweight, architecture-agnostic framework that equips VLMs with explicit continuous spatial representations without requiring additional inference-time modules. By distilling scene-level 3D geometry and object-centric spatial attributes into continuous latent tokens, our method enables these modalities to be directly incorporated into a chain-of-thought reasoning process, thereby improving the VLM's spatial reasoning capabilities. At the same time, the learned representations can be explicitly decoded to verify that they encode meaningful geometric information, while the unified token interface remains extensible to additional modalities. Experiments on VSI-Bench improve Qwen3-VL-8B by 4.3% and SenseNova-SI-1.3 by 1.3%, while achieving state-of-the-art performance on object size (79.2%) and room size estimation (75.7%). These results demonstrate that continuous spatial tokens provide an effective, interpretable, and computationally efficient mechanism for integrating geometric reasoning into large vision-language models.
Hunter Schofield, Mohammed Elmahgiubi, Mohammad Mahdavian +4
York University · Huawei Technologies Canada · University of Toronto
Camera pose matters. The position and orientation of each viewpoint define a shared spatial coordinate frame that relates observations across video frames. Yet this signal is largely absent from multimodal LLMs (MLLMs) for video understanding, which process frames as isolated 2D snapshots, instead of the persistent scene humans perceive. We revisit pose as a lightweight supervisory signal and introduce Cambrian-P, a video MLLM augmented with per-frame learnable camera tokens and a pose regression head. With a carefully designed sampling scheme, the model achieves substantial gains of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, generalizes across eight additional spatial and general video QA benchmarks, and, as a byproduct, achieves state of the art streaming pose estimation on ScanNet. Surprisingly, training on pseudo-annotated poses from in-the-wild video further improves general video QA benchmarks, showing pose helps beyond spatial reasoning. Together, these results position camera pose as a fundamental signal for video models that reason about the physical world.
Large Vision Language Models (LVLMs) excel at semantic understanding but struggle with fine grained spatial grounding, as the model must implicitly infer complex geometry without ever producing a spatial interpretation. We present Perceptio, a perception enhanced LVLM with 2D and 3D spatial reasoning abilities, enabled via explicit semantic segmentation tokens and depth tokens generated directly within the autoregressive sequence. Concretely, we (i) distill a VQVAE depth codebook from a strong monocular teacher to tokenize dense depth into compact sequences, and (ii) integrate SAM2 based semantic segmentation tokens and VQ-VAE depth tokens inside the LLM so the model first emits spatial tokens and then answers. To stabilize depth token generation, we introduce novel composite depth-token objectives (marker, token, and count losses) and a soft-merging technique for differentiable reconstruction. We adopt a multi-task co-training strategy across diverse datasets, letting the model learn perception tokens to tackle multiple downstream tasks. Building on InternVL, Perceptio achieves state-of-the-art performance across benchmarks: improving referring expression segmentation by +0.8/+1.4/+1.1 cIoU on RefCOCO/+/g HardBLINK spatial understanding accuracy by 10.3%, and MMBench accuracy by 1.0%, demonstrating that explicit spatial chain-of-thought materially strengthens spatial grounding in LVLMs.