Training a spatial VLM to answer which side an object lies on does not require it to preserve the metric displacement that determines that answer, while supervision for a distance estimate may leave categorical relations largely unconstrained. When post-training emphasizes one readout, improvements can therefore concentrate on that interface while transfer to others remains weak or even degrades; we call this observed specialization interface lock-in. ChronoReplay addresses it by replaying heterogeneous spatial supervision as chronologically ordered episodes, retaining source-native targets, and balancing supervision across families. Under the same observation access, a single ChronoReplay checkpoint improves OVO-S-Bench, OST-Bench, and UCS-Bench together, including a 13.41-point gain on OST-Bench, and achieves the highest three-benchmark mean among the supervision settings compared here. Same-data controls further separate the ingredients of this transfer: answer serialization redistributes gains across OST and UCS, family-balanced learning improves over ordinary joint SFT and natural turn weighting, and chronological replay exceeds a shifted-order control by about 2.41 OST points across two paired seeds. Adaptive frame selection then changes evidence access while keeping the trained model fixed, providing an additional inference-time gain at lower visual cost. Streaming spatial performance depends on what distinctions supervision preserves, how heterogeneous histories are organized during learning, and what causal evidence inference makes available at test time.
Figures & tables
Figure 1: Why reusable spatial state. Categorical preservation and trajectory supervision constrain complementary spatial distinctions. Causal Replay combines them in shared parameters. The illustrated readouts span categorical, metric, temporal, and reference-frame queries.
Figure 2: Overview of ChronoReplay. Stage I (Causal Replay construction) builds ordered trajectory supervision DT and single-turn preservation examples DP . Stage II (family-balanced learning) combines their normalized native-format risks with weights 1−λ and λ . Out of scope marks observations beyond a query’s causal frontier.
Level 1
Level 2
Level 3
Level 4
Overall
Frames
Model
1.1
1.2
1.3
2.1
2.2
2.3
3.1
3.2
3.3
4.1
4.2
4.3
Representative official prefix-access models
InternVL-3.5-8B
47.86
54.60
35.31
40.95
40.31
56.22
54.00
26.98
60.61
34.65
55.00
28.30
44.57
128
Qwen3-VL-4B
32.91
53.19
43.65
47.26
34.25
63.04
57.33
38.10
68.18
25.74
70.00
28.30
46.83
128
Qwen3-VL-32B
48.83
57.07
44.51
50.71
34.58
70.43
60.67
26.98
68.18
32.67
68.33
22.64
48.80
128
InternVL-3.5-38B
58.64
55.56
49.74
50.30
48.00
64.91
61.33
22.22
53.03
38.61
68.33
18.87
49.13
128
Table 1: Complete task-family results on OVO-S-Bench (%), twelve official task families. Our rows report accuracy within each family; Overall is their unweighted mean. Published reference scores are quoted from the benchmark paper. Uniform-128 is the matched training comparison; the final row applies adaptive access to the same trained model.
Avg
Agent State
Agent Visible Information
Agent–Object Spatial Relationship
Position
Orientation
Existence
Quantity
Diversity
Order
Direction
Distance
Model
JUD.
EST.
JUD.
EST.
JUD.
TEMP.
CNT.
JUD.
JUD.
JUD.
TEMP.
EST.
JUD.
TEMP.
EST.
Representative published MLLMs
InternVL-2.5-8B
44.6
51.7
26.1
49.4
40.4
86.3
51.3
56.4
60.7
38.4
37.2
33.8
22.8
43.0
42.9
27.9
Qwen2.5-VL-72B
45.6
57.0
27.6
52.2
37.1
86.1
64.5
61.5
75.7
34.5
41.4
21.1
8.2
44.5
39.3
18.7
Qwen2.5-VL-32B
46.9
51.0
31.1
53.5
39.4
85.3
64.8
59.2
73.4
41.8
39.5
24.9
25.7
43.6
39.1
20.3
Table 2: Fine-grained results on OST-Bench (%), official fifteen subtypes. Published rows are reference points under their reported full-split protocols; shaded rows use our matched scene-disjoint evaluation. Δ is ChronoReplay minus base. JUD., TEMP., CNT., and EST. denote judgement, temporal localization, counting, and estimation.
Trajectory
Position
Proximity
Category
Overall
Model
Size
Frames
All
My
Obj.
All
Ego
Rel.
All
Dist.
Reach
All
Cat.
Qty.
Representative official references
DeepSeek-V3
–
50
42.0
41.0
44.8
33.2
35.4
31.6
37.7
45.9
31.2
32.3
36.8
30.3
35.6
GPT-5
–
50
44.5
43.3
48.1
37.8
40.2
36.3
55.4
52.9
57.5
41.4
58.8
33.5
43.7
InternVideo2.5
8B
64
44.2
43.9
45.2
39.4
42.4
37.5
52.7
49.1
55.5
42.2
53.0
37.4
43.8
Qwen3-VL-8B
8B
64
44.6
43.1
48.7
40.1
43.6
37.9
50.9
49.2
52.1
42.9
52.8
38.4
44.0
Table 3: Complete UCS-Bench decomposition over all 8,114 timestamp-causal questions (%). Each group reports its aggregate (All) and two subcategories; category names follow Appendix B . Published rows are reference points under their reported evaluation protocols; Δ is ChronoReplay minus base.
(a) Single-family supervision
(b) Mechanism controls
Training
OVO-S
OST
UCS
Mean
Control: alternative
Alt.
ChronoReplay
Scope
Official base
48.98
50.52
45.33
48.28
Format: Canonical
58.03
58.75
OST/UCS
Categorical-only
58.03
47.48
46.72
50.74
Order: Shifted
61.81
64.22
OST
Trajectory-only
52.03
51.03
49.22
50.76
Mass: α=1.0
60.92
61.20
OVO-S/OST
ChronoReplay
53.89
63.93
52.77
56.86
Topology: 4 GPUs
57.20
56.86
All three
Table 4: Diagnostic analysis of ChronoReplay (%). Panel (a): categorical-only, trajectory-only, and full ChronoReplay on Uniform-128 OVO-S and complete OST/UCS. ChronoReplay uses the canonical model from Tables 1 – 3 ; Mean averages the three benchmarks. Panel (b): each alternative is compared with its corresponding ChronoReplay reference under the scope shown. Order averages two seed pairs; the mass sweep uses adaptive OVO-S.
Figure 3: One origin, two native readouts on an OST-Bench episode. Left: the five observation rounds and two questions. Q1 (Round 4): when were both the bathtub and the folded towel on the observer’s left? Q2 (Round 5): using the Round-2 facing as forward, did the observer move left or right? Right: ChronoReplay answers Round 2 and Left, both correct, whereas the origin-shifted control ( Shifted ) returns Round 4 and Right under identical observations. Teal marks annotated states, red the control’s errors; the full-size episode is in Appendix F .
Checkpoint
Access
L1
L2
L3
L4
Overall
Frames
Official base
Uniform-128
54.07
57.21
49.48
35.16
48.98
128
ChronoReplay
Uniform-128
58.60
61.68
55.11
40.17
53.89
128
ChronoReplay
Adaptive
67.26
65.10
57.26
42.30
57.98
80.50
Training effect (Uniform-128 − base)
+4.54
+4.46
+5.63
+5.01
+4.91
–
Access effect (adaptive − Uniform-128)
+8.66
+3.43
+2.15
+2.13
+4.09
–
Table 5: Training–access decomposition on OVO-S (%). Levels average their three family accuracies; Overall averages all twelve. Summary rows separate training and access effects, using unrounded scores.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Family
Nf
Base
Uniform-128
Adaptive
1.1
167
96
105
116
1.2
194
117
125
138
1.3
268
119
130
164
2.1
198
108
117
119
2.2
168
79
87
98
2.3
147
103
109
113
Appendix
Table 6: Correct answers underlying the OVO-S main results. Uniform-128 and Adaptive denote ChronoReplay; Base uses Uniform-128.
Family
Targets
Sequences
Supervised states
Loss mass
Preservation
Categorical
30,635
30,635
1/3
Numerical
2,545
2,545
1/3
Trajectory
Mixed spatial
6,381
45,006
1/3
Appendix
Table 7: Composition and relative loss mass of the Causal Replay pool at α=0.5 . A supervised state is one causal prefix with a native target. Mass denotes the sum of training weights, normalized over the whole pool.
Input: base VLM θ0 ; preservation pool DP with weights wr ; multi-round sources; trajectory mass α
Output: trained parameters θ
Stage I: Causal Replay construction
1:
DT←∅
2:
for each multi-round source episode i , turns kept in original order do
Algorithm 1 ChronoReplay: Causal Replay construction and family-balanced learning.
Setting
Value
Base model
Qwen3-VL-8B-Instruct
Updated components
Language model and visual merger
Epochs / global batch size
1 / 32
Learning rate
2×10−6
Schedule
Cosine decay with 3% warmup
Optimizer
AdamW, β=(0.9,0.999)
Appendix
Table 8: Shared optimization settings for ChronoReplay and its matched controls.
Training objective
OVO-S
OST
UCS
Mean
Base
48.98
50.52
45.33
48.28
Ordinary token SFT
51.8
60.7
49.8
54.10
Natural turn weighting
52.9
62.3
51.5
55.57
ChronoReplay
53.89
63.93
52.77
56.86
Appendix
Table 9: Joint-training objective comparison (%). OVO-S uses Uniform-128; OST and UCS use the same fixed protocols as the main results. Mean is the unweighted average of the three benchmark scores.
OVO-S
OST
UCS
Pair
Natural
Full
Δ
Natural
Full
Δ
Natural
Full
Δ
1
52.90
53.89
+0.99
62.30
63.93
+1.63
51.50
52.77
+1.27
2
52.71
53.33
+0.62
62.58
63.83
+1.25
51.68
52.56
+0.88
3
53.02
54.17
+1.15
62.14
63.56
+1.42
51.39
52.74
+1.35
Mean
52.88
53.80
+0.92
62.34
63.77
+1.43
51.52
52.69
+1.17
SD of Δ
–
–
0.27
–
–
0.19
–
–
0.25
Appendix
Table 10: Three-seed replication of the objective comparison (%). Natural: natural turn weighting; Full: ChronoReplay; Δ : Full minus Natural in percentage points. Pair indices enumerate runs. Differences and summaries use the displayed scores; summary values are rounded after aggregation.
α
OVO-S
OST
Mean
0.25
57.56
62.73
60.15
0.50
58.27
64.13
61.20
1.00
57.80
64.04
60.92
Appendix
Table 11: Sensitivity to the relative weight of trajectory supervision in a separate matched run set (%). OVO-S uses the fixed adaptive access policy.
Training order
Run 1
Run 2
Cyclic shift
61.51
62.11
Chronological
63.93
64.52
Appendix
Table 12: Chronological and shifted replay on OST (%). Each row compares models trained with the same seed and evaluated on the same scenes.
Run
GPUs
OVO-U128
OVO-Adaptive
OST
UCS
Model-only mean
Headline checkpoint
8
53.89
57.98
63.93
52.77
56.86
Fresh replication
4
54.10
58.00
63.24
54.26
57.20
Appendix
Table 13: Fresh training-topology replication under complete model-only evaluation (%), together with the frozen OVO-S system protocol. Observation access and scorers are fixed within each reported column.
Training
OVO-S
OST
UCS
Mean
Official base
48.98
50.52
45.33
48.28
Categorical-only
58.03
47.48
46.72
50.74
Trajectory-only
52.03
51.03
49.22
50.76
ChronoReplay
53.89
63.93
52.77
56.86
Appendix
Table 14: Single-family diagnostic on OVO-S, OST, and UCS (%). The ChronoReplay row uses the canonical model from Tables 1 – 3 . Mean averages the three benchmarks under the evaluation protocols described above.
Local
Multiscale
Global
Query scope
Current state
Motion and event order
Layout and historical references
Coverage
Recent observations
Last 30 s; 30–300 s; earlier history
Full observed prefix
Sampling
Dense tail at 4 fps
60% / 30% / 10% of the frame budget
Uniform
Frame budget
8
128
128
Appendix
Table 15: Temporal coverage for different spatial queries. Frame budgets are upper limits; short histories and duplicate removal can yield fewer frames.
Figure 4: Historical context in spatial reasoning. The first question retrieves an earlier state in which the bathtub and folded towel lie on the observer’s left. The second evaluates a later location relative to the facing direction in Round 2. Teal indicates annotated states, and red shows the control’s incorrect retrieval or displacement. Both questions depend on a reference established earlier in the episode.
Figure 5: First appearance and category order. In scene 1mp3d_0067_region16 , ChronoReplay returns Round 3 for the first appearance of the toilet and the order door, shelf, toilet paper. The matched origin-shifted control returns Round 1 and reverses the latter two categories. The early white fixture is a bidet; the toilet is partially visible at the lower-right edge of the final Round 3 frame.
Figure 6: Historical retrieval and rotation. In scene Validation_42898818 , the Round 5 question asks which round contained both the plant and sink; ChronoReplay returns Round 2, whereas the control returns Round 4. A Round 7 question defines the Round 1 facing as the orientation reference and specifies a rotation below 180∘ . ChronoReplay correctly returns counterclockwise, while the control returns clockwise.
Figure 7: Reference displacement and nearest-state retrieval. In scene scene0496_00 , ChronoReplay identifies leftward displacement relative to the Round 1 facing, while the control answers right. When asked in Round 4 for the closest blackboard state, ChronoReplay retrieves Round 2 rather than the control’s Round 4. Left/right denotes displacement, not turning direction; the closest-state label follows the source annotation rather than apparent object size in a single image.
Figure 8: Two native interfaces on one mirror history. Selected chronological observations precede a left/right judgement and a horizontal-distance estimate. Round labels identify the source observation turns. Base, Preservation, Trajectory, and Joint replay use the same observations at each query endpoint. Joint retains the correct direction; Trajectory has slightly lower absolute distance error, and both tie at 0.70 under the official metric.
Figure 9: A fixed-observation global-layout failure. The official Base chooses B and ChronoReplay chooses D on OVO-S L4.3. The upper panel shows eight timestamped frames from their shared Uniform-128 input. The lower panels reproduce the two source option diagrams in their original orientations, with enlarged labels beside the key objects. Captions describe positions within these diagrams. The fireplace and table placements make the disagreement visible.
Model
Params
L1
L2
L3
L4
Overall
Proprietary MLLMs
GPT-5.4
–
54.64
57.59
50.80
40.53
50.89
Gemini-3.1-Pro
–
61.92
64.04
55.90
54.90
59.19
Gemini-3.1-Flash-Lite
–
54.09
52.19
54.14
42.84
50.81
Grok-4.1-Fast
–
44.80
46.59
48.49
35.04
43.73
Open-source MLLMs
Appendix
Table 16: Additional published OVO-S aggregate reference rows (%). These rows provide external context and are not used for matched training attribution.
Model
Overall
A. State
A. Info
AO
Judgement
Temporal
Counting
Estimation
Proprietary MLLMs (published)
Claude-3.5-Sonnet
47.8
45.6
65.6
32.9
57.4
46.7
57.9
26.9
Gemini-2.0-Flash
49.5
45.1
70.8
33.8
61.1
45.8
59.4
28.1
Gemini-2.0-Flash (Thinking)
54.2
47.1
72.3
42.8
63.6
61.1
61.9
28.9
GPT-4o
48.7
38.8
72.8
33.5
59.8
48.6
59.8
24.0
GPT-4.1
53.4
47.2
76.5
37.7
66.4
51.7
60.6
29.1
Appendix
Table 17: Aggregate OST-Bench summary and published reference set (%). Published rows use the released full split; the final block is compared only within the matched scene-disjoint protocol.
Model
Size
Frames
Trajectory
Position
Proximity
Category
Overall
Human and proprietary references
Human
–
–
91.7
87.6
96.8
90.0
91.0
DeepSeek-V3
–
50
42.0
33.2
37.7
32.3
35.6
Doubao-Seed-1.6V
–
50
49.6
42.8
56.0
44.6
47.3
GPT-5
–
50
44.5
37.8
55.4
41.4
43.7
Gemini-3-Flash
–
50
43.8
38.0
52.9
38.9
42.2
Appendix
Table 18: Published UCS aggregate reference set and matched 8,114-question evaluation (%). Published rows use their reported frame protocols.
Variant
OVO-S
OST
UCS
Readout
ChronoReplay, no relation loss
53.89
63.93
52.77
–
Local compatibility, K=1
53.73
64.08
52.65
63.33
Same type, K=4
53.50
64.06
51.70
62.29
Cross type, K=4
52.07
64.34
51.77
66.88
State-path firewall
52.18
63.42
54.41
58.13
Appendix
Table 19: Overview of relation-supervision diagnostics. Native columns are complete model-only benchmark scores; Readout is the matched withheld compatibility diagnostic (%).
Setting
Value
Base-model revision
0c351dd01ed87e9c1b53cbc748cba10e6187ff3b
Headline / target-format seed
42
Chronology seeds
42 (Run 1); 73 (Run 2)
Headline and chronology topology
8 GPUs; gradient accumulation 2
Target-interface topology
4 GPUs; gradient accumulation 4
Appendix
Table 20: Execution settings for the headline, target-format, and chronology comparisons in Appendix D .
Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two crucial deployment-critical requirements: long-horizon streaming inference and active perception when the current view is insufficient. To address this gap, we introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration, where queries are temporally grounded to specific timestamps and must be answered using only observations available up to that moment. S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions, and real-world streaming videos that capture practical sensing artifacts for rigorous generalization evaluation. Overall, it spans 10K+ scenes and 26K+ trajectories, with dedicated training (S3-Train) and evaluation (S3-Eval) splits. We further propose AMF-VLM, which supports streaming spatial reasoning under bounded computing via (i) memory folding, which compresses long-horizon observations into compact structured memory, and (ii) active exploration, which outputs explicit actions (e.g. move/rotate/scan) to acquire missing evidence before answering. Extensive experiments demonstrate that, compared to models using identical training data, our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively, while maintaining competitive transferability to standard spatial benchmarks.
Yuxi Wei, Wei Huang, Qirui Chen +2
The University of Hong Kong · Shanghai Jiao Tong University · Yinwang Intelligent Technology Co. Ltd.
Multimodal agents in robotics, AR, and autonomous driving must reason about places and layouts from continuous egocentric streams, often using evidence outside the current view. Existing benchmarks either evaluate offline over full videos or target events rather than spatial structure. We introduce OVO-S-Bench, a fully human-annotated benchmark for streaming spatial intelligence, comprising 1,680 questions over 348 source videos. Annotation involves 12 trained annotators, each also serving as a blind cross-reviewer, across roughly 804 person-hours of multi-round quality assurance. Each question carries a query timestamp and an evidence interval, and at evaluation, the model sees only the prefix preceding the query. Questions span four levels of increasing abstraction: instantaneous egocentric perception, spatiotemporal context tracking, spatial simulation and reasoning, and allocentric mapping. Across 38 proprietary and open-source MLLMs, Gemini-3.1-Pro trails human experts by 27 points, 59.2 vs. 86.6, with allocentric mapping as the dominant bottleneck. Notably, streaming and spatially fine-tuned MLLMs underperform their own backbones. We further find that chain-of-thought reasoning amplifies spatial errors when ungrounded in the stream. By exposing these limitations, OVO-S-Bench establishes a demanding testbed for next-generation streaming spatial MLLMs.
Yifei Li, Pengyiang Liu, Yuhang Zang +4
Tsinghua University · Shanghai AI Laboratory · Beihang University
Embodied agents must reason about 3D space while the video is still arriving, answering questions as soon as they have observed enough of the scene. VLMs that incorporate 3D geometric priors achieve strong spatial reasoning, but they operate offline, i.e., the full video must be available before they produce an answer. Streaming VLMs process frames causally and decide for themselves when to respond, yet they lack explicit 3D representations. We present SpaTime, a streaming VLM that fuses causal geometry tokens into the language model at every frame, using only the frames observed so far. To supervise when the model answers, we propose a response-time loss that maps per-frame response probabilities to a differentiable expected response time and penalizes the distance from the ground-truth frame. For evaluation, we construct StreamVSTI-Bench and StreamVSI-Bench, streaming adaptations of VSTI-Bench and VSI-Bench. On StreamVSTI-Bench, SpaTime reaches 49.2% overall accuracy and reduces the mean response-time error by 66% relative to the strongest streaming baseline.
Hairong Yin, Huangying Zhan, Shin-Fang Chng +2
Department of Computer Science, Purdue University, USA · Goertek Alpha Labs, USA