See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning

Yuxi Wei; Wei Huang; Qirui Chen; Lu Hou; Xiaojuan Qi

cs.CVMar 25, 2026

See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning

Authors: Yuxi Wei, Wei Huang, Qirui Chen, Lu Hou, Xiaojuan Qi

Organizations: The University of Hong Kong · Shanghai Jiao Tong University · Yinwang Intelligent Technology Co. Ltd.

Abstract

Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two crucial deployment-critical requirements: long-horizon streaming inference and active perception when the current view is insufficient. To address this gap, we introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration, where queries are temporally grounded to specific timestamps and must be answered using only observations available up to that moment. S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions, and real-world streaming videos that capture practical sensing artifacts for rigorous generalization evaluation. Overall, it spans 10K+ scenes and 26K+ trajectories, with dedicated training (S3-Train) and evaluation (S3-Eval) splits. We further propose AMF-VLM, which supports streaming spatial reasoning under bounded computing via (i) memory folding, which compresses long-horizon observations into compact structured memory, and (ii) active exploration, which outputs explicit actions (e.g. move/rotate/scan) to acquire missing evidence before answering. Extensive experiments demonstrate that, compared to models using identical training data, our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively, while maintaining competitive transferability to standard spatial benchmarks.

Figures & tables

Figure 1: Overview of the $S^{3}$ -Bench and AMF-VLM. Unlike traditional offline methods that answer questions post-video, our $S^{3}$ -Bench targets streaming spatial understanding with timestamp-specific, ego-centric, and temporally dependent queries. The figure outlines the AMF-VLM pipeline and presents its performance comparison with baselines on $S^{3}$ -Eval.

Figure 2: $S^{3}$ Benchmark data construction pipeline. Simulated digital assets are filtered and rendered into videos via geometry-guided path planning. Next, object bounding boxes and frame-level visibility are extracted to generate template-based QA pairs, which are then rewritten for linguistic diversity.

Figure 3: Distribution and examples of QA categories in the $S^{3}$ -Bench. This figure illustrates the statistical distribution of different QA categories alongside representative examples. The benchmark primarily comprises three question types: (1) Ego-centric , requiring understanding based on the current camera position; (2) Spatial relation , assessing the comprehension of the scene’s spatial information; and (3) Temporal dependency , involving streaming queries that rely on preceding context.

Ego-centric
Task [Format]	Question
camera_displacement [Num]	How far did the camera move from the past X seconds?
current_room_area [Num]	What is the total area of the current room?
cam_obj_distance [Num]	What is the distance between the camera and the A?
identification_closest [MC]	Which object is currently spatially closest to the camera?
camera_motion_target [MC]	Which object did the camera approach the most?

Table 1: Categories and examples of

S^{3}

-Bench QA pairs. Tasks span three domains: Ego-centric, Spatial Relation, and Temporal Dependency. Each entry specifies the task, answer format (Numerical/Multiple Choice), and a question template.

Figure 4: AMF-VLM architecture. Visual features are extracted via a vision encoder, optionally supplemented by a spatial encoder. For efficient streaming, recent frames are densely sampled while long-term history is sparsely retained and compressed via memory folding. Finally, the model outputs either an exploration action, a direct answer, or a memory update summary.

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	Id-	Rel-	Area	C-	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Dist	Mo	Clo	Ori		Area				Id
Random	25.6	-	21.0	-	-	29.1	-	-	23.7	22.2	30.7	-	-	-	-	26.2	26.6
Frequent	35.6	46.1	26.1	47.6	46.7	26.0	33.0	39.3	26.9	27.7	37.1	37.6	35.9	46.5	39.6	26.8	25.8
Human level	80.5	71.1	92.6	70.4	70.9	93.0	71.3	71.7	98.6	92.6	97.1	73.4	70.9	66.1	66.0	93.5	92.8
GPT-5.2 [ 1 ]	44.9	38.4	29.7	37.4	34.6	41.8	35.9	45.5	61.4	58.6	44.4	35.6	44.4	51.5	53.4	49.6	52.7
Gemini-3-flash [ 45 ]	41.9	35.6	27.6	34.5	31.5	38.8	32.7	42.3	58.1	55.5	41.5	32.5	41.6	48.5	49.6	46.8	49.6

Table 2: Performance comparison of baselines across all

S^{3}

-Eval simulation part. (*) indicates models also trained on our data. (

\dagger

) indicates that the model is modified as streaming inference.

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	Id-	Rel-	Area	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Dist	Mo	Clo	Ori					Id
Random	26.1	-	23.1	-	-	25.2	-	-	25.1	24.6	34.4	-	-	-	23.5	24.5
Frequent	29.0	25.3	25.8	32.6	34.0	25.9	33.5	24.1	25.9	26.1	33.7	11.0	29.5	37.5	26.8	26.9
Human level	82.1	69.1	90.8	69.1	74.7	91.5	68.5	77.5	91.9	91.3	95.6	75.8	69.1	74.3	96.1	91.9
GPT-5.2 [ 1 ]	41.7	28.1	29.8	31.5	25.8	59.3	25.5	39.5	61.8	51.4	44.7	48.2	27.8	54.7	49.5	52.2
Gemini-3-flash [ 45 ]	38.2	24.6	27.2	26.7	22.9	56.3	21.9	36.2	57.2	48.2	41.6	47.2	23.1	51.4	46.6	48.9

Table 3: Performance comparison of baselines across all

S^{3}

-Eval real part. (*) indicates models also trained on our data. (

\dagger

) indicates streaming inference modification.

Table 8

Figure 5: Qualitative examples of Active Exploration. Restricted initial views can lead to incorrect direct predictions. To mitigate this, the model executes exploration actions to gather sufficient visual context before responding.

Figure 6: Qualitative examples of Memory Folding. AMF-VLM periodically compresses observed scenes into concise text, efficiently retaining long-term spatial memory and mitigating forgetting over extended sequences.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	Id-	Rel-	Area	C-	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Dist	Mo	Clo	Ori		Area				Id
Qwen3-VL-8B [ 4 ]	41.1	36.0	25.8	34.0	29.7	39.6	32.0	41.1	56.7	54.2	40.2	32.5	41.5	48.1	48.8	45.4	49.1
Think3D [ 68 ]	44.0	36.5	27.4	35.4	31.4	43.7	33.4	44.9	59.0	59.5	47.2	34.9	46.3	46.9	52.5	49.3	50.6
CoV [ 69 ]	43.2	36.8	26.1	35.5	30.0	41.3	33.2	42.4	57.1	57.3	44.8	34.4	45.1	51.2	54.1	47.6	48.6
Ours	62.9	57.6	35.0	55.6	59.4	70.4	70.8	67.7	75.6	72.8	46.3	67.0	68.1	70.8	61.8	60.9	60.6

Appendix

Table 7: Performance comparison of active vision baselines across all

S^{3}

-Eval simulation part.

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	Id-	Rel-	Area	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Dist	Mo	Clo	Ori					Id
Qwen3-VL-8B [ 4 ]	39.0	25.3	27.7	27.6	23.7	57.3	22.1	36.1	59.8	49.3	41.6	48.5	25.0	51.9	47.3	50.0
Think3D [ 68 ]	42.0	28.0	30.1	29.4	25.6	61.0	24.7	37.8	63.9	53.3	46.9	50.3	28.0	54.1	49.9	52.0
CoV [ 69 ]	40.8	26.6	28.8	28.7	24.8	57.2	22.5	38.2	65.4	51.6	44.4	49.6	25.8	54.5	48.8	51.5
Ours	57.8	41.0	37.0	48.2	50.1	71.8	45.1	77.5	75.5	59.7	44.5	49.7	70.9	64.0	72.0	64.4

Appendix

Table 8: Performance comparison of active vision baselines across all

S^{3}

-Eval real part.

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	Id-	Rel-	Area	C-	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Dist	Mo	Clo	Ori		Area				Id
K=2	60.2	53.2	33.8	51.3	57.9	68.5	66.7	65.3	69.1	69.5	45.1	65.7	64.5	69.7	58.9	59.2	58.8
K=5	61.6	58.0	33.3	53.4	58.7	69.9	68.4	64.2	73.5	69.5	43.4	69.6	68.3	70.4	61.7	59.2	57.5
K=10	62.9	57.6	35.0	55.6	59.4	70.4	70.8	67.7	75.6	72.8	46.3	67.0	68.1	70.8	61.8	60.9	60.6
K=15	62.4	56.8	34.5	50.8	59.7	70.9	73.8	68.2	72.8	69.7	45.9	64.8	69.1	69.7	61.7	61.1	61.9

Appendix

Table 9: Ablation under different K.

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	Id-	Rel-	Area	C-	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Dist	Mo	Clo	Ori		Area				Id
10K	52.8	48.3	29.4	43.8	48.9	60.9	59.5	57.8	64.9	61.2	40.6	57.6	55.9	56.9	52.0	48.5	52.2
100K	58.4	53.9	33.3	52.6	52.3	65.5	66.5	63.0	68.6	70.2	43.0	63.2	61.3	65.3	55.0	58.3	56.3
300K	60.4	54.0	34.3	52.7	57.4	68.0	70.5	63.5	74.9	70.6	45.5	65.1	65.3	65.7	59.6	57.2	54.5
600K	62.9	57.6	35.0	55.6	59.4	70.4	70.8	67.7	75.6	72.8	46.3	67.0	68.1	70.8	61.8	60.9	60.6

Appendix

Table 10: Performance comparison under different data scale.

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	C-	Id-	Rel-	Area	C-	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Rel	Dist	Mo	Clo	Ori		Area				Id
GPT-5.2 [ 1 ]	26.0	7.8	31.2	35.8	39.7	33.7	26.2	31.1	20.0	38.6	28.8	48.4	7.5	28.6	1.8	1.0	34.9	35.0
Gemini-3-flash [ 45 ]	27.1	8.2	33.0	38.7	43.2	36.4	26.5	30.2	22.1	40.3	29.8	47.4	7.3	30.6	1.8	1.0	37.2	35.0
Gemini-3-pro [ 45 ]	26.8	8.1	31.2	36.6	42.0	35.7	26.3	32.6	21.9	41.0	28.7	47.6	7.4	29.7	1.8	1.0	34.7	37.9
llava-video [ 66 ]	21.0	6.4	24.8	28.7	33.5	28.2	20.9	24.4	16.9	30.0	22.7	37.5	6.3	23.4	1.3	0.8	27.3	29.7
llava-ov-1.5-4B [ 2 ]	19.7	5.7	23.0	25.6	29.3	26.0	20.1	22.9	14.9	30.4	20.9	35.6	5.7	22.8	1.3	0.8	27.6	28.1

Appendix

Table 11: Blind testing of baselines across all

S^{3}

-Eval simulation part. (*) indicates models also trained on our data. (

\dagger

) indicates streaming inference modification.

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	C-	Id-	Rel-	Area	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Rel	Dist	Mo	Clo	Ori					Id
GPT-5.2 [ 1 ]	26.5	8.9	28.2	32.0	36.1	43.1	24.1	34.3	6.3	25.8	36.2	45.7	7.3	2.0	0.2	32.0	34.2
Gemini-3-flash [ 45 ]	28.3	9.0	28.6	34.3	40.6	45.7	27.3	36.4	6.8	28.1	38.1	49.9	7.6	2.1	0.2	33.8	35.1
Gemini-3-pro [ 45 ]	27.6	9.0	27.8	36.4	37.3	46.4	27.2	35.4	6.6	27.1	35.8	44.2	7.2	2.1	0.2	35.0	35.5
llava-video [ 66 ]	27.5	9.0	30.0	32.6	40.4	46.4	24.7	35.9	6.6	27.4	36.1	45.9	6.9	2.0	0.2	33.5	35.8
llava-ov-1.5-4B [ 2 ]	26.5	8.7	27.9	33.7	37.9	44.4	26.2	34.6	6.3	27.0	34.6	43.0	6.9	2.0	0.2	30.8	33.3

Appendix

Table 12: Blind testing of baselines across all

S^{3}

-Eval real part. (*) indicates models also trained on our data. (

\dagger

) indicates streaming inference modification.

Method	Over-	T-C	T-	T-Spa	Spa-	Spa-	C-	C-	C-	Id-	Rel-	Area	C-	Attr	Cou	Seq	Seq
	all	-O-D	Hor	-D	D	Pro	O-D	Dist	Mo	Clo	Ori		Area				Id
Baseline (data)	56.0	54.9	31.0	51.8	45.2	58.3	66.0	60.9	67.0	65.6	40.0	59.1	60.1	63.2	55.8	55.2	55.0
+ active	58.1	54.9	31.0	51.8	53.0	64.8	66.0	60.9	67.0	65.6	43.6	63.4	64.5	67.4	59.2	55.2	55.0
+ folding	60.9	57.2	34.5	55.0	58.4	69.7	67.0	62.2	68.9	67.1	45.9	66.4	65.2	70.3	60.9	60.0	59.9
+ 3D enc	62.9	57.6	35.0	55.6	59.4	70.4	70.8	67.7	75.6	72.8	46.3	67.0	68.1	70.8	61.8	60.9	60.6

Appendix

Table 13: Detailed module ablation study.

Method	Over-	App-	Abs-	Obj-	Dir-	Dir-	Dir-	Rel-	Obj-	Room-	Route
	all	Order	Dist	Count	Easy	Hard	Med	Dist	Size	Size
Ours*	55.0	61.2	38.1	57.2	61.3	61.1	61.1	61.0	58.8	53.5	33.0
Ours	67.5	75.1	46.8	70.2	75.1	75.1	75.1	74.9	72.2	65.7	40.2

Appendix

Table 14: Detailed results on VSI-Bench. (*) denotes that the model is not trained on our data.

Method	Over-	Sim-	Count-	Depth	Jigsaw	Art	Fun.	Sem.	Spatial	Local.	Vis.	Multi-	Reflect.	Foren-	IQ
	all	ilarity	ing				Corr.	Corr.			Corr.	view		sic
Ours*	62.4	88.2	75.0	87.1	62.0	75.2	33.8	47.9	88.1	67.2	80.8	42.9	30.6	71.2	26.0
Ours	65.2	92.6	78.3	90.3	64.7	78.6	35.4	49.3	91.6	70.4	84.9	45.1	32.1	74.2	26.7

Appendix

Table 15: Detailed results on BLINK. (*) denotes that the model is not trained on our data.

Method	Over-	Cam.-	Obj.-	Reg.-	Cam.-	Obj.-	Reg.-	Meas.	Appr.	Cam.	Obj.	MSR
	all	Cam.	Obj.	Reg.	Obj.	Reg.	Cam.
Ours*	31.2	29.0	29.8	29.6	31.4	35.3	31.3	29.7	21.2	24.3	47.4	31.8
Ours	30.5	28.0	28.7	28.4	30.2	34.1	30.1	29.7	21.2	24.3	46.1	31.8

Appendix

Table 16: Detailed results on MMSI-Bench. (*) denotes that the model is not trained on our data.

Figure 7: Single round QA pair samples of $S^{3}$ -Bench.

Figure 8: Temporal dependency QA pair samples of $S^{3}$ -Bench.

Figure 9: Samples of meshes of the simulation part of $S^{3}$ -Bench, with the related diverse planned trajectories.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 159 papers

SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning
Oct 6, 2026Hairong Yin, Huangying Zhan, Shin-Fang Chng +2Vision-Language Models Spatiotemporal Reasoning
Department of Computer Science, Purdue University, USA · Goertek Alpha Labs, USA
OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs
Jun 2, 2026Yifei Li, Pengyiang Liu, Yuhang Zang +4Spatial Reasoning Benchmarks Spatiotemporal Reasoning
Tsinghua University · Shanghai AI Laboratory · Beihang University
MemAct: Streaming Spatial Understanding with Memory and Active Exploration
Oct 13, 2026Yuxi Wei, Wei Huang, Qirui Chen +2Memory-Augmented Agents Embodied Reasoning
The University of Hong Kong · Shanghai Jiao Tong University · Yinwang Intelligent Technology Co. Ltd

Oct 6, 2026J/K move · Enter open · S save

SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning

Hairong Yin, Huangying Zhan, Shin-Fang Chng, Yi Xu+1

Department of Computer Science, Purdue University, USA · Goertek Alpha Labs, USA

Vision-Language Models Spatiotemporal Reasoning VLM Reasoning 3D VLMs Temporal Video Understanding

Open paper AskPDF arXiv

Embodied agents must reason about 3D space while the video is still arriving, answering questions as soon as they have observed enough of the scene. VLMs that incorporate 3D geometric priors achieve strong spatial reasoning, but they operate offline, i.e., the full video must be available before they produce an answer. Streaming VLMs process frames causally and decide for themselves when to respond, yet they lack explicit 3D representations. We present SpaTime, a streaming VLM that fuses causal geometry tokens into the language model at every frame, using only the frames observed so far. To supervise when the model answers, we propose a response-time loss that maps per-frame response probabilities to a differentiable expected response time and penalizes the distance from the ground-truth frame. For evaluation, we construct StreamVSTI-Bench and StreamVSI-Bench, streaming adaptations of VSTI-Bench and VSI-Bench. On StreamVSTI-Bench, SpaTime reaches 49.2% overall accuracy and reduces the mean response-time error by 66% relative to the strongest streaming baseline.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning

OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs

MemAct: Streaming Spatial Understanding with Memory and Active Exploration