MemAct: Streaming Spatial Understanding with Memory and Active Exploration

Yuxi Wei; Wei Huang; Qirui Chen; Lu Hou; Xiaojuan Qi

cs.CVOct 13, 2026

MemAct: Streaming Spatial Understanding with Memory and Active Exploration

Authors: Yuxi Wei, Wei Huang, Qirui Chen, Lu Hou, Xiaojuan Qi

Organizations: The University of Hong Kong · Shanghai Jiao Tong University · Yinwang Intelligent Technology Co. Ltd

Abstract

Spatial understanding is essential for embodied agents navigating and interacting with the physical world. As observations arrive incrementally, agents must reason in a streaming manner. Yet streaming understanding must also be closed-loop: task-relevant evidence may require both recalling past observations and actively acquiring new views. Existing streaming approaches largely emphasize processing incoming observations, leaving task-driven active exploration underexplored. We introduce MemAct, a framework for streaming spatial understanding with memory and action. A shared vision--language model incrementally writes structured historical memory, retrieves records and past frames on demand, and acquires additional evidence through movement and turning. Retrieved evidence and action observations inform subsequent decisions, enabling the agent to continue exploring or answer within a unified reasoning loop. We develop the MemAct suite, a data generation pipeline and collection across real and simulated environments, with approximately 1M training examples for spatial reasoning, memory writing, retrieval, and exploration. Its MemAct-bench comprises 20,000 main QA pairs evaluating current-observation understanding, historical evidence use, and active exploration, plus 2,000 separate Timing questions. MemAct combines supervised fine-tuning for memory and interaction with counterfactual action optimization using cumulative evidence gains, action costs, and terminal answer quality to improve exploration and stopping decisions. Experiments show that MemAct outperforms existing spatial understanding and streaming baselines and approaches frontier multimodal models on MemAct-bench while retaining streaming inference efficiency.

Figures & tables

Figure 1: Offline, passive streaming, and closed-loop streaming spatial understanding. MemAct combines reasoning over incremental observations, historical memory retrieval, and active acquisition of missing evidence in a shared exploration-and-answering loop.

Figure 2: MemAct suite. Aligned temporal, camera, object, visibility, and scene metadata supports memory and spatial QA supervision from real and simulated observations. Dedicated rendering and geometric/evidence checks yield discrete multi-step camera-action data; all supervision forms causal streams.

Figure 3: MemAct training and inference. A shared VLM learns incremental memory writing, answering, historical retrieval, and discrete-action view acquisition through SFT. Counterfactual optimization improves exploration and stopping using cumulative evidence gains, action costs, and terminal answer quality.

Figure 4: Current/Memory/Action QA: available observations, historical evidence, and newly acquired views.

Model / baseline	Current		Memory		Action		Overall QA		Timing	Latency
	ACC	MRA	ACC	MRA	ACC	MRA	ACC	MRA	T / JA / JM	s/QA
Closed-source models
GPT-6 Astra	68.16	53.04	64.43	68.68	59.57	49.82	66.74	61.15	81.3 / 54.2 / 49.6	–
Gemini-3.8-Flash	59.26	46.29	56.41	59.55	52.20	43.42	58.17	53.14	67.1 / 38.9 / 36.0	–
Reference baselines
Uniform options	23.22	–	22.71	–	25.00	–	23.11	–	–	–

Table 1: MemAct-bench results. ACC and MRA assess multiple-choice and numerical questions, respectively; on-time rate (T), joint timing-and-answer accuracy (JA), and timing-gated MRA (JM) assess response timing. Latency (seconds) spans image/context readiness to complete answer output (exclusions: Section 5.1 ).

A. Memory components
Setting	ACC	MRA
No explicit memory	54.52	52.24
Full memory	68.11	67.72
GT memory	70.77	71.06
w/o Frames	64.92	63.72
w/o Count	68.10	66.62

Table 2: MemAct-9B Memory QA ablations: (1) content, (2) retention length, (3) write frequency.

A. Training strategy
Setting	ACC (%)	MRA (%)	Evidence gain	Effective gain	Mean
	( $\Delta$ pp)	( $\Delta$ pp)	(points)	rate (%)	steps
No action	47.45 (+0.00)	35.27 (+0.00)	0.00	0.00	0.00
SFT only	52.92 (+5.47)	41.87 (+6.60)	14.26	43.41	2.19
SFT + Bandit (full)	58.28 (+10.83)	48.74 (+13.47)	29.13	71.14	2.02
SFT + GRPO	57.92 (+10.47)	48.20 (+12.93)	28.34	69.78	2.08

Table 3: MemAct-9B Action QA ablations: (1) RL algorithm, (2) reward design, and (3) action-step budget.

Model	VSI- Bench	ReVSI	MMSI- Bench	View Spatial
Closed-source models
GPT-5.2	54.49	50.90	41.30	47.30
Gemini-3-Pro	55.10	60.90	45.20	50.40
Open-source models
Qwen3.5-9B	57.90	51.60	36.50	48.20
Spatial models

Table 4: MemAct versus published results on spatial understanding benchmarks.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Template	Example and construction
approached object category	Question: What is the category whose object had the largest decrease in camera-to-3D-center distance among objects visible both now and 5 seconds ago? Choose from the listed options. Answer: storage unit. Construction: Compare objects visible at both endpoints. Require a distance decrease of at least 0.20 m and a 0.10 m lead over the runner-up.
became visible category	Question: Select the option that identifies the category that became visible between 5 seconds ago and now. Answer: plant. Construction: Take the difference between endpoint category sets and require one newly visible category.
camera motion bucket	Question: What is the range of the camera’s straight-line endpoint displacement over the last 5 seconds? Choose from the listed options. Answer: almost stationary (at most 0.2 meters). Construction: Discretize straight-line camera endpoint displacement, excluding ambiguous interval boundaries.
camera path length m	Question: Estimate the total length of the camera’s trajectory over the last 5 seconds. Answer in meters. Answer: 0.0. Construction: Sum Euclidean translations between consecutive camera positions in the specified window.
camera rotation deg	Question: Provide your numerical estimate of the net difference between the camera’s starting and ending orientations over the last 5 seconds. Answer in degrees. Answer: 120.0. Construction: Compute net rotation between endpoint camera orientations, expressed in degrees.
camera to object distance bucket	Question: What is the camera-to-3D-center distance range for the closest visible sofa? Choose from the listed options. Answer: very near (at most 1.25 meters). Construction: Bind the nearest visible instance of the named category and discretize its center distance to the camera.

Appendix

Table 5: Current QA templates and examples.

Template	Example and construction
first appearance order	Question: Identify the first-observation order of the categories armchair, dining table, storage unit since this episode began by selecting an option. Answer: storage unit $\rightarrow$ dining table $\rightarrow$ armchair. Construction: Sort three category-first-observation times, separated by at least two seconds and preceding the current window.
first seen category then timestamp	Question: Determine the category seen earliest among the listed choices so far in this episode and select the matching choice. Answer: lighting. Construction: Identify the earliest category among the options, then query its original first-observation timestamp.
first seen object category	Question: Among the choices, which one gives the category seen earliest among the listed choices so far in this episode? Answer: lighting. Construction: Compare first-observation times of the listed categories; require a unique earliest option before the current window.
first seen timestamp s	Question: Estimate the first-seen timestamp of the lighting category, measured from the start of this episode. Answer in seconds. Answer: 0.0. Construction: Return the earliest observed timestamp for the category over the complete causal prefix.
historical coappearance category	Question: Choose the answer corresponding to the listed category also visible alongside the lighting at timestamp 3 seconds. Answer: cabinet. Construction: Use the explicitly referenced historical frame; the correct category is co-visible with the anchor and distractors are absent.
historical relative direction	Question: From the options below, indicate the image-plane position of the storage unit relative to the table in the observation at timestamp 44 seconds. Answer: left. Construction: Use uniquely bound objects in the historical frame and their same-row or same-column image-plane relationship.

Appendix

Table 6: Memory QA templates and examples.

Template	Example and construction
object size m	Question: Estimate the longest physical side length of the 3D bounding box of the drawer chest / corner cabinet visible in the initial view. Answer in meters. Answer: 1.185. Construction: Use source physical dimensions or transformed mesh bounding-box dimensions to obtain the longest physical side in meters; hold the bound object fixed across views.
object to object distance bucket	Question: Identify the 3D center-to-center distance range between the sofa in the middle left of the initial view and the table in the lower left of the initial view by selecting an option. Answer: near (1 to 1.75 meters). Construction: Map the bound pair’s center distance to an unambiguous interval; require evidence for both objects.
object to object distance m	Question: Report the 3D center-to-center distance between the sofa in the lower left of the initial view and the chair visible in the initial view as a number. Answer in meters. Answer: 3.041. Construction: Compute the Euclidean distance between the bound 3D centers; retain the same pair throughout the action path.

Appendix

Table 7: Action QA templates and examples.

Source	Layouts / recordings	Episodes / segments
3D-FRONT	4,823	14,534
ScanNet	1,412	1,511
ScanNet++	796	1,457
ARKitScenes	4,681	5,764
Real-world total	6,889	8,732

Appendix

Table 8: Episode composition. The simulation row covers Current and Memory QA; real-world rows count original recordings and unique retained segments.

Family	Training QA turns
Current	302,661
Memory	402,662
Action	294,677
Total	1,000,000

Appendix

Table 9: Training QA composition by family. Counts denote answer turns.

Source	Current	Memory	Action
3D-FRONT	135,671	218,983	294,677
ScanNet	31,304	30,431	0
ScanNet++	19,160	29,201	0
ARKitScenes	116,526	124,047	0

Appendix

Table 10: Training QA composition by source.

Actions	Training QA turns
1	216,418
2	46,858
3	31,401

Appendix

Table 11: Training action demonstration lengths after selection.

Group	QA turns	Fraction
Current	302,661	30.27%
Memory	402,662	40.27%
Action	294,677	29.47%
ARKitScenes	240,573	24.06%
ScanNet	61,735	6.17%
ScanNet++	48,361	4.84%

Appendix

Table 12: Training-set composition by QA family and source. Each fraction uses the 1,000,000 training QA turns as its denominator.


Template	Training QA turns
Current QA
approached object category	18,733
became visible category	18,733
camera motion bucket	18,733
camera path length m	18,733

Appendix

Table 13: Template counts in the training release. Linked templates count all retained answer turns.

Memory-Augmented Agents

Memory-augmented agents use stored experience and knowledge to act coherently across long tasks, changing environments, and repeated interactions.

See all 346 papers

See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning
Mar 25, 2026Yuxi Wei, Wei Huang, Qirui Chen +2Spatial Reasoning Benchmarks Visual Spatial Reasoning
The University of Hong Kong · Shanghai Jiao Tong University · Yinwang Intelligent Technology Co. Ltd.
StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance
Jun 12, 2026Guanming Liu, Yuqi Ren, Hansu Gu +5Agent Memory Long-Horizon Agent Evaluation
1Fudan University · 2Amazon
ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video
Oct 21, 2026Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun +6Agent Memory Memory Agent Benchmarks
LMU Munich · Technical University of Munich · Stanford University +1

Mar 25, 2026J/K move · Enter open · S save

See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning

Yuxi Wei, Wei Huang, Qirui Chen, Lu Hou+1

The University of Hong Kong · Shanghai Jiao Tong University · Yinwang Intelligent Technology Co. Ltd.

Spatial Reasoning Benchmarks Visual Spatial Reasoning Embodied QA Active Perception Memory-Augmented VLMs

Open paper AskPDF arXiv

Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two crucial deployment-critical requirements: long-horizon streaming inference and active perception when the current view is insufficient. To address this gap, we introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration, where queries are temporally grounded to specific timestamps and must be answered using only observations available up to that moment. S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions, and real-world streaming videos that capture practical sensing artifacts for rigorous generalization evaluation. Overall, it spans 10K+ scenes and 26K+ trajectories, with dedicated training (S3-Train) and evaluation (S3-Eval) splits. We further propose AMF-VLM, which supports streaming spatial reasoning under bounded computing via (i) memory folding, which compresses long-horizon observations into compact structured memory, and (ii) active exploration, which outputs explicit actions (e.g. move/rotate/scan) to acquire missing evidence before answering. Extensive experiments demonstrate that, compared to models using identical training data, our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively, while maintaining competitive transferability to standard spatial benchmarks.

Abstract

Memory-Augmented Agents

Explore similar work

See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning

StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance

ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video