Spatial understanding is essential for embodied agents navigating and interacting with the physical world. As observations arrive incrementally, agents must reason in a streaming manner. Yet streaming understanding must also be closed-loop: task-relevant evidence may require both recalling past observations and actively acquiring new views. Existing streaming approaches largely emphasize processing incoming observations, leaving task-driven active exploration underexplored. We introduce MemAct, a framework for streaming spatial understanding with memory and action. A shared vision--language model incrementally writes structured historical memory, retrieves records and past frames on demand, and acquires additional evidence through movement and turning. Retrieved evidence and action observations inform subsequent decisions, enabling the agent to continue exploring or answer within a unified reasoning loop. We develop the MemAct suite, a data generation pipeline and collection across real and simulated environments, with approximately 1M training examples for spatial reasoning, memory writing, retrieval, and exploration. Its MemAct-bench comprises 20,000 main QA pairs evaluating current-observation understanding, historical evidence use, and active exploration, plus 2,000 separate Timing questions. MemAct combines supervised fine-tuning for memory and interaction with counterfactual action optimization using cumulative evidence gains, action costs, and terminal answer quality to improve exploration and stopping decisions. Experiments show that MemAct outperforms existing spatial understanding and streaming baselines and approaches frontier multimodal models on MemAct-bench while retaining streaming inference efficiency.
Figures & tables
Figure 1: Offline, passive streaming, and closed-loop streaming spatial understanding. MemAct combines reasoning over incremental observations, historical memory retrieval, and active acquisition of missing evidence in a shared exploration-and-answering loop.
Figure 2: MemAct suite. Aligned temporal, camera, object, visibility, and scene metadata supports memory and spatial QA supervision from real and simulated observations. Dedicated rendering and geometric/evidence checks yield discrete multi-step camera-action data; all supervision forms causal streams.
Figure 3: MemAct training and inference. A shared VLM learns incremental memory writing, answering, historical retrieval, and discrete-action view acquisition through SFT. Counterfactual optimization improves exploration and stopping using cumulative evidence gains, action costs, and terminal answer quality.
Figure 4: Current/Memory/Action QA: available observations, historical evidence, and newly acquired views.
Model / baseline
Current
Memory
Action
Overall QA
Timing
Latency
ACC
MRA
ACC
MRA
ACC
MRA
ACC
MRA
T / JA / JM
s/QA
Closed-source models
GPT-6 Astra
68.16
53.04
64.43
68.68
59.57
49.82
66.74
61.15
81.3 / 54.2 / 49.6
–
Gemini-3.8-Flash
59.26
46.29
56.41
59.55
52.20
43.42
58.17
53.14
67.1 / 38.9 / 36.0
–
Reference baselines
Uniform options
23.22
–
22.71
–
25.00
–
23.11
–
–
–
Table 1: MemAct-bench results. ACC and MRA assess multiple-choice and numerical questions, respectively; on-time rate (T), joint timing-and-answer accuracy (JA), and timing-gated MRA (JM) assess response timing. Latency (seconds) spans image/context readiness to complete answer output (exclusions: Section 5.1 ).
Table 4: MemAct versus published results on spatial understanding benchmarks.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Template
Example and construction
approached object category
Question: What is the category whose object had the largest decrease in camera-to-3D-center distance among objects visible both now and 5 seconds ago? Choose from the listed options. Answer: storage unit. Construction: Compare objects visible at both endpoints. Require a distance decrease of at least 0.20 m and a 0.10 m lead over the runner-up.
became visible category
Question: Select the option that identifies the category that became visible between 5 seconds ago and now. Answer: plant. Construction: Take the difference between endpoint category sets and require one newly visible category.
camera motion bucket
Question: What is the range of the camera’s straight-line endpoint displacement over the last 5 seconds? Choose from the listed options. Answer: almost stationary (at most 0.2 meters). Construction: Discretize straight-line camera endpoint displacement, excluding ambiguous interval boundaries.
camera path length m
Question: Estimate the total length of the camera’s trajectory over the last 5 seconds. Answer in meters. Answer: 0.0. Construction: Sum Euclidean translations between consecutive camera positions in the specified window.
camera rotation deg
Question: Provide your numerical estimate of the net difference between the camera’s starting and ending orientations over the last 5 seconds. Answer in degrees. Answer: 120.0. Construction: Compute net rotation between endpoint camera orientations, expressed in degrees.
camera to object distance bucket
Question: What is the camera-to-3D-center distance range for the closest visible sofa? Choose from the listed options. Answer: very near (at most 1.25 meters). Construction: Bind the nearest visible instance of the named category and discretize its center distance to the camera.
Appendix
Table 5: Current QA templates and examples.
Template
Example and construction
first appearance order
Question: Identify the first-observation order of the categories armchair, dining table, storage unit since this episode began by selecting an option. Answer: storage unit → dining table → armchair. Construction: Sort three category-first-observation times, separated by at least two seconds and preceding the current window.
first seen category then timestamp
Question: Determine the category seen earliest among the listed choices so far in this episode and select the matching choice. Answer: lighting. Construction: Identify the earliest category among the options, then query its original first-observation timestamp.
first seen object category
Question: Among the choices, which one gives the category seen earliest among the listed choices so far in this episode? Answer: lighting. Construction: Compare first-observation times of the listed categories; require a unique earliest option before the current window.
first seen timestamp s
Question: Estimate the first-seen timestamp of the lighting category, measured from the start of this episode. Answer in seconds. Answer: 0.0. Construction: Return the earliest observed timestamp for the category over the complete causal prefix.
historical coappearance category
Question: Choose the answer corresponding to the listed category also visible alongside the lighting at timestamp 3 seconds. Answer: cabinet. Construction: Use the explicitly referenced historical frame; the correct category is co-visible with the anchor and distractors are absent.
historical relative direction
Question: From the options below, indicate the image-plane position of the storage unit relative to the table in the observation at timestamp 44 seconds. Answer: left. Construction: Use uniquely bound objects in the historical frame and their same-row or same-column image-plane relationship.
Appendix
Table 6: Memory QA templates and examples.
Template
Example and construction
object size m
Question: Estimate the longest physical side length of the 3D bounding box of the drawer chest / corner cabinet visible in the initial view. Answer in meters. Answer: 1.185. Construction: Use source physical dimensions or transformed mesh bounding-box dimensions to obtain the longest physical side in meters; hold the bound object fixed across views.
object to object distance bucket
Question: Identify the 3D center-to-center distance range between the sofa in the middle left of the initial view and the table in the lower left of the initial view by selecting an option. Answer: near (1 to 1.75 meters). Construction: Map the bound pair’s center distance to an unambiguous interval; require evidence for both objects.
object to object distance m
Question: Report the 3D center-to-center distance between the sofa in the lower left of the initial view and the chair visible in the initial view as a number. Answer in meters. Answer: 3.041. Construction: Compute the Euclidean distance between the bound 3D centers; retain the same pair throughout the action path.
Appendix
Table 7: Action QA templates and examples.
Source
Layouts / recordings
Episodes / segments
3D-FRONT
4,823
14,534
ScanNet
1,412
1,511
ScanNet++
796
1,457
ARKitScenes
4,681
5,764
Real-world total
6,889
8,732
Appendix
Table 8: Episode composition. The simulation row covers Current and Memory QA; real-world rows count original recordings and unique retained segments.
Family
Training QA turns
Current
302,661
Memory
402,662
Action
294,677
Total
1,000,000
Appendix
Table 9: Training QA composition by family. Counts denote answer turns.
Source
Current
Memory
Action
3D-FRONT
135,671
218,983
294,677
ScanNet
31,304
30,431
0
ScanNet++
19,160
29,201
0
ARKitScenes
116,526
124,047
0
Appendix
Table 10: Training QA composition by source.
Actions
Training QA turns
1
216,418
2
46,858
3
31,401
Appendix
Table 11: Training action demonstration lengths after selection.
Group
QA turns
Fraction
Current
302,661
30.27%
Memory
402,662
40.27%
Action
294,677
29.47%
ARKitScenes
240,573
24.06%
ScanNet
61,735
6.17%
ScanNet++
48,361
4.84%
Appendix
Table 12: Training-set composition by QA family and source. Each fraction uses the 1,000,000 training QA turns as its denominator.
Template
Training QA turns
Current QA
approached object category
18,733
became visible category
18,733
camera motion bucket
18,733
camera path length m
18,733
Appendix
Table 13: Template counts in the training release. Linked templates count all retained answer turns.
Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two crucial deployment-critical requirements: long-horizon streaming inference and active perception when the current view is insufficient. To address this gap, we introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration, where queries are temporally grounded to specific timestamps and must be answered using only observations available up to that moment. S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions, and real-world streaming videos that capture practical sensing artifacts for rigorous generalization evaluation. Overall, it spans 10K+ scenes and 26K+ trajectories, with dedicated training (S3-Train) and evaluation (S3-Eval) splits. We further propose AMF-VLM, which supports streaming spatial reasoning under bounded computing via (i) memory folding, which compresses long-horizon observations into compact structured memory, and (ii) active exploration, which outputs explicit actions (e.g. move/rotate/scan) to acquire missing evidence before answering. Extensive experiments demonstrate that, compared to models using identical training data, our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively, while maintaining competitive transferability to standard spatial benchmarks.
Yuxi Wei, Wei Huang, Qirui Chen +2
The University of Hong Kong · Shanghai Jiao Tong University · Yinwang Intelligent Technology Co. Ltd.
A central role of personal-agent memory is to turn stored information and prior interactions into future-oriented assistance. In daily use, useful cues come from what the agent observes and how the user interacts with the agent, and the agent must carry them forward from the current request to similar future tasks. Existing memory benchmarks usually test dialogue recall or task improvement in isolation, leaving the trajectory from streaming observations to later assistance largely untested. We introduce StreamMemBench, a streaming benchmark that constructs a two-step task sequence around each evidence anchor from EgoLife egocentric streams. The initial task tests evidence use, while the follow-up task tests whether feedback and interaction experience are reused. Four metrics diagnose evidence recall, initial evidence use, feedback incorporation, and follow-up reuse. Experiments with eight memory systems across two backbones show that current systems often fail to use observed evidence or turn feedback into reliable follow-up behavior, even when evidence is stored or feedback is incorporated locally. StreamMemBench is publicly available at https://github.com/landian60/StreamMemBench.
Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should instead proactively acquire and maintain useful memory as the user's surroundings evolve. Doing so on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user stays focused on their activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The benchmark defines its protocol at two levels of proactiveness: the Notification Level asks for a single timely notification that satisfies the user's intent, while the deeper Intervention Level asks the agent to maintain memory on the user's behalf through an active start-to-stop tool lifecycle. Parallel ProactEgoMem-Daily and ProactEgoMem-Sci tracks contain 300/100 clips covering daily and scientific use cases, paired with 1000/500 questions in ProactEgoMem-QA, which measures the quality of the proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory both timely and accurately. ProactEgoMem provides a rigorous evaluation foundation for proactive multimodal assistance and for video agents that co-exist with their users.
Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun +6
LMU Munich · Technical University of Munich · Stanford University +1