Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration

Zhongyi Cai; Yi Du; Chen Wang; Yu Kong

cs.CVDec 2, 2025

Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration

Authors: Zhongyi Cai, Yi Du, Chen Wang, Yu Kong

Organizations: ACTION Lab, Michigan State University · SAIR Lab, University at Buffalo

Abstract

Embodied agents are expected to assist humans by actively exploring unknown environments and reasoning about spatial contexts. When deployed in real life, agents often face sequential tasks where each new task follows the completion of the previous one and may include infeasible objectives, such as searching for non-existent objects. However, most existing research focuses on isolated goals, overlooking the core challenge of sequential tasks: the ability to reuse spatial knowledge accumulated from previous explorations to guide subsequent reasoning and exploration. In this work, we investigate this underexplored yet practically significant embodied AI challenge. Specifically, we propose 3DSPMR, a 3D SPatial Memory Reasoning framework that utilizes Field-of-View (FoV) coverage as an explicit geometric prior. By integrating FoV-based constraints, 3DSPMR significantly enhances an agent's memory, reasoning, and exploration capabilities across sequential tasks. To facilitate research in this area, we further introduce SEER-Bench, a novel Sequential Embodied Exploration and Reasoning Benchmark that spans two foundational tasks: Embodied Question Answering (EQA) and Embodied Multi-modal Navigation (EMN). SEER-Bench uniquely incorporates both feasible and infeasible tasks to provide a rigorous and comprehensive evaluation of agent performance. Extensive experiments verify that 3DSPMR achieves substantial performance gains on both sequential EQA and EMN tasks.

Figures & tables

Figure 1: Illustration of 3DSPMR completing sequential EQA tasks. After navigating to the kitchen and answering the first question, the agent receives the second query. Using its spatial memory, it recognizes two previously passed but insufficiently explored bathrooms, revisits them, and answers the question after sufficient exploration. For the third question, the agent locates the study but cannot find a bookshelf; once the study is fully explored and no alternative candidate rooms remain, 3DSPMR correctly identifies the task as infeasible.

Figure 2: Top-down Projection of Agent FoV Coverage.

Figure 3: Overview of 3DSPMR. (a) Unified Spatial Memory: Raw observations are selectively stored to maintain an informative structured spatial representation. (b) Geo-Reasoning: A grounded inference module that ensures robust decision-making. (c) Geo-Sem Exploration: A strategic module that identifies optimal frontier targets.

Figure 4: Top-down Projection of FoV-based Gating. After a $90^{\circ}$ rotation, the agent obtains a new observation. Using the historical FoV coverage (yellow cells) and the coverage from the new observation (blue cells), we calculate the overlap (green cells) to decide whether to save this observation.

Figure 5: Top-down Projection of Room Coverage. The blue boundary defines the area of Room 1, while white cells mark covered regions $\mathrm{C_{history}}$ . The room coverage ratio is calculated as the fraction of the total room area $\mathrm{S_{room}}$ (cells within the boundary) that has been covered by the agent.

Figure 6: Unified Spatial Memory Update. The update process comprises four steps. (a) Step 1: compute the novelty score $\tau_{\!novel}$ of the new observation. If $\tau_{\!novel}>\tau_{0}$ , update memory in later steps. (b) Step 2: update geometric cues by merging the historical coverage map with the new image’s FoV coverage. (c) Step 3: update local visual cues by saving this image into the image bank. (d) Step 4: update global relational cues by updating the graph and establishing bidirectional links between the graph and the image.

Figure 7: Top-down Projection of Frontier Exploration FoV. The red lines delineate the FoV projected ahead of a frontier node. Red cells indicate unknown cells within the FoV, whose area is denoted as $U^{\mathrm{FoV}}_{t}$ in Eq. 3 .

Blind LLMs
	GPT-5				Gemini-3.6-Flash				LLaMA-3.3-70B
Method	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Qwen3	0.7	$\backslash$	26.0	$\backslash$	0.3	$\backslash$	26.2	$\backslash$	0.7	$\backslash$	28.9	$\backslash$
GPT-5	1.8	$\backslash$	26.6	$\backslash$	1.7	$\backslash$	27.5	$\backslash$	1.5	$\backslash$	28.3	$\backslash$
Open-Sourced Qwen2.5VL Exploration
Explore-Mem	0.3	0.2	22.1	8.5	0.2	0.1	22.1	8.6	0.2	0.1	23.2	8.8

Table 1: SEER-Bench EQA Results.

GOAT-Bench Baselines
Method	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Modular GOAT	0.0	0.0	18.0	10.6
SenseAct-NN	0.0	0.0	29.3	24.5
Open-Sourced Qwen2.5VL Exploration
Explore-Mem	1.7	0.4	38.7	32.0
GraphEQA	1.7	0.8	35.0	20.3

Table 2: SEER-Bench EMN Results.

Method	SR $\uparrow$	SPL $\uparrow$
Explore-Mem	30.6	24.5
3D-Mem	30.8	25.6
3DSPMR (Ours)	35.8	27.9

Table 3: Single-Task Experiments on HM-EQA.

Method	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
3DSPMR w/o FoV-Reason	8.3	5.2	56.3	49.6
3DSPMR w/o FoV-Explore	11.7	9.2	53.7	43.0
3DSPMR w/o Geo	5.0	3.3	47.3	42.7
3DSPMR	25.0	15.5	68.0	59.2

Table 4: We conducted a detailed ablation study on EMN track. (1) w/o FoV-Reason, (2) w/o FoV-Explore, (3) w/o Geo (with neither FoV-Reason nor FoV-Explore), and (4) the full 3DSPMR, which integrates both modules and achieves peak performance across all metrics. Bold indicates the best performance.

Blind LLMs
	GPT-5				Gemini-3.6-Flash				Llama-3.3-70B
Method	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Qwen3-14B	0.0	$\backslash$	28.5	$\backslash$	0.0	$\backslash$	27.9	$\backslash$	0.0	$\backslash$	30.6	$\backslash$
GPT-5	0.0	$\backslash$	23.1	$\backslash$	0.0	$\backslash$	24.0	$\backslash$	0.0	$\backslash$	25.2	$\backslash$
Open-Sourced Qwen2.5VL Exploration
Explore-Mem	0.0	0.0	16.9	5.0	0.0	0.0	15.0	5.0	0.0	0.0	17.1	5.2

Table 5: SEER-Bench EQA Results over Long Horizons.

Figure 8: Memory Efficiency Comparison. On both tracks of SEER-Bench, 3DSPMR saves significantly fewer images than 3D-Mem in the memory.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Data statistics and Unanswerable Question Examples of EQA in SEER-Bench.

Figure 10: Data statistics of the EMN track in SEER-Bench and infeasible goal examples.

Figure 11: Illustration of Geo-Reasoning in 3DSPMR for EQA tasks. (a) Phase 1: Reasoning Modules leverage the graph and images in spatial memory to generate an initial inference. (b) Phase 2: Geometric Examination mechanism utilizes FoV coverage to assess task completion. If the relevant spatial regions are fully explored, the system either confirms the inferred answer or flags task infeasibility; otherwise, it triggers Geo-Sem Exploration to determine the next optimal exploration direction.

Table 17

	GPT-5				Gemini-3.6-Flash				Llama-3.3-70B
MLLM Base	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Qwen2.5VL	5.0	4.6	43.3	34.2	5.8	5.3	44.2	35.6	8.3	7.9	46.3	35.9
GPT-5	32.5	23.0	58.8	45.7	28.3	18.8	56.7	44.3	30.8	21.0	58.8	44.1

Appendix

Table 8: EQA Experiments using different MLLMs.

Threshold	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	ANI $\downarrow$
0.1	18.3	11.4	65.3	57.1	52.3
0.25	18.3	12.8	62.0	54.3	38.4
0.5	25.0	15.5	68.0	59.2	25.2
0.8	21.7	12.5	66.0	56.8	17.0

Appendix

Table 9: Experiments on the SEER-Bench EMN track with different Geometric Novelty Score Thresholds (

\tau_{0}

Figure 12: Agent Performance Analysis with Different $w_{\mathrm{exp}}$ . Increasing $w_{\mathrm{exp}}$ encourages the agent to prioritize regions containing extensive unexplored areas. When $w_{\mathrm{exp}}$ $\in\{0.3,0.7\}$ , the agent decides to explore the region with larger unexplored areas (illustrated within the red bounding boxes). When $w_{\mathrm{exp}}=0.7$ , the agent is reluctant to fully explore this room and prefers to explore other underexplored rooms (illustrated within the purple bounding boxes).

Figure 13: Agent Performance Analysis with Different $w_{\mathrm{dist}}$ . Increasing $w_{\mathrm{dist}}$ encourages the agent to prioritize proximal unexplored regions. In the highlighted red boxes, it could be observed that as $w_{\mathrm{dist}}$ increases, the agent becomes less inclined to navigate toward far-off regions, preferring to exhaust the closest unexplored spaces first.

	GPT-5				Gemini-3.6-Flash				Llama-3.3-70B
Setting	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SSR $\uparrow$	SSPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
w/o Map Prior	7.5	7.1	38.8	29.8	5.0	4.6	35.4	26.7	7.5	7.2	43.8	33.6
w/ Map Prior	11.7	10.7	47.9	40.0	6.7	6.2	44.6	36.7	16.7	15.7	54.6	45.3

Appendix

Table 10: Effect of the map prior on SEER-Bench EQA Track.

Figure 14: Comparative Navigation Trajectories for Sequential EQA Tasks within One Episode. The figure presents the results for three distinct scenes, with each row corresponding to a different scene and its sequential tasks (Left). The middle column shows the 3D map of the scene. The right column compares the navigation trajectories (normalized to the same step numbers) generated by our Proposed Method (yellow line) and the 3D-Mem Baseline (red line).

Figure 15: Comparative Navigation Trajectories for Sequential EMN Tasks within One Episode. The figure presents the results for three distinct scenes, with each row corresponding to a different scene and its sequential tasks (Left). The middle column shows the 3D map of the scene. The right column compares the navigation trajectories (normalized to the same step numbers) generated by our Proposed Method (yellow line) and the 3D-Mem Baseline (red line).

Figure 16: Feasible vs. Infeasible Task: Incorrect Attribute. This figure illustrates 3DSPMR’s reasoning and exploration process on two EQA tasks: one feasible and one infeasible due to an incorrect object attribute. In the feasible case, 3DSPMR locates the referenced object and answers correctly. In the infeasible case, it detects the attribute mismatch, determines the task is infeasible, and halts further exploration.

Figure 17: Feasible vs. Infeasible Task: Object not Present. This figure illustrates 3DSPMR’s reasoning and exploration process on two EQA tasks: one feasible and one infeasible due to the absence of the mentioned object Christmas Tree . In the feasible case, 3DSPMR locates the Christmas Tree and answers correctly. In the infeasible case, it identifies the Christmas Tree ’s absence, determines that the task is infeasible, and terminates further exploration.

Figure 18: Feasible vs. Infeasible Task: Room not Present. This figure illustrates 3DSPMR’s reasoning and exploration process on two EQA tasks: one feasible and one infeasible due to the absence of the mentioned room bedroom . In the feasible case, 3DSPMR locates the referenced lamp within the actual bedroom and answers correctly. In the infeasible case, although it detects lamps, it finds that they are located in the living room rather than the targeted bedroom . After exploring all potentially relevant areas, 3DSPMR concludes that the bedroom is absent, identifies the task as infeasible, and halts further exploration.

Figure 21: Annotation prompt for generating unanswerable QA pairs where the referenced object is not present.

Figure 22: Annotation prompt for generating unanswerable QA pairs where the referenced room is not present.

Robotic Spatial Memory

See all 36 papers

GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning
Mar 19, 2026Yiren Lu, Yi Du, Disheng Liu +3Long-Term Memory for Embodied Agents Embodied AI
Case Western Reserve University · Spatial AI & Robotics (SAIR) Lab, University at Buffalo
RenderMem: Rendering as Spatial Memory Retrieval
Mar 15, 2026JooHyun Park, HyeongYeop KangLong-Term Memory for Embodied Agents Agent Memory
Korea University, South Korea
Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering
Jul 23, 2026Zikui Cai, Kaushal Janga, Tan Dat Dao +15Agent Memory Embodied QA
University of Maryland, College Park. · The University of Texas at Austin. · University of Illinois Urbana-Champaign.

Mar 19, 2026J/K move · Enter open · S save

GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning

Yiren Lu, Yi Du, Disheng Liu, Yunlai Zhou+2

Case Western Reserve University · Spatial AI & Robotics (SAIR) Lab, University at Buffalo

Long-Term Memory for Embodied Agents Embodied AI Agent Memory Embodied QA Vision-Language Navigation

Open paper AskPDF arXiv

Effective embodied exploration requires agents to accumulate and retain spatial knowledge over time. However, existing scene representations, such as discrete scene graphs or static view-based snapshots, lack post-hoc re-observability. If an initial observation misses a target, the resulting memory omission is often irrecoverable. To bridge this gap, we propose GSMem, a zero-shot embodied exploration and reasoning framework built upon 3D Gaussian Splatting (3DGS). By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial memory that endows the agent with Spatial Recollection: the ability to render photorealistic novel views from optimal, previously unoccupied viewpoints. To operationalize this, GSMem employs a retrieval mechanism that simultaneously leverages parallel object-level scene graphs and semantic-level language fields. This complementary design robustly localizes target regions, enabling the agent to ``hallucinate'' optimal views for high-fidelity Vision-Language Model (VLM) reasoning. Furthermore, we introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with geometric coverage. Extensive experiments on embodied question answering and lifelong navigation demonstrate the robustness and effectiveness of our framework

Abstract

Robotic Spatial Memory

Explore similar work

GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning

RenderMem: Rendering as Spatial Memory Retrieval

Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering