Effective embodied exploration requires agents to accumulate and retain spatial knowledge over time. However, existing scene representations, such as discrete scene graphs or static view-based snapshots, lack post-hoc re-observability. If an initial observation misses a target, the resulting memory omission is often irrecoverable. To bridge this gap, we propose GSMem, a zero-shot embodied exploration and reasoning framework built upon 3D Gaussian Splatting (3DGS). By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial memory that endows the agent with Spatial Recollection: the ability to render photorealistic novel views from optimal, previously unoccupied viewpoints. To operationalize this, GSMem employs a retrieval mechanism that simultaneously leverages parallel object-level scene graphs and semantic-level language fields. This complementary design robustly localizes target regions, enabling the agent to ``hallucinate'' optimal views for high-fidelity Vision-Language Model (VLM) reasoning. Furthermore, we introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with geometric coverage. Extensive experiments on embodied question answering and lifelong navigation demonstrate the robustness and effectiveness of our framework
Figures & tables
Figure 1: With GS-Mem, previously explored regions can be retrieved and re-observed directly from the 3DGS memory without physically navigating to them.
Figure 2: Demonstration of Multi-level Retrieval-Rendering. Given a task-related target, the agent retrieves ROIs based on object-level and semantic-level cues. Subsequent viewpoint selection and rendering enable the agent to re-observe these regions for further reasoning.
Figure 3: Demonstration of our Hybrid Exploration Strategy. When frontier observations do not contain sufficient task-related cues for the VLM to make a decision, we incorporate an information gain-based score to select the most informative frontier for further exploration.
Method
LLM-Match ↑
LLM-Match SPL ↑
Blind LLMs
GPT-4o
35.9
-
LLaMA-3.1-8B-Instruct
32.2
-
LLM with Frame Captions
GPT-4o w/ LLaVA-1.5-13B
33.11
-
LLaMA-3.1 w/ LLaVA-1.5-13B
30.84
-
Table 1: Experiments on A-EQA. “CG” denotes ConceptGraphs. All the VLM exploration methods use GPT-4o as VLM backbone.
Method
Success Rate ↑
SPL ↑
GOAT-Bench Baselines
Modular GOAT [ 2 ]
24.9
17.2
Modular CLIP on Wheels [ 5 ]
16.1
10.4
SenseAct-NN Monolithic [ 16 ]
12.3
6.8
SenseAct-NN Skill Chain [ 16 ]
29.5
11.3
VLM Exploration
Table 2: Experiments on GOAT-Bench “Val Unseen" split.
Figure 4: Case analysis. We analyze several cases where scene-graph and view-based representations fail, and demonstrate the advantages of 3DGS-based memory. The images shown correspond to the views selected by the VLM for answering the questions. The examples (a-c) illustrate failures of the scene-graph detector, while the last example (d) highlights how optimal viewpoint rendering benefits the VLM’s reasoning.
Table 7
Figure 5: Runtime analysis for real-world multi-process setup.
Embodied reasoning is inherently viewpoint-dependent: what is visible, occluded, or reachable depends critically on where the agent stands. However, existing spatial memory systems for embodied agents typically store either multi-view observations or object-centric abstractions, making it difficult to perform reasoning with explicit geometric grounding. We introduce RenderMem, a spatial memory framework that treats rendering as the interface between 3D world representations and spatial reasoning. Instead of storing fixed observations, RenderMem maintains a 3D scene representation and generates query-conditioned visual evidence by rendering the scene from viewpoints implied by the query. This enables embodied agents to reason directly about line-of-sight, visibility, and occlusion from arbitrary perspectives. RenderMem is fully compatible with existing vision-language models and requires no modification to standard architectures. Experiments in the AI2-THOR environment show consistent improvements on viewpoint-dependent visibility and occlusion queries over prior memory baselines.
Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on textual summaries, which discard rich visual and spatial details and remain brittle in non-stationary environments. In this work, we propose a non-parametric memory framework that explicitly disentangles episodic and semantic memory for embodied exploration and question answering. Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning, enabling robust reuse of past observations without rigid geometric alignment. In parallel, we introduce a program-style rule extraction mechanism that converts experiences into structured, reusable semantic memory, facilitating cross-environment generalization. Extensive experiments demonstrate state-of-the-art performance on embodied question answering and exploration benchmarks, yielding a 7.3% gain in LLM-Match and an 11.4% gain in LLM MatchXSPL on A-EQA, as well as +7.7% success rate and +6.8% SPL on GOAT-Bench. Analyses reveal that our episodic memory primarily improves exploration efficiency, while semantic memory strengthens complex reasoning of embodied agents.
Ji Li, Bo Wang, Jing Xia +2
The University of Hong Kong, Hong Kong SAR. · Beijing Institute of Technology, Beijing, China.
Constructing structured spatial memory is essential for enabling long-horizon reasoning in complex embodied navigation tasks. Current memory construction predominantly relies on a decoupled, two-stage paradigm: agents first aggregate environmental data through exploration, followed by the offline reconstruction of spatial memory. However, this post-hoc and geometry-centric approach precludes agents from leveraging high-level semantic intelligence, often causing them to overlook navigationally critical landmarks (e.g., doorways and staircases) that serve as fundamental semantic anchors in human cognitive maps. To bridge this gap, we propose ABot-Explorer, a novel active exploration framework that unifies memory construction and exploration into an online, RGB-only process. At its core, ABot-Explorer leverages Large Vision-Language Models (VLMs) to distill Semantic Navigational Affordances (SNA), which act as cognitive-aligned anchors to guide the agent's movement. By dynamically integrating these SNAs into a hierarchical SG-Memo, ABot-Explorer mirrors human-like exploratory logic by prioritizing structural transit nodes to facilitate efficient coverage. To support this framework, we contribute a large-scale dataset extending InteriorGS with SNA and SG-Memo annotations. Experimental results demonstrate that ABot-Explorer significantly outperforms current state-of-the-art methods in both exploration efficiency and environment coverage, while the resulting SG-Memo is shown to effectively support diverse downstream tasks.