SPARK: Self-Correcting Embodied Memory via Physical Re-Observation

Dmytro Kurdydyk; Aadi Palnitkar; Sebastian Scherer; Seungchan Kim

cs.ROOct 30, 2026

SPARK: Self-Correcting Embodied Memory via Physical Re-Observation

Authors: Dmytro Kurdydyk, Aadi Palnitkar, Sebastian Scherer, Seungchan Kim

Organizations: Davidson College · University of Maryland · Carnegie Mellon University

Abstract

Retrieval-augmented generation (RAG) systems have seen growing adoption in embodied settings as language-queryable memories of the environment. However, existing approaches often assume a reliable single-episode knowledge base (KB) or accumulate observations across episodes without correcting prior errors. Such errors can persist through incorrect semantic labels, inaccurate object positions, and inconsistent spatial relationships. To address this, we present SPARK, a corrective multi-episode framework for embodied RAG. After the first episode, a semantic critic flags regions of the KB, a targeted planner generates revisits, and a selective merge adds, removes, or replaces KB information based on the new observations. We evaluate SPARK on four outdoor AirSim maps under two degraded viewpoint conditions, using perception-built KBs, a visibility-constrained oracle used only for evaluation, and a fixed set of 158 embodied question answering (EQA) questions scored before and after correction. A single corrective episode improves downstream EQA on every map under both conditions while storing 13-41% fewer objects than append-only updating, with gains up to 2.7x on one map-condition pair. Extending correction through Episode 4 brings every map above the EQA performance of its undegraded reference. Ablations show that the critic's three signals each contribute to performance, that targeted revisits achieve a larger mean gain than untargeted ones under the same budget, and that selective merging achieves gains comparable to append-only updating with a smaller KB. Overall, SPARK demonstrates the value of correcting embodied memory through targeted re-observation, particularly as embodied systems scale and the risk of accumulated errors increases.

Figures & tables

Figure 1: Overview of the SPARK correction loop. Episode 1 flies the source waypoints under a degraded view and builds memory $H^{(1)}$ from its own RGB-D observations (1–2). The semantic critic reads $H^{(1)}$ and the Episode 1 images alone, flagging regions by plausibility, density, and graph coverage (3). The planner turns those into viewpoints offset from the source waypoints for Episode 2 (4), and the selective merge removes, replaces, and adds entries to produce $H^{(2)}$ (5), which a fixed EQA set scores against $H^{(1)}$ (6). The oracle (blue) is evaluation-only and never reaches the critic, planner, or merge.

Figure 2: Hierarchical memory and evaluation environments. Left: objects are grouped into clusters and clusters into areas, as in Embodied-RAG, and the first cluster level is the unit the critic flags. Middle: example cluster and object records. A corrective episode acts at the object tier, removing flagged entries that a revisit looked at and did not find, replacing entries it observes again, and adding ones it newly recovers, after which the affected clusters are rebuilt. Right: the four outdoor maps spanning an enclosed courtyard, an open industrial yard, a construction area, and a downtown plaza.

Table 3

Table 4

Figure 3: Fixed-set EQA across four episodes, with each map’s undegraded reference as a dashed line.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Object-labeling prompt. Each retained mask is sent in one call with three images: the padded crop, the masked crop, and the full frame with the bounding box drawn. A label equal to one of the six background terms ( ground to water ) rejects the mask, and the free-text specific type is the name $n$ that the memory stores. The person rule shown is the one in the first call, and a label that fails person verification is requested again with person excluded. Prompt text in this and the following prompt figures is copied from the code that produced the reported results: blue placeholders are filled at run time, […] marks text omitted for length, and the small headings are added for readability.

Figure 5: Critic prompts. The scene description is generated once per memory and inserted into the plausibility prompt. Density and plausibility are separate calls per waypoint, while graph coverage makes no model call and is shown as the rule it applies.

Figure 6: EQA judge prompt. The same prompt scores answers from $H^{(1)}$ and $H^{(2)}$ against the oracle ground truth, and each condition is scored three times.

Figure 7: One fixed-set question per family, with the oracle answer and the answers from $H^{(1)}$ and $H^{(2)}$ . Memory answers are shown verbatim, and […] marks a shortened evidence list. In the rare-presence example, both memories correctly report no pole at a waypoint where the map has poles elsewhere.

Long-Term Memory for Embodied Agents

See all 42 papers

HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering
Feb 17, 2026Ji Li, Bo Wang, Jing Xia +2Long-Term Memory for Embodied Agents Memory-Augmented Agents
The University of Hong Kong, Hong Kong SAR. · Beijing Institute of Technology, Beijing, China.
Improving Retrieval-Augmented Generation without Taxonomy-based Error Categorization
Apr 16, 2026Gongbo Zhang, Yifan Peng, Chunhua WengLLM Self-Correction Retrieval-Augmented Generation
Columbia University · Weill Cornell Medicine
eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents
Jun 2, 2026A. Haroon Rasheed, Maria KabtoulLLM Agent Memory Memory Agent Benchmarks
MIND · Automatika Robotics Inria, France · CHROMA

Feb 17, 2026J/K move · Enter open · S save

HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering

Ji Li, Bo Wang, Jing Xia, Mingyi Li+1

The University of Hong Kong, Hong Kong SAR. · Beijing Institute of Technology, Beijing, China.

Long-Term Memory for Embodied Agents Memory-Augmented Agents Embodied AI Embodied QA Multimodal Large Language Models

Open paper AskPDF arXiv

Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on textual summaries, which discard rich visual and spatial details and remain brittle in non-stationary environments. In this work, we propose a non-parametric memory framework that explicitly disentangles episodic and semantic memory for embodied exploration and question answering. Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning, enabling robust reuse of past observations without rigid geometric alignment. In parallel, we introduce a program-style rule extraction mechanism that converts experiences into structured, reusable semantic memory, facilitating cross-environment generalization. Extensive experiments demonstrate state-of-the-art performance on embodied question answering and exploration benchmarks, yielding a 7.3% gain in LLM-Match and an 11.4% gain in LLM MatchXSPL on A-EQA, as well as +7.7% success rate and +6.8% SPL on GOAT-Bench. Analyses reveal that our episodic memory primarily improves exploration efficiency, while semantic memory strengthens complex reasoning of embodied agents.

Abstract

Long-Term Memory for Embodied Agents

Explore similar work

HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering

Improving Retrieval-Augmented Generation without Taxonomy-based Error Categorization

eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents