Reflect After Failure: Brain-Inspired Self-Evolving Agents for Embodied Spatial Intelligence

Xin Li; Xiaohuan Gu; Yifan Wang; Shaobin Ling; Tianli Shi; Jingao Xu; Zhimeng Yin; Dapeng Wu

cs.AIOct 14, 2026

Reflect After Failure: Brain-Inspired Self-Evolving Agents for Embodied Spatial Intelligence

Authors: Xin Li, Xiaohuan Gu, Yifan Wang, Shaobin Ling, Tianli Shi, Jingao Xu, Zhimeng Yin, Dapeng Wu,

Organizations: City University of Hong Kong · East China Normal University · University of Electronic Science and Technology of China · The Chinese University of Hong Kong, Shenzhen ·

Abstract

Vision-Language Models (VLMs) have made substantial progress in spatial reasoning, and self-evolving agents further extend these capabilities to embodied environments by learning from interactions. However, current self-evolution remains particularly challenging in embodied spatial tasks, complex and multi-step decisions from partial visual evidence making it difficult to determine why an execution fails. The failure may stem from misperception, insufficient exploration, or incorrect planning, yet execution feedback alone rarely reveals which component is responsible. As a result, simply accumulating interaction experience can reinforce ineffective strategies. To address this, we propose HiMeco, a brain-inspired self-evolving agent for embodied spatial intelligence, inspired by metacognitive monitoring and hippocampal memory consolidation. Through Hippocampal-Metacognitive Reflection, HiMeco diagnoses failures in perception-exploration-planning (PEP), then compares execution attempts to derive reusable strategies and guidance for avoiding recurring errors. It also abstracts interaction programs into parameterized skills, using evidence contracts and execution-based validation to determine their retention and reuse. These strategies and skills guide subsequent interactions, with memory updated only between rounds and model parameters kept frozen. HiMeco achieves state-of-the-art performance on ESI-Bench, and it also improves accuracy on EmbSpatial-Bench by 3.94 percentage points without evolution.

Figures & tables

Figure 1: Motivation of our method. (a) Previous methods risk reinforcing errors through inaccurate failure attribution. (b) Inspired by metacognitive monitoring and hippocampal consolidation, our method diagnoses failures to guide future interactions.

Figure 2: Overview of HiMeco . (A) The frozen planner interacts with the environment and collects evidence. (B) The PEP diagnostic module identifies perception, exploration, and planning failures. (C) Strategies $\mathcal{E}$ and skills $\mathcal{K}$ are updated between rounds and used in later interactions.

Our evaluation
Backbone / Method		Metric Comparison	Cognitive Mapping	Perceptual Grounding	Action Sequencing	Spatial Relations	Specular Reflection	Enumerative Perception	Temporal Understanding	Physical Dynamics	Avg.
Selected subtask		Spatial Distance	Connectivity	Partial Occlusion	Action Order Inference	Physical Contact	Correspondence	Counting w/ Occlusion	Unobserved Change	Inclined Plane	-
$n$		20	27	20	20	25	20	20	29	27	-
GPT-5.6	Passive	45.0	59.3	45.0	35.0	68.0	65.0	10.0	34.5	37.0	44.31
	Active	60.0	70.4	50.0	40.0	64.0	75.0	10.0	75.9	66.7	56.89
	HiMeco	85.0	74.1	65.0	55.0	88.0	80.0	15.0	82.8	77.8	69.19

Table 1: Accuracy (%) on ESI-Bench.

Figure 3: Qualitative examples of embodied spatial reasoning. Incorrect and correct predictions are shown with their observations and explanations.

Figure 4: Failure diagnosis and execution efficiency on ESI-Bench. Round-0 versus the final evolution round: (a) Primary failure counts normalized by all evaluation episodes. (b) Changes in mean diagnostic scores. (c) Relative changes in mean wall-clock time per episode. Negative values in (b) and (c) indicate reductions.

$\mathcal{E}$	$\mathcal{K}$	$\mathcal{U}$	Evol.	Spatial Distance	Action Order Inference	Physical Contact	Inclined Plane
				60	40	64	66.7
✓		✓	✓	65	55	80	74.1
	✓	✓	✓	70	45	76	70.4
✓	✓		✓	65	45	80	74.1
✓	✓	✓	✓	85	55	88	77.8

Table 2: Component ablations. The diagnostic ablation replaces structured PEP-guided reflection with generic reflection while retaining general execution reflection and program-change suggestions.

Relation	Gemini-3-Flash	HiMeco w/o Evolution
Left	95.33	92.33
Right	84.67	82.67
Above	75.33	88.00
Under	73.67	85.00
Far	57.33	65.00
Close	67.33	64.33

Table 3: Accuracy (%) on EmbSpatial-Bench.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Additional qualitative examples of embodied spatial reasoning. Incorrect and correct predictions are presented with their observations and explanations.

Round	Candidates per episode	Update afterward	Purpose
0	1	Yes	Initial active baseline and first consolidation
1	3	Yes	Candidate comparison and skill trials
2	3	Yes	Candidate comparison and skill validation
3	1	No	Final execution with frozen memory

Appendix

Table 4: Execution schedule of the full repeated-instance method. Every candidate in a row uses the same memory fixed before the round.

Task	$n$	Episode IDs
Spatial Distance	20	0781, 0787, 0788, 0822, 0835, 0838, 0841, 0844, 0851, 0856, 0857, 0866, 0867, 0869, 0871, 0874, 0877, 0894, 0907, 0910
Connectivity	27	0078, 0081, 0086, 0087, 0089, 0090, 0092, 0096, 0098, 0100, 0101, 0102, 0103, 0107, 0108, 0109, 0115, 0120, 0122, 0123, 0125, 0126, 0128, 0131, 0132, 0133, 0134
Partial Occlusion	20	1146, 1155, 1160, 1164, 1171, 1175, 1178, 1186, 1194, 1196, 1197, 1204, 1205, 1206, 1207, 1212, 1215, 1218, 1221, 1233
Action Order Inference	20	0004, 0005, 0008, 0009, 0012, 0013, 0015, 0029, 0033, 0034, 0035, 0037, 0041, 0044, 0045, 0048, 0056, 0061, 0062, 0068
Physical Contact	25	2804, 2810, 2812, 2819, 2824, 2829, 2833, 2835, 2837, 2841, 2850, 2854, 2855, 2862, 2868, 2873, 2877, 2881, 2890, 2895, 2896, 2898, 2899, 2916, 2921
Correspondence	20	2923, 2937, 2941, 2948, 2952, 2955, 2958, 2959, 2964, 2967, 2968, 2974, 2978, 2981, 2982, 2985, 2988, 2989, 2990, 2998

Appendix

Table 5: Reported ESI-Bench episode subset. IDs are listed within each subtask.

Task	Evidence checked by the visual module
Spatial Distance	A canonical candidate answer, visibility of the reference and both candidates, at least two cited views with per-view geometry, geometric viewpoint diversity, and no recorded contradictions.
Connectivity	Identified regions and supporting cues; either evidence of a direct shared passage or explicit negative boundary/separation evidence. An intermediate region must not be mistaken for direct adjacency.
Partial Occlusion	At least two cited views, tracking of the same target region, distinction between target and foreground blocker, an observed reduction in occlusion, identity cues from a revealing view, and geometric diversity.
Action Order Inference	An ordered direction pair, a cited reference view, explicit selection of the correctly sized A/C candidates, and direction evidence consistent with the returned pair.
Physical Contact	Evidence bound to the two named objects, a visible boundary or gap across views, rejection of ambiguous occlusion and misleading shadow/floor cues, and geometric diversity.
Correspondence	A canonical answer among the allowed positions and structured evidence associating the reflected target with a real candidate while rejecting alternatives and contradictions.

Appendix

Table 6: Task-specific visual evidence checks. Semantic fields are reported by the visual model; checks assess their format and consistency.

Condition	$u_{\mathrm{per}}$
A successful visual call reports an invalid output contract	1.00
Visual evidence is explicitly invalid and a symbolic count was returned	0.95
The last symbolic count disagrees with an available simulator-spawned target count	1.00
$u_{\mathrm{exp}}\leq 0.35$ , a visual call executed, and the answer is not unknown	0.95
$u_{\mathrm{exp}}\leq 0.35$ , a visual call executed, and the answer is unknown	0.80
A visual call executed, but the preceding rules do not apply	0.55

Appendix

Table 7: Ordered rules for perception diagnosis after an incorrect answer.

Setting	Value
Round-0 planner temperature / top- $p$	0 / 1
Intermediate planner temperature / top- $p$	0.7 / 0.9
Final-generation temperature / top- $p$	0.7 / 0.9 in the reference runner
Planner output limit	1,800 tokens
Program repair attempts after initial generation	1
Program statement limit / branch depth	48 / 4

Appendix

Table 8: Common reference-configuration settings. Task-dependent exceptions are given below.

Embodied Spatial Reasoning

See all 20 papers

Spatial Memory Agent: Experience-Grounded Procedural Memory for Spatial Intelligence
Aug 13, 2026Haokai Zhang, Yuhang Ding, Yunshu Zhou +5Memory-Augmented VLMs Visual Spatial Reasoning
Zhejiang University · Shanghai Jiao Tong University · Shanghai Innovation Institute
EvoSpatial: Experience-Driven Skill and Tool Evolution for Spatial Reasoning Agents
Oct 12, 2026Bin Yu, Shijie Lian, Zhaolong Shen +10Tool-Using Agents Spatial Reasoning
Harbin Institute of Technology · Zhongguancun Academy · DeepCybo +5
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
May 18, 2026Yining Hong, Jiageng Liu, Han Yin +5Active Perception Spatial Reasoning Benchmarks
1Stanford University · 2UCLA · 3Northwestern University

Aug 13, 2026J/K move · Enter open · S save

Spatial Memory Agent: Experience-Grounded Procedural Memory for Spatial Intelligence

Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du+4

Zhejiang University · Shanghai Jiao Tong University · Shanghai Innovation Institute

Memory-Augmented VLMs Visual Spatial Reasoning LLM Agent Memory Self-Evolving Agents

Open paper AskPDF arXiv

Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLMs, existing work has mainly followed two lines. One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through parameter-update-free self-evolution, without depending on external expert spatial tools at inference time? We present Spatial Memory Agent (SMA), an experience-grounded runtime memory framework that converts verified spatial experience into reusable transferable lessons. Specifically, SMA first queries the frozen VLM in a verifiable spatial environment, obtains a predicted answer and reward, and uses verifier-guided reflection to distill compact transferable lessons stored in memory cards. SMA further assigns each memory card a Transfer Reliability Score (TRS), which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During read-only deployment, SMA retrieves memory cards through semantic filtering and combined similarity--TRS ranking, allowing the retrieved memory to guide frozen model inference. Experiments across five representative spatial benchmarks show that SMA achieves the best macro-average accuracy for all four base VLMs and the best accuracy in most individual evaluations, establishing a practical parameter-update-free path for spatial self-evolution through reusable experience.

Abstract

Embodied Spatial Reasoning

Explore similar work

Spatial Memory Agent: Experience-Grounded Procedural Memory for Spatial Intelligence

EvoSpatial: Experience-Driven Skill and Tool Evolution for Spatial Reasoning Agents

ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop