cs.ROMar 12, 2026
SaveBrainMem: Brain-Inspired Evolving Memory for Embodied Agent Task Planning
Organizations: Nanyang Technological University, Singapore · Tianjin University, Tianjin, China
Abstract
Embodied task planning requires agents to execute long-horizon, goal-directed actions in complex 3D environments, where success depends on both immediate perception and accumulated experience across tasks. However, most existing LLM-based planners are stateless and reactive, operating without persistent memory and therefore repeating errors and struggling with spatial or temporal dependencies. We propose BrainMem(Brain-Inspired Evolving Memory), a training-free hierarchical memory system that equips embodied agents with working, episodic, and semantic memory inspired by human cognition. BrainMem continuously transforms interaction histories into structured knowledge graphs and distilled symbolic guidelines, enabling planners to retrieve, reason over, and adapt behaviors from past experience without any model fine-tuning or additional training. This plug-and-play design integrates seamlessly with arbitrary multi-modal LLMs and greatly reduces reliance on task-specific prompt engineering. Extensive experiments on four representative benchmarks, including EB-ALFRED, EB-Navigation, EB-Manipulation, and EB-Habitat, demonstrate that BrainMem significantly enhances task success rates across diverse models and difficulty subsets, with the largest gains observed on long-horizon and spatially complex tasks. These results highlight evolving memory as a promising and scalable mechanism for generalizable embodied intelligence.
Figures & tables
Figure 1: Typical failure modes of stateless planners. (Top) The agent repeatedly toggles the faucet without completing the washing step. (Bottom) The agent fails to compose “pick up” and “place” actions due to missing object-combination experience.
Figure 2: Overall framework of the proposed BrainMem system. At execution time (left), the Experience Agent accumulates memory across three levels: Working Memory (short-term), Episodic Memory (task-specific), and Semantic Memory (task-agnostic, long-term insights). At inference (right), the Planner Agent retrieves relevant memory to guide goal-directed action generation.
| Model | EB-ALFRED | EB-Habitat | ||||||||||||
| Avg | Base | Comm. | Comp. | Visual | Spatial | Long | Avg | Base | Comm. | Comp. | Visual | Spatial | Long | |
| Proprietary MLLMs | ||||||||||||||
| GPT-4o | 56.3 | 64.0 | 54.0 | 68.0 | 46.0 | 52.0 | 54.0 | 59.0 | 86.0 | 44.0 | 56.0 | 68.0 | 36.0 | 64.0 |
| GPT-4o-mini | 24.0 | 34.0 | 28.0 | 36.0 | 24.0 | 22.0 | 0.0 | 32.7 | 74.0 | 22.0 | 32.0 | 22.0 | 32.0 | 14.0 |
| Claude-3.7-Sonnet | 67.7 | 68.0 | 68.0 | 70.0 | 68.0 | 62.0 | 70.0 | 58.7 | 90.0 | 58.0 | 58.0 | 62.0 | 38.0 | 46.0 |
| Claude-3.5-Sonnet | 64.0 | 72.0 | 66.0 | 76.0 | 60.0 | 58.0 | 52.0 | 68.0 | 96.0 | 68.0 | 78.0 | 70.0 | 38.0 | 58.0 |
Table 1: Task success rates on 6 subsets of EB-ALFRED and EB-Habitat. The bold value indicates the best value per column and the underline value represents the best performance of previous methods.
| Model | EB-Navigation | EB-Manipulation | ||||||||||
| Avg | Base | Comm. | Comp. | Visual | Long | Avg | Base | Comm. | Comp. | Visual | Spatial | |
| Proprietary MLLMs | ||||||||||||
| GPT-4o | 57.7 | 55.0 | 60.0 | 58.3 | 60.0 | 55.0 | 28.9 | 39.6 | 29.2 | 29.2 | 19.4 | 25.0 |
| GPT-4o-mini | 32.8 | 31.7 | 33.3 | 35.0 | 28.3 | 33.3 | 4.8 | 4.2 | 6.3 | 2.1 | 0.0 | 10.4 |
| Claude-3.7-Sonnet | 45.0 | 50.0 | 61.7 | 50.0 | 36.7 | 26.7 | 28.5 | 31.3 | 20.8 | 43.8 | 25.0 | 20.8 |
| Claude-3.5-Sonnet | 44.7 | 66.7 | 51.7 | 41.7 | 36.7 | 26.7 | 25.4 | 37.5 | 16.7 | 29.2 | 19.4 | 22.9 |
Table 2: Task success rates on 5 subsets of EB-Navigation and EB-Manipulation. The bold value indicates the best value per column and the underline value represents the best performance of previous methods.
| Method | EB-ALFRED | EB-Habitat | ||||||||
| Average | Base | Long | Base | Long | ||||||
| SR | GC | SR | GC | SR | GC | SR | GC | SR | GC | |
| Voyager (NeurIPS 2023) | 46.5 | 66.4 | 56.0 | 73.2 | 32.0 | 54.2 | 76.0 | 87.0 | 22.0 | 51.0 |
| Reflexion (NeurIPS 2023) | 38.3 | 51.1 | 48.0 | 54.0 | 10.0 | 33.0 | 80.0 | 84.2 | 15.0 | 33.0 |
| Cradle (ICLR 2024) | 44.5 | 57.0 | 54.0 | 67.9 | 32.0 | 41.0 | 62.0 | 67.0 | 30.0 | 52.1 |
| RoboOS (2025.6) | 25.5 | 33.0 | 32.0 | 38.4 | 12.0 | 17.6 | 38.0 | 47.8 | 20.0 | 28.2 |
Table 3: Comparison of Success Rates (SR) and Goal Condition Success Rates (GC) across difficulty levels (Base/Long) on EB-ALFRED and EB-Habitat. All methods are evaluated with Qwen2.5-VL-72B-Ins. Values are reported in percentages (%).
| Working Memory | Episodic memory | Semantic memory | Feedback | SR (%) |
| ✗ | ✗ | ✗ | ✗ | 64.0 |
| ✓ | ✗ | ✗ | ✗ | 68.0 |
| ✓ | ✓ | ✗ | ✗ | 70.0 |
| ✓ | ✓ | ✓ | ✗ | 76.0 |
| ✓ | ✓ | ✓ | ✓ | 82.0 |
Table 4: Ablation of memory modules on EB-ALFRED. Success rates (%) are averaged over six subsets.
| Method | Base subset | Long subset |
| Baseline (no memory) | 2.08 | 4.20 |
| +BrainMem | 2.92 | 5.89 |
Table 10: Total LLM token usage (input + output) on EB-ALFRED.
Figure 3: Visualization of task execution without and with memory. The example task is “Rinse off lettuce and put it down on the countertop.” Without memory (top), the agent repeatedly opens and closes the faucet without placing the lettuce inside, failing to complete the task. With memory (bottom), the agent recalls that the object must be placed in the sink before turning on the faucet, executing the correct sequence.
Figure 4: Error distribution on EB-ALFRED using GPT-4o. The left chart shows the breakdown of 131 failure cases without memory, dominated by reasoning and planning errors. The right chart shows 75 failure cases with BrainMem memory, where planning errors are notably reduced and reflection errors disappear, demonstrating the stabilizing effect of hierarchical memory.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | EB-ALFRED | EB-Habitat | ||||||||||||
| Avg | Base | Common | Complex | Visual | Spatial | Long | Avg | Base | Common | Complex | Visual | Spatial | Long | |
| Proprietary MLLMs | ||||||||||||||
| GPT-4o | 65.1 | 74.0 | 60.3 | 74.0 | 58.3 | 61.3 | 62.5 | 70.7 | 90.7 | 56.0 | 68.0 | 75.2 | 62.1 | 72.2 |
| GPT-4o-mini | 34.3 | 47.8 | 35.3 | 43.5 | 33.3 | 29.0 | 17.0 | 44.0 | 77.5 | 32.5 | 42.0 | 33.1 | 57.8 | 21.3 |
| Claude-3.5-Sonnet | 65.3 | 72.0 | 66.0 | 76.7 | 63.0 | 59.7 | 54.5 | 70.8 | 97.5 | 68.5 | 79.5 | 72.0 | 43.8 | 63.3 |
| Gemini-1.5-Pro | 67.4 | 74.3 | 66.7 | 76.5 | 62.8 | 59.0 | 65.0 | 61.0 | 92.5 | 53.5 | 49.5 | 59.4 | 50.0 | 61.2 |
Table 11: Subgoal success rates on 6 subsets of EB-ALFRED and EB-Habitat, with the best proprietary model in bold and open-source model underlines per column.
| Model | EB-ALFRED | EB-Habitat | ||
| Plan | Env | Plan | Env | |
| GPT-4o | 4.4 | 16.3 | 5.5 | 13.1 |
| GPT-4o-mini | 7.7 | 20.6 | 7.4 | 18.8 |
| Claude-3.5-Sonnet | 4.0 | 12.1 | 4.2 | 10.9 |
| Gemini-1.5-Pro | 3.9 | 15.7 | 5.4 | 12.6 |
| Gemini-2.0-flash | 4.4 | 16.3 | 6.8 | 14.8 |
Table 12: Average planner and environment steps for EB-ALFRED and EB-Habitat.
| Model | EB-Navigation | EB-Manipulation | ||
| Plan | Env | Plan | Env | |
| GPT-4o | 6.2 | 15.5 | 2.6 | 12.9 |
| GPT-4o-mini | 7.6 | 17.5 | 3.4 | 14.7 |
| Claude-3.5-Sonnet | 6.2 | 15.6 | 2.7 | 13.3 |
| Gemini-1.5-Pro | 8.8 | 16.5 | 2.7 | 13.4 |
| Gemini-2.0-flash | 9.2 | 16.0 | 2.8 | 14.0 |
Table 13: Average planner and environment steps for EB-Navigation and EB-Manipulation.
| Shuffle Order | 1 | 2 | 3 | 4 | 5 |
| SR (%) | 80 | 82 | 82 | 80 | 84 |
| Mean Std | |||||
Table 14: Robustness to 5 random episode orders on Base Set.
Figure 5: Habitat task visualization (without vs. with memory). Task: “Find a wrench and move it to the left counter.” Without memory, the agent repeatedly re-searches previously visited regions, failing to maintain a record of explored areas. This leads to inefficient looping behavior and missing the wrench despite previously observing it. With memory, the agent recalls explored zones and retrieved spatial cues, enabling efficient search and successful delivery to the counter.
Figure 6: Habitat task visualization (without vs. with memory). Task: “I spilled something and need to clean it. Bring something to the black table to help.” Without memory, the agent briefly sees the cleaning tool (cleanser) but fails to track its relevance and instead wanders into unrelated rooms, losing sight of the correct object and ultimately failing. With memory, the agent retains the observation of the cleanser and consistently pursues the correct sequence, successfully retrieving and placing the cleaning item at the target location.
Figure 7: Navigation task visualization (without vs. with memory). Task: “Navigate to the toaster in the room and be as close as possible to it.” Without memory, the agent sees the toaster in the first four steps but repeatedly moves rightward, increasing the distance instead of approaching it. This reflects a lack of persistent spatial grounding. With memory, the agent maintains awareness of the toaster’s location and navigates directly toward it, successfully reaching the target.
Figure 8: Navigation task visualization (without vs. with memory). Task: “Navigate to the laptop in the room and be as close as possible to it.” Without memory, although the laptop is visible early, the agent moves forward in a way that narrows the field of view and loses track of the laptop, showing weak spatial depth reasoning. With memory, the agent retains knowledge of the laptop’s position and adjusts its trajectory accordingly, reaching the goal reliably.
Figure 9: Manipulation task visualization (without vs. with memory). Task: “Place the prism into the black container.” Without memory , the agent misjudges the physical constraints and repeatedly collides the prism against the rim of the black container, causing the object to bounce away and leading to failure. This reflects a lack of physical interaction awareness and no recall of prior unsuccessful attempts. With memory , the agent recalls stable placement trajectories and aligns the prism carefully before lowering it smoothly into the container, completing the task successfully.
Figure 10: Manipulation task visualization (without vs. with memory). Task: “Stack the red cylinder and then the magenta cylinder in sequence.” Without memory , the agent releases objects prematurely without accounting for vertical alignment—dropping cylinders from a distance instead of gently lowering them—resulting in unstable stacking and repeated failures. With memory , the agent retrieves previous successful stacking behaviors, aligns the red cylinder first, then lowers the magenta cylinder smoothly, achieving a stable two-level stack.