cs.CLOct 15, 2026
SaveMultimodal Filesystem Memory For Long-Horizon Embodied Agents
Abstract
Agents operating over long horizons need an effective memory that allows them to reuse their experience throughout long interactions. If the entire history of an interaction is retained, it becomes difficult to find the relevant experience, while compressing it into summaries may result in the loss of visual and spatial details that are required later on. We present MFM, a multimodal filesystem memory architecture that keeps complementary types of agent memory. Episodic memory stores the observations, actions, and outcomes, and spatial memory associates experiences with locations. Semantic and procedural memory build up general knowledge and reusable methods of acting, and working memory holds the current plans. MFM makes use of searchable textual records to connect these different representations to the underlying multimodal episodes. The agents are able to find previous experiences by means of contextual cues and to examine the original observations in order to recover details that are missing from their textual descriptions. Through consolidation, the agents convert experience into higher-level knowledge and procedures while still keeping references to the original episodes. In the case of long-horizon Minecraft and computer-use tasks, MFM leads to improved performance with three base models. With GPT-6-Astra, MFM completes Mine Diamond from Scratch and reaches 16 out of 24 milestones on the Ender Dragon task, as against 12.7 for the best baseline. On CUABench-KiCad it attains a mean reward of 0.440, as compared to 0.400 for the best baseline and 0.267 in the case of no persistent memory.
Figures & tables
Figure 1: MFM’s Memory Evolution in Minecraft. Over 36,000 steps, MFM continuously updates its memory and retrieves from it to support decision-making.
| Method | Raw multimodal evidence | Index with evidence pointers | Spatial memory | Agent-writable memory | Knowledge linked to evidence | Retrieval |
| Voyager ( Wang et al., 2024 ) | ✗ | ✗ | ✗ | ✓ a | ✗ | Embedding similarity |
| JARVIS-1 ( Wang et al., 2025 ) | ✓ b | ✗ | ✗ | ✓ b | ✗ | Multimodal similarity |
| MrSteve ( Park et al., 2025 ) | ✓ | ✗ | ✓ | ✗ | ✗ | Place/event lookup |
| PRO-LONG ( Fox et al., 2026 ) | ✗ c | ✗ | ✗ | ✓ d | ✗ d | Regex / code over log |
| VISTA ( Han et al., 2026 ) | ✓ | ✗ e | ✗ | ✓ e | ✗ | Inspect by frame ID |
| MFM (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | Cue query + evidence inspection |
Table 1: MFM versus prior agent memory systems across key memory properties.
Figure 2: Overview of MFM. Five forms of memory in one filesystem. Harness-written episodic memory records observations, states, actions, and outcomes every step, with searchable cues for locating them; spatial memory ties experiences to locations. Agent-written semantic , procedural , and working memory hold facts, reusable procedures, and the current plan, each citing its supporting episodes. The agent retrieves by searching cues and inspecting original evidence (Sec. 3.3 ), and consolidates experience into knowledge, procedures, and plans without modifying episodic records (Sec. 3.4 ).
Figure 3: Memory contents and operations in MFM. A Minecraft trajectory shows how each form of memory is populated and used. (a) An episode: observation, state, action, and outcome stored as persistent files. (b) Its index entry: searchable cues, including spatial coordinates, with references to the records. (c) Retrieval: the agent locates episodes by step and inspects the referenced frame. (d) Consolidation: the evidence updates semantic ( GUIDE.md ) and working memory while earlier entries and episode references are preserved. Bottom: where each form of memory lives at this step.
| No visual obs. | Flat | No agent procedures | No consolidation | |||||
| Model | Diamond | Dragon | Diamond | Dragon | Diamond | Dragon | Diamond | Dragon |
| GPT-5.6-Sol | ||||||||
Table 4: Ablation of MFM’s memory components. Effects of removing stored frames, associative indexing, agent-written procedures or consolidation on the Diamond and Dragon tasks with GPT-5.6-Sol.
Figure 4: Qualitative behavior on the Nether portal task. MFM successfully completes the Nether portal, whereas VISTA and PRO-LONG fail to finish the structure.
| Iron shield | Climb out | Cook meat | ||||||
| w.o. | w. | w.o. | w. | w.o. | w. | |||
| 2422 | 1705 | 2010 | 1867 | 640 | 476 | |||
Table 5: Skills learned from one task speed up new tasks. Mean environment steps (over 5 seeds each) to solve three new tasks without (w.o.) and with (w.) the scripts an earlier life wrote installed. Every life solved its task.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Milestone progression on Kill the Ender Dragon with GPT-6-Astra. Each curve shows one run’s progression through the Minecraft technology tree against environment steps (log scale). Within the 36,000-step budget (dashed line), MFM reaches 16 of 24 milestones, compared with 13 for VISTA, 12 for PRO-LONG, and 6 for Direct Evaluation. The shaded region marks milestones that no method reached. One run is shown per method; Table 3 reports seed-averaged results.
Figure 8: Environment interaction per agent turn. Rolling median of environment steps executed per agent turn over each run. Columns are the three base models; rows are Mine Diamond from Scratch (top) and Kill the Ender Dragon (bottom). Higher values mean more environment interaction between successive model calls; the vertical axis is logarithmic. Shaded intervals mark where MFM exceeds both VISTA and PRO-LONG, and the annotation in each panel gives the shaded fraction.
Figure 9: Correcting and confirming perception with visual memory. (a–b) At step 4,588, the agent records a pale animal on the hillside. Magnifying the stored frame shows a wolf rather than a sheep, so the agent does not attack and looks for food elsewhere. (c–d) At step 4,917, the agent is unsure whether an animal on the slope is a pink sheep. Magnifying the stored frame confirms a sheep rather than a pig, and the agent hunts it for food.
Figure 10: Memory-guided spatial relocation. The retrieved and current views line up closely, which lets the agent identify the hole’s rim and recover its exact location.
Figure 11: Cross-life visual memory retrieval. After dying, the agent retrieves a frame recorded in an earlier life and compares it with its current view. It recognizes the grass bank ahead as the entrance to that life’s iron mine, digs in, and mines iron ore at step 6,565.
Figure 12: Using visual memory to recover mining orientation. The agent inspects the frame of an earlier cut that failed to open a level passage. It sees that the view was pitched upward rather than at the mining face. It then re-aims with look_abs and resumes mining at the correct orientation.
Figure 13: Qualitative comparison on CUABench-KiCad. Final schematics produced by MFM (top) and VISTA (bottom); each column is one task. (a, d) Task 55f2eefb, which switches an LED circuit’s supply from 9 V to 5 V: VISTA’s label still reads +9 V even though Change Symbol returned “OK”. (b, e) Task c4baee36, a buck regulator: both agents place the same components and wiring but use different supply labels. (c, f) Task b7148878, a two-op-amp circuit: MFM leaves one pin unconnected, compared with three for VISTA.
Figure 14: Self-verification of edits on CUABench-KiCad. Each row shows one task. MFM inspects the schematic against the specification (left), opens the Electrical Rules Checker (middle), and reads the report to confirm that only the expected errors remain (right).
Figure 15: Cross-task reuse of episodic memory in KiCad. While working on the current task (top), the agent retrieves an episode from an earlier task (bottom) with a nearly identical action sequence. The sequence is: save the blank schematic, open Export Netlist , select the native KiCad format, and set the output path. The retrieved episode serves as a concrete precedent for the current task.
Figure 16: What follows a look at a stored frame. Left: the share of turns whose plan names a destination the current plan did not have, after the agent opened a stored episode frame (blue) against turns in which it opened no image at all (orange). Right: of those new destinations, the share falling within blocks of the position at which the retrieved episode was recorded (blue), against the share of the whole episode store that would match the same coordinate by position alone (orange).
| Capability | Direct | PRO-LONG | VISTA | MFM |
| Standing brief | 12 KB | 12 KB | 12 KB | 12 KB |
| Procedure set | 28 atoms | 28 atoms | 28 atoms | 28 atoms |
| Per-turn [RECENT] / [MAP] | no (as direct agent is not supposed to have any memory) | yes | yes | yes |
| Episode store and query tool | no | no | archived frames via inspect | yes |
| Write access to own files; procedure execution | yes | yes | yes | yes |
| Agent-written procedures (observed) | 0 | 0 | 0 | 4 written / 14 calls |
Table 6: Configuration of each method on Minecraft tasks. All methods share the action and procedure reference, the knowledge sections, the ground-truth rules, and the per-turn state block; Direct Evaluation omits the block’s [RECENT] and [MAP] fields. All methods with memory can write files and execute stored procedures through the same interface.
| Atom | Description |
| Aiming and walking | |
| look | Turn the camera by a relative (pitch, yaw) in degrees, in safe increments. |
| look_at | Aim the crosshair at a world coordinate, using the exact position from [STATE] . |
| look_abs | Steer the camera to an absolute pitch and yaw, closed-loop against [STATE] at every step; works from any starting pose. |
| go_to | Walk to a world coordinate , re-aiming against [STATE] at every step until within reach. |
| watch | Wait up to n steps, stopping as soon as a condition read from the game state holds: none (plain wait), inventory (any count changes), gui (a screen opens), health, damage, y_below:<v> , y_above:<v> , or moved:<blocks> . |
Table 7: The 28 atomic procedures shared by all methods ( procedures.ATOMS ). They include closed-loop movement and aiming primitives, GUI cursor control, one-call crafting, and placement.
| Quantity | Value |
| Environment steps | 12,000 |
| Model calls | 130 |
| Episodes stored | 636 |
| Episode directory on disk | 46 MB |
| Index ( index.jsonl ) size | 313 KB |
| Full-resolution frames on disk | 720 MB |
Table 8: Storage, retrieval, and memory-use statistics for the GPT-6-Astra Mine Diamond run that reached all 10 milestones.
| Method | Milest. obs. | Calls | Tokens (M) | USD ($) | |
| MFM | 10 | 2,129 | 47 | 108 | |
| VISTA | 9 | 2,177 | 69 | 172 | |
| PRO-LONG | 8 | 895 | 57 | 168 | |
| Direct | 6 | 3,934 | 155 | 969 |
Table 9: Costs of Ender Dragon runs with Claude Opus 5 (36,000 steps) using Claude Code.
Figure 17: Milestone progression for the Minecraft tasks. Mine Diamond from Scratch contains 10 milestones, while Kill Ender Dragon shares the same initial progression and adds 14 milestones required to reach and defeat the Ender Dragon. Numbers in the left column index the Mine Diamond list; inner and right-column numbers index the Ender Dragon list, in which Diamond ore is inserted as milestone 10, so Diamond becomes 11 and the added milestones continue from 12. Note that the milestones do not need to be finished strictly in this order.
Figure 18: Milestone structure of Diamond and three related tasks. Columns align the tasks by Diamond’s ten milestones. Blue indicates shared, scored milestones; orange indicates task-specific milestones; gray indicates milestones provided at initialization and excluded from scoring. Empty cells denote Diamond milestones not scored in the corresponding task.
More in this topic
Agent Memory
Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.
See all 183 papers