A Matrix with Steps: An Episodic and Procedural Memory Method for Tool-Using Agents and Robots
Authors: Jürgen Paulhart, Peter Schüller
Organizations: [core] Informationstechnologie / Eldric · NeuraMix AI GmbH and Technische Universität Wien · NeuraMix AI GmbH · Technische Universität Wien
Agents that call tools and robots that execute tasks are challenged by two shortcomings: they repeat what failed before, and they learn from runs that they only believe succeeded. We describe the construction of a memory that addresses both, built for a production agent platform. Every run is recorded as an episode, and only an outcome confirmed by an external check, a test, a tool result or a person counts; the agent's own claim never does. What is learned from episodes is kept in a memory with two parts: an associative matrix, written with the delta rule, which finds the procedure that applies to a situation; and a section of ordered, addressable step records that answers what comes after a given step, where a step is a goal of the agent, not a motion. Because the matrix sums its associations, it keeps no order and cannot return or delete a single one; it serves as the finder, and order, outcome counts and deletion live in the records. We describe how execution outcomes are stored, how episodes are recalled and used for solving tasks, and the file format that stores the memory. In simulation, a robot that judged its own success and stopped when it believed it was done reported 96% of its tasks as done and solved 35%; with external verification it solved 79. In a store of 100000 synthetic episodes (457MB on disk), the oldest episode was still found first, no episode from another person's memory was returned, and a success vouched for only by the model was never recalled; a lookup took 16ms on a laptop and 40ms on a server node (median). Task outcomes were measured up to about 10000 confirmed records, where the exact records solved every revisit at the first attempt.
Figures & tables
Figure 1: The two parts of one memory. The matrix is the finder: a goal finds the procedure, and a bound step finds the next step. The records hold order, outcome counts and deletion, and the matrix is rebuilt from them.
Figure 2: The .emm version-6 file, read front to back. The procedure trailer is always the last trailer; its checksum covers codec, length and body, so a flipped codec byte reads as damage.
Figure 3: Packing memories into one file (behaviour only; the container’s byte layout is not part of this paper). Left: one container with typed sections, appendable while in use. Right: a bundle of members, each of one kind; a procedural member holds the procedural matrix and its records (first released in 5.0.207, and in 5.0.208; not yet written by the running system).
Figure 4: Next step correct on real text, 10 write passes. Left: with the writer’s bound chains, a linear widening holds almost as much as the sign widening. Right: without the binding, the linear widening falls to the square matrix, and only the sign widening keeps its capacity. The values are in the table of this section.
Figure 5: Procedures found (absolute) with 1024×1024 shards on 3 000 real procedures. Left: by goal text. Right: by rewordings never seen in training; a small trained router recovers most of what the true shard allows; writing the training rewordings into the shards as extra keys and choosing between two shards by the router’s confidence raises it further (test set without 76 rewordings identical to a training text). Gray lines: one large matrix and the exact scan over all procedures, on the same queries.
Robots can use historical observations to answer spatiotemporal queries in familiar environments, but the presence of relevant evidence does not imply that existing tools support the operations a task requires. We introduce ReSight, a task-driven self-improving memory system for embodied agents that expands the capabilities for processing and accessing historical observations as task demands arise. ReSight retains keyframes and their spatiotemporal associations, separating observation evidence from extensible derived fields so that stored observations can be reprocessed for new needs. An execution agent and a coding agent share this memory through different interfaces: the former calls registered tools within a bounded online budget, while the latter uses a programmable workspace and a larger exploration budget to inspect evidence, write programs, and acquire models as needed. To realize capability expansion in ReSight, we develop Explore-Distill-Mount (EDM), which identifies capability gaps from execution and exploration records without task reference answers and converts exploration-derived solutions into reusable memory builders, query tools, and skills, extending memory representations and access capabilities as needed. Experiments on three public benchmarks demonstrate effective base memory access and capability expansion. With the execution model and online budget held fixed within each benchmark, EDM reduces mean position error on OC-NaVQA from 33.10 m to 24.72 m and improves high-level success on FindingDory from 38.3% to 63.8%. With the tools and skill frozen, success still improves by 28.3 percentage points on FindingDory episodes excluded from exploration and distillation. Experiments on real robot patrol memories further demonstrate reuse of these capabilities for subsequent queries and new memories, with fewer online tool calls on average.
Lirong Che, Yang Chen, Yimeng Su +8
Tsinghua University · AGIBOT · Nanjing University +1
Task success, retrieval quality, and cost do not show whether an LLM agent used a delivered memory or what followed from that use. We introduce MemUse, which evaluates memory at its point of use in recorded trajectories: each memory delivery is compared with what the agent could already see and followed through the agent's later actions and their consequences. An agentic judge rates each delivered entry's relevance, accuracy, novelty, influence, and observed effect; programmatic checks enforce complete reading of cited evidence, and fixed rules aggregate the ratings into verdicts linked to memory histories and delivery cost. On 8,000 SWE-bench Verified episodes from three memory systems, resolve rates stay within 0.4 percentage points of a no-memory baseline, yet MemUse shows that most delivered entries were irrelevant, most relevant entries went unused, and in two systems most deliveries restated content the agent could still see. It also surfaces effects that outcomes hide: 23.2% of episodes with beneficial memory use still failed, and 5 of 16 with harmful use still succeeded.
Changran Xu, Zeju Li, Qiang Xu
Department of Computer Science and Engineering The Chinese University of Hong Kong
Modern large language model (LLM) agents can use external tools to help users solve complex tasks. However, for problems that require learning from long-term historical events or from previous agent-environment interactions, LLM agents are required to use memory mechanisms to store and retrieve experiences. While sophisticated memory systems exist for dialogue agents, few studies have empirically examined how to improve agents' tool-using capabilities through past user-agent conversations. We propose MemToolAgent, a framework that improves tool use through memory management. Our approach contains a memory extraction module that processes past experiences into structured memory entries, and a retrieval module that dynamically selects a subset of the stored memory entries. This enables more personalized and accurate responses aligned with user preferences and feedback without requiring LLM fine-tuning. In summary, this work has three main contributions: (1) a unified memory entry format that improves both general-purpose and personalized tool use without LLM fine-tuning, (2) a reflection-based memory extraction that uses environment and user feedback to distill wrong executions into critiques to store, and (3) a retrieval module that chooses how many past experiences to use based on the memory similarity distribution. MemToolAgent achieves 29%, 80%, and 17% relative improvements compared to strong baselines on the WorkBench, NESTFUL, and PEToolBench benchmarks, respectively.
Suleyman Armagan Er, Danilo Ribeiro, Yogesh Virkar +5