AstroMemBench: A Benchmark for Memory Systems in Embodied Spacecraft Agents
Organizations: School of Aeronautics and Astronautics, Zhejiang University · College of Information Science and Electronic Engineering, Zhejiang University · Institute of Advanced Technology, Zhejiang University
Abstract
Long-term spacecraft autonomy requires embodied agents to use operational history under communication delays, time-critical operations, environmental uncertainty, complex physical interactions, power and computational constraints, and safety and reliability requirements. We introduce AstroMemBench and a Scenario-Challenge-Capability design method that maps these requirements to executable memory tests. The benchmark covers 16 memory subchallenges through 140 templates and 2240 instances across nine tasks in planetary surface operations, on-orbit servicing, and deep-space exploration. A deterministic environment scores actions for task completion, state changes, safety, and resource use. Ablations under nine conditions show that the benchmark is sensitive to how accurately agents recover and use historical information and to differences in memory-system design. Main experiments compare six configurations. Mem0, Chain-of-Memory, LightMem, and retrieval-augmented generation (RAG) improve same-session mean scores over native Pi Agent, with no clear advantage of the three memory systems over RAG. After a session restart, Mem0 outperforms RAG by 12.25 points. Benefits favor instruction retention and result reuse over ongoing state tracking and energy reservation. AstroMemBench supports the design, development, and selection of memory systems for embodied spacecraft agents. AstroMemBench 1.0 and its evaluation code are publicly available: https://github.com/flyingView/AstroMemBench-v1.
Figures & tables
| General capabilities | Space-relevant capabilities | ||||||||||
| Benchmark | Factual recall | Time assoc. | State maint. | Visual memory | Exper. transfer | Instr. validity | Action window | Unk. environ. | Phys. interact. | Res. cost | Safety and rel. |
| LoCoMo [ 16 ] | — | — | — | — | — | — | |||||
| LongMemEval [ 17 ] | — | — | — | — | — | — | — | ||||
| MemoryAgentBench [ 13 ] | — | — | — | — | — | — | — | ||||
| Mem-Gallery [ 19 ] | — | — | — | — | — | — | |||||
| LoCoMo-Plus [ 20 ] | — | — | — | — | — | — | |||||
| Subchallenge | Construction of the memory test |
|---|---|
| C1 Instruction retention | Distribute ground instructions across the history; require continued adherence to still-valid operational requirements after communication is lost. |
| C2 Rule disambiguation | Present issuance, replacement, and revocation records out of order; require action selection based on validity relationships. |
| C3 Downlink confirmation | Mix transmission attempts, failure records, and reception acknowledgments; require determining whether delivery is complete. |
| T1 Validity assessment | Provide validity conditions for rules, data, or credentials; require assessing their continued applicability at the current time. |
| T2 Deferred tasks | Register subsequent tasks early in the history; require recalling and executing them when temporal or event triggers occur. |
| E1 Experience matching | Provide experience from different environments or operating conditions; require selecting an operational plan applicable to current conditions. |
| Scenario task | Templates | Instances per template | Total instances | ||
| Easy | Medium | Hard | |||
| P Planetary surface spacecraft | |||||
| P1 Area surveying | 16 | 6 | 6 | 4 | 256 |
| P2 Sampling at designated sites | 16 | 6 | 6 | 4 | 256 |
| P3 Cargo transport | 16 | 6 | 6 | 4 | 256 |
| O On-orbit servicing spacecraft | |||||
| Template average (%) | Template maximum (%) | |
|---|---|---|
| 1 | 55.6601 | 99.8824 |
| 4 | 4.8511 | 66.7455 |
| 8 | 0.0229 | 1.7686 |
| 16 |
| Item | Configuration |
|---|---|
| Hardware platform | NVIDIA Thor (ARM64) |
| Runtime environment | Docker, Ubuntu 24.04 |
| Base runtimes | Python 3.12.3; Node.js 22.23.2 |
| Main reasoning model | GLM-5.3-Flash |
| Embedding model | Qwen3-VL-Embedding-2B |
| Embedding dimensions | 2048 |
| Group | Configuration | History | Window | Session |
| A | Pi | Clean | 262K | Same-session |
| B | Pi | Clean | 262K | New-session |
| C | Pi | With background | 262K | Same-session |
| D | Pi | With background | 262K | New-session |
| E | Pi | With background | 128K | Same-session |
| F | Pi+Mem0 | Clean | 262K | Same-session |
| Comparison theme | Comparison Contrast Mean score difference [95% CI] |
|---|---|
| Role of raw history | Clean history: same-session minus new-session With background: same-session minus new-session |
| Benefits of memory integration | Mem0 integration in the same session RAG integration in the same session Mem0 integration in a new session RAG integration in a new session |
| Preservation of cross-session execution | Mem0: same-session minus new-session RAG: same-session minus new-session |
| System differences across session boundaries | Same-session: Mem0 minus RAG New-session: Mem0 minus RAG |
| Effects of routine background | Same-session: clean minus background New-session: clean minus background Between-session difference in background loss |
| Changes in native history management | 128K native compaction minus 262K |
| Configuration (group) | Same-session | New-session | Session gap |
|---|---|---|---|
| Pi (A/B) | 83.12 | 21.10 | 62.02 |
| Pi+RAG (H/I) | 88.63 | 59.43 | 29.20 |
| Pi+Mem0 (F/G) | 87.56 | 75.71 | 11.85 |
| Configuration | Mean score [95% CI] | Full-score rate (%) | /% | Tokens (M) | Score efficiency |
| Same-session (128K) | |||||
| Pi | 78.20 [76.80,79.57] | 69.82 | 0.00 | 2721.58 | 28.73 |
| Pi+ RAG | 82.74 [81.45,83.99] | 75.71 | 20.82 | 3089.99 | 26.78 |
| Pi+ Mem0 | 81.70 [80.38,82.99] | 74.60 | 16.07 | 3982.69 | 20.51 |
| Pi+ CoM | 82.07 [80.79,83.34] | 74.64 | 17.75 | 3585.64 | 22.89 |
| Pi+ LightMem | 79.59 [78.17,80.96] | 71.38 | 6.38 | 3077.36 | 25.86 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Comparison column | Criteria for direct and partial coverage | Corresponding subchallenges |
|---|---|---|
| Factual recall | Historical objects, attributes, events, or parameters support current answers or actions. | All six categories |
| Temporal association | Uses event order, counts, or duration; merely retaining history in order counts as partial coverage. | C2, T, I2 |
| State maintenance | Accumulates task states, uses updated facts, or assesses the applicability of prior experience. | C3, E2, I1/I2 |
| Visual memory | Uses objects, attributes, or locations in earlier images to determine current judgments. | E3-P1/P2 |
| Experience transfer | Applies rules or skills to new objects, starting points, or subsequent tasks; matching familiar cases, reusing preferences, and exact imitation count as partial coverage. | E1 (partial) |
| Instruction validity | Uses history to retain or identify currently valid instructions; inferring persistent rules only from demonstrations counts as partial coverage. | C1/C2, T1 |
| Benchmark and source location | History and current task | Reuse and interaction | Evidence for capability coverage |
|---|---|---|---|
| LoCoMo [ 16 ] §4.1/4.3; Fig. 2 | Multisession text and images; QA, summarization, and generation | Predetermined session histories; offline outputs | Retrospective temporal QA, state generation, and historical photographs support the corresponding memory capabilities; preference reuse is related to transfer. |
| LongMemEval [ 17 ] §3; Fig. 1; Table 1 | Multisession text; QA and abstention | Predetermined session histories; offline answers | Extraction, temporal reasoning, and knowledge updating directly test the first three general capabilities; recommendations based on historical preferences are related to transfer. |
| MemoryAgentBench [ 13 ] §3.1–3.3 | Incremental materials; QA, classification, and summarization | Incremental reading of the same materials; offline outputs | EventQA tests order, FactConsolidation tests factual updates, and test-time classification tests transfer. |
| Mem-Gallery [ 19 ] §3.3; Appendix A.3; Fig. 16 | Multisession text and images; QA and evidence retrieval | Predetermined session histories; offline outputs | Conflict resolution and visual reasoning test state and visual memory; classification of new images tests transfer; efficiency is measured as system runtime. |
| LoCoMo-Plus [ 20 ] §3–4; Appendix C | Conversations with implicit constraints; constraint-consistent answers | Predetermined session histories; offline answers | The complete framework retains factual and temporal tasks; persistent goals and emotions test state; adherence to implicit constraints is related to instruction retention. |
| AMA-Bench [ 21 ] §3.1–3.3; Figs. 4–5 | Real and synthetic trajectories; experience QA | A single existing trajectory; offline answers | Trajectory QA tests recall, time, and updates through questions about historical actions. |
| Benchmark and source location | History and current task | Reuse and interaction | Evidence for capability coverage |
|---|---|---|---|
| Mem2ActBench [ 22 ] §3.3–4.1; Tables 8–10 | Multisession text; tool and argument generation | Predetermined histories; offline call generation | Evolution chains test updated arguments and requirements; dates and budgets are parameters recovered for tool calls. |
| PM-Bench [ 23 ] §3.1–3.3 | Schedules, events, and queries; actions when due | The same situation across days; queries and execution | Cancellation and rescheduling test valid intentions; time windows and event triggers test action timing; queries through hidden channels are related to information acquisition in unknown environments. |
| MemoryArena [ 24 ] §3.1; Fig. 2 | Own execution and feedback; task completion | Interdependent subtasks; tool interaction | Prior requirements constrain later choices; travel dates, tool searches, and monetary budgets are related to action windows, unknown environments, and resource cost, respectively. |
| FindingDory [ 26 ] §3; Table 2 | Images, poses, and actions; navigation | Navigation after prescribed experiences | Historical attributes, rearrangements, and images test four general capabilities; unobservable locations support unknown-environment coverage; navigation is related to physical interaction. |
| LMEE-Bench [ 27 ] §3–5; Figs. 3–4 | Exploration images and locations; navigation and QA | Exploration and execution in the same scenario | Visit progress and images support state and visual memory; SPL is related to resource cost; instructions are issued incrementally during exploration. |
| MIKASA-Robo [ 28 ] §6; Appendix K | Images and robot states; physical operations | A single interaction process; current operations | Color sequences, relative orientations, and stage progress test memory; exact imitation is related to transfer; precision constraints govern current operations. |
| Subchallenge | History to retain | Current use | Execution and scoring criteria |
|---|---|---|---|
| C1 Instruction retention | Still-valid requirements from before an interruption | Continue without new ground confirmation | Meet all applicable requirements and constraints. |
| C2 Rule disambiguation | Instruction issuance, replacement, revocation, and applicability | Resolve conflicting or out-of-order records | Follow valid rules, rather than reception order. |
| C3 Downlink confirmation | Transmission attempts, targets, and reception acknowledgments | Assess delivery completion | Distinguish requests from delivery; correctly resume, retransmit, or terminate. |
| T1 Validity assessment | Activation and expiration conditions for rules, data, or credentials | Assess current applicability | Use information only within valid periods or conditions. |
| T2 Deferred tasks | Accepted tasks and their time or event triggers | Recall tasks when triggered | Execute retained requirements within their timing constraints. |
| E1 Experience matching | Experience under known environments or conditions | Match current objects and conditions | Apply suitable experience without misusing similar records. |
| Component | FM-I1-O3-001 | FM-I1-O3-002 |
|---|---|---|
| Operation rules | H1 and H2 can each connect to at most one port. To retract a target line, first disconnect it from its actual port if it is still connected; otherwise, stow it directly. Plans do not change physical states, and the other line remains in place. | |
| Shared history | Turn 2: H1 connects to P01 and H2 to P04. Turn 5: H1 is successfully disconnected from P01. Turns 10 and 27: plans to connect H1 to P05 are proposed but not executed. Subsequent H2 operations are identical in both instances, with a final connection to P01. | |
| Difference in turn 7 | Executes connection of H1 to P02; feedback indicates success; H1 is currently connected to P02. | Proposes a plan to connect H1 to P02; no physical operation has begun, and existing connections are unchanged. |
| Current request | Begin retracting H1 and submit the next operation; pump shutdown, isolation, and depressurization are already complete. | |
| Public action menu | Both instances have the same 12-option menu: disconnect H1 or H2 from one of P01–P05, or stow the line. The menu is not filtered by current state. Option 7 disconnects H1 from P02; option 4 stows H1. | |
| State reconstructed from history | H1 is connected to P02 | H1 is not connected to any port |
| Module and version | Memory writing and organization | Retrieval and delivery parameters | Adaptation and updating |
|---|---|---|---|
| RAG | Single index; 1000-character text chunks with 100-character overlap; image-containing histories jointly encode text and images from the same turn | Cosine top5; one joint query with current text and images per turn; returns original historical text, images, sources, and timestamps | Supports active search; direct retrieval without additional summarization or reranking |
| Mem0 JS OSS 3.3.1 94c3fe9f | Independent SQLite; retains native extraction, updating, and semantic memory | Retrieval limit 20; threshold 0.1 | Retains native image-description generation and text-memory pathways |
| CoM 52aa7ffd | Stores public user text and ordinarily visible assistant responses by protocol turn, forming persistent raw nodes | Cosine top20; 3 anchors, product gating, 0.5 cutoff; chain context uses average mode; nodes are embedded using complete text | Automatic retrieval: 16384 characters; 32768 including tool search in the same turn; preserves chain order, retaining the beginning and end of long entries within a shared budget and marking omissions; raw storage remains complete |
| LightMem 8449d574 | flat LightMemory; LLMLingua-2 compression and segmentation; compression rate 0.6; sensory and short-term buffers each 512 | Retrieval top60; offline update similarity threshold 0.9; completes final-segment extraction and offline organization after history ingestion | Fixes global source ID–topic correspondence, final-segment loss, and reuse of stale vectors after updating; re-embeds updated memories and records usage |
| Aspect examined | Historical evidence and actual execution | Score and consequences |
| S3-O2-001: W8 on the current unit has not been test-run; W8 on another unit Q has completed its test run | ||
| Valid evidence | H08 describes the current unit’s state. The agent cites H08 and assigns removal of A and C to arms A and B, respectively, advancing currently executable operations. | 100 points |
| Object mismatch | Retrieves H08 and H16 but cites feedback H16 from unit Q and closes repair of the current unit. | 0 points |
| Missing evidence | Assigns removal of A and C without submitting valid historical evidence. | 0 points |
| R3-O2-001: J01 stores successful parameter 12 for current equipment A, operation X, configuration F11, and operating condition T1 | ||
| Reuse of an applicable result | The current combination matches J01, so the agent directly submits 12; parameter 19 for equipment B does not apply to the current equipment. | Energy: 0; 100 points |
| Category | Pi | RAG | Mem0 | CoM | Light- Mem | Space- Mind ∗ | |
|---|---|---|---|---|---|---|---|
| C1 Instruction retention | 144 | 74.69/ 16.10 | 86.11/ 33.67 | 93.30/ 85.58 | 87.99/ 39.34 | 81.78/ 28.52 | 39.80/ 15.78 |
| C2 Rule disambiguation | 144 | 64.58/ 22.92 | 68.75/ 45.14 | 71.53/ 52.78 | 69.44/ 40.97 | 62.50/ 27.08 | 31.94/ 20.83 |
| C3 Downlink confirmation | 144 | 63.35/ 6.21 | 65.37/ 16.37 | 68.73/ 60.80 | 70.93/ 18.45 | 65.46/ 7.30 | 15.31/ 8.02 |
| T1 Validity assessment | 144 | 76.81/ 29.56 | 79.86/ 64.06 | 77.38/ 61.80 | 73.96/ 56.42 | 77.20/ 32.06 | 36.35/ 29.56 |
| T2 Deferred tasks | 144 | 79.86/ 33.33 | 84.72/ 61.81 | 87.50/ 61.81 | 83.33/ 54.86 | 81.25/ 31.25 | 53.47/ 31.25 |
| E1 Experience matching | 144 | 87.33/ 25.12 | 91.96/ 74.99 | 93.06/ 88.67 | 94.68/ 76.23 | 87.73/ 51.25 | 69.81/ 18.31 |
| Category | Pi | RAG | Mem0 | CoM | Light- Mem | Space- Mind ∗ | |
| P1 Area surveying | 256 | 78.35/ 17.35 | 84.27/ 55.79 | 78.98/ 67.14 | 83.44/ 51.37 | 78.26/ 23.42 | 42.55/ 17.34 |
| P2 Sampling at designated sites | 256 | 76.48/ 21.03 | 82.98/ 52.89 | 80.83/ 63.76 | 82.71/ 48.89 | 81.95/ 33.47 | 41.79/ 22.64 |
| P3 Cargo transport | 256 | 78.49/ 21.49 | 81.74/ 50.37 | 81.35/ 64.51 | 82.56/ 48.68 | 76.31/ 29.84 | 39.77/ 19.47 |
| O1 Target inspection | 240 | 79.01/ 20.17 | 79.10/ 53.83 | 81.79/ 62.95 | 82.40/ 53.06 | 77.79/ 27.37 | 45.70/ 21.75 |
| O2 Equipment repair | 256 | 79.08/ 22.59 | 82.58/ 52.84 | 80.41/ 66.40 | 78.67/ 48.57 | 78.95/ 37.46 | 46.02/ 21.14 |
| O3 On-orbit replenishment | 256 | 84.17/ 21.72 | 81.87/ 53.84 | 82.70/ 62.22 | 83.58/ 51.82 | 79.89/ 30.01 | 45.34/ 19.98 |
| Group | Configuration and condition | Mean score [95% CI] | Full-score rate/% |
| A | Pi; Clean/same/262K | 83.12 [81.94,84.29] | 74.46 |
| B | Pi; Clean/new/262K | 21.10 [19.92,22.30] | 11.96 |
| C | Pi; Background/same/262K | 74.16 [72.72,75.56] | 65.62 |
| D | Pi; Background/new/262K | 20.27 [19.11,21.42] | 11.12 |
| E | Pi; Background/same/128K | 77.69 [76.36,79.07] | 69.20 |
| F | Pi+Mem0; Clean/same/262K | 87.56 [86.42,88.66] | 81.65 |
Memory Agent Benchmarks
Memory agent benchmarks evaluate how AI agents store, retrieve, update, and use information across interactions and changing tasks.
See all 183 papers