Recent advances in large language models have driven the emergence of intelligent agents operating in open-world, multimodal environments. To support long-term reasoning, such agents are typically equipped with external memory systems. However, most existing multimodal agent memories rely primarily on neural representations and vector-based retrieval, which are well-suited for inductive, intuitive reasoning but fundamentally limited in supporting analytical, deductive reasoning critical for real-world decision making. To address this limitation, we propose NS-Mem, a long-term neuro-symbolic memory framework designed to advance multimodal agent reasoning by integrating neural memory with explicit symbolic structures and rules. Specifically, NS-Mem is operated around three core components of a memory system: (1) a three-layer memory architecture that consists episodic layer, semantic layer and logic rule layer, (2) a memory construction and maintenance mechanism implemented by SK-Gen that automatically consolidates structured knowledge from accumulated multimodal experiences and incrementally updates both neural representations and symbolic rules, and (3) a hybrid memory retrieval mechanism that combines similarity-based search with deterministic symbolic query functions to support structured reasoning. Experiments on real-world multimodal reasoning benchmarks demonstrate that Neural-Symbolic Memory achieves an average 4.35% improvement in overall reasoning accuracy over pure neural memory systems, with gains of up to 12.5% on constrained reasoning queries, validating the effectiveness of NS-Mem.
Figures & tables
Figure 1. An example of a vector-centric multimodal agent on a constrained query.
Figure 2. Overview of the NS-Mem framework. Raw multimodal data is processed through a three-layer memory prototype, maintained via the SK-Gen mechanism for distillation and incremental updates, and accessed through a hybrid retrieval framework designed for complex reasoning.
Figure 3. Case study on vector-centric Memory and NS-Mem.
M3-Bench-robot
M3-Bench-web
Method
MD
MH
CM
HU
GK
All
MD
MH
CM
HU
GK
All
Socratic Models
Qwen2.5-Omni-7b
2.1
1.4
1.5
1.5
2.1
2.0
8.9
8.8
13.7
10.8
14.1
11.3
Qwen2.5-VL-7b
2.9
3.8
3.6
4.6
3.4
3.4
11.9
10.5
13.4
14.0
20.9
14.9
Gemini-1.5-Pro
6.5
7.5
8.0
9.7
7.6
8.0
18.0
17.9
23.8
23.1
28.7
23.2
GPT-4o
9.3
9.0
8.4
10.2
7.3
8.5
21.3
21.9
30.9
27.1
39.6
28.7
Table 1. Performance comparison on M3-Bench-robot and M3-Bench-web. MD: Multi-Detail, MH: Multi-Hop, CM: Cross-Modal, HU: Human Understanding, GK: General Knowledge. Best results in each column are underlined .
Method
Factual
Procedural
Constrained
M3-Agent
52.5
23.8
25.0
NS-Mem
54.3
35.7
37.5
Δ
+1.8
+11.9
+12.5
Table 2. Accuracy by Query Type
Dataset
Metric
Baseline
NS-Mem
Δ
Robot
Avg. Rounds
4.01
3.38
-15.8%
Avg. Time (sec)
45.47
42.11
-7.4%
Web
Avg. Rounds
3.14
2.84
-9.6%
Avg. Time (sec)
36.04
34.57
-4.1%
Table 3. Efficiency Comparison on M3-Bench Datasets
Figure 4. Hyper-parameter analysis across different thresholds and weights. (a) Impact of τ on accuracy across query types. (b) Impact of of δ on knowledge consolidation and merge (c) Impact of α on accuracy and efficiency.
Configuration
Description
Web
Robot
Baseline
M3-Agent
48.9
30.7
w/o Symbolic
+Neuro +DAG
50.0
31.7
w/o Neuro
+Symbolic +DAG
51.7
33.1
Full NS-Mem
+Neuro +Symbolic +DAG
53.6
34.7
Table 4. Ablation Study on M3-Bench Datasets
Metric
Dataset
Static
Dynamic
Δ
Accuracy (%)
Robot
33.8
34.7
+0.9
Web
53.0
53.6
+0.6
Avg. Rounds
Robot
3.52
3.38
-0.14
Web
2.92
2.84
-0.08
Table 5. Incremental Update Evaluation.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
M
Memory system M=(Lepi,Lsem,Llogic,E)
Lepi,Lsem,Llogic
Episodic, Semantic, and Logic layers
O={o1,o2,…}
Multimodal observation stream
ot=(otv,ota,ots)
Observation at time t (visual, audio, text)
Q,A
Query space and answer space
q∈Q
A user query in natural language
Appendix
Table 6. Key notations used throughout this paper.
We present NeuSymMS, an adaptive memory system that enables large language model (LLM) agents to learn, remember, and reason about users across sessions via a hybrid neuro-symbolic architecture. NeuSymMS couples neural fact extraction from unstructured dialogue using LLMs and a CLIPS-based expert system that classifies, deduplicates, and reconciles facts under explicit lifecycle rules. The system represents knowledge as subject-relation-value triples stored in relational database management system. It supports user/agents/agent-to-agent scoping, and implements a dual-horizon (short-term and long-term) memory model. IT leverages access-based promotion and time-based pruning of the memory on both horizpons. NeuSymMS maintains continuity of memory while avoiding context-window bloat and cross-entity contamination. We argue that this architecture offers a practical path to trustworthy, auditable memory for production agentic systems and discuss its novelty relative to log retrieval, summarization, and key-value approaches.
Personalized agents are required to reason over long-term history interactions to infer both explicit preferences and implicit behavioral evidence. While early flat retrieval methods score memory fragments independently and neglect the distributed information, current structured memory frameworks rely on query-agnostic static graphs that fail to capture the context-dependent relations. Crucially, raw textual memories are inherently entangled and noisy, making fine-grained personalization and cross-session reasoning computationally prohibitive. To this end, we present LGM, a novel neuro-symbolic framework that shifts long-term memory disentanglement into a continuous latent space. Specifically, (i) instead of persisting fixed graphs, we design a tailored latent graph construction with a sparse autoencoder. Subject to each query, it maps historical interactions into latent memory nodes and disentangles the memory traces into sparse concept activations, dynamically synthesizing query-aware relational edge weights. (ii) A graph encoder then treats the query embedding as a conditioning preference to direct non-linear message passing across the task-specific latent subgraph. This yields a highly expressive memory representation for effective activations. Extensive experiments on long-term personalization benchmarks demonstrate that LGM significantly outperforms state-of-the-art baselines in capturing both explicit and implicit preferences while enabling personalized responses.
Cai Ke, Xinghao Chen, Xiaoyu Shen +6
Tencent Youtu Lab · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, EIT, Ningbo, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China +1
Long-term memory is essential for LVLM agents to maintain consistency and integrate information across extended multimodal interactions. Existing agent memory systems, however, often reduce visual experiences into textual summaries or rely on static retrieve-then-reason pipelines, which are inefficient at query time and brittle when questions require image-text binding, temporal updates, or visual details. We propose Prospective Multimodal Memory Compilation, a framework that shifts part of the memory reasoning process from query time to memory consolidation time. Given accumulated multimodal interactions, a Questioner predicts future question candidates, a Planner compiles question-conditioned multimodal memory programs, and a Doubter verifies whether the planned evidence path can support the predicted answer. The verified question-program pairs form a structured question bank for efficient query-time routing and evidence retrieval. Experiments on multimodal long-term memory benchmarks show that our method improves answer quality and visual evidence recall while reducing query-time token and latency costs. Extensive ablations analyze the effects of self-feedback, dynamic planning, raw-image access, and question bank coverage.
Jingyu Sun, Yan Lin, Yuyang Xue +10
The University of Manchester · The University of Newcastle · The University of Edinburgh +5