Recent advances in large language models have driven the emergence of intelligent agents operating in open-world, multimodal environments. To support long-term reasoning, such agents are typically equipped with external memory systems. However, most existing multimodal agent memories rely primarily on neural representations and vector-based retrieval, which are well-suited for inductive, intuitive reasoning but fundamentally limited in supporting analytical, deductive reasoning critical for real-world decision making. To address this limitation, we propose NS-Mem, a long-term neuro-symbolic memory framework designed to advance multimodal agent reasoning by integrating neural memory with explicit symbolic structures and rules. Specifically, NS-Mem is operated around three core components of a memory system: (1) a three-layer memory architecture that consists episodic layer, semantic layer and logic rule layer, (2) a memory construction and maintenance mechanism implemented by SK-Gen that automatically consolidates structured knowledge from accumulated multimodal experiences and incrementally updates both neural representations and symbolic rules, and (3) a hybrid memory retrieval mechanism that combines similarity-based search with deterministic symbolic query functions to support structured reasoning. Experiments on real-world multimodal reasoning benchmarks demonstrate that Neural-Symbolic Memory achieves an average 4.35% improvement in overall reasoning accuracy over pure neural memory systems, with gains of up to 12.5% on constrained reasoning queries, validating the effectiveness of NS-Mem.
Figures & tables
Figure 1. An example of a vector-centric multimodal agent on a constrained query.
Figure 2. Overview of the NS-Mem framework. Raw multimodal data is processed through a three-layer memory prototype, maintained via the SK-Gen mechanism for distillation and incremental updates, and accessed through a hybrid retrieval framework designed for complex reasoning.
Figure 3. Case study on vector-centric Memory and NS-Mem.
M3-Bench-robot
M3-Bench-web
Method
MD
MH
CM
HU
GK
All
MD
MH
CM
HU
GK
All
Socratic Models
Qwen2.5-Omni-7b
2.1
1.4
1.5
1.5
2.1
2.0
8.9
8.8
13.7
10.8
14.1
11.3
Qwen2.5-VL-7b
2.9
3.8
3.6
4.6
3.4
3.4
11.9
10.5
13.4
14.0
20.9
14.9
Gemini-1.5-Pro
6.5
7.5
8.0
9.7
7.6
8.0
18.0
17.9
23.8
23.1
28.7
23.2
GPT-4o
9.3
9.0
8.4
10.2
7.3
8.5
21.3
21.9
30.9
27.1
39.6
28.7
Table 1. Performance comparison on M3-Bench-robot and M3-Bench-web. MD: Multi-Detail, MH: Multi-Hop, CM: Cross-Modal, HU: Human Understanding, GK: General Knowledge. Best results in each column are underlined .
Method
Factual
Procedural
Constrained
M3-Agent
52.5
23.8
25.0
NS-Mem
54.3
35.7
37.5
Δ
+1.8
+11.9
+12.5
Table 2. Accuracy by Query Type
Dataset
Metric
Baseline
NS-Mem
Δ
Robot
Avg. Rounds
4.01
3.38
-15.8%
Avg. Time (sec)
45.47
42.11
-7.4%
Web
Avg. Rounds
3.14
2.84
-9.6%
Avg. Time (sec)
36.04
34.57
-4.1%
Table 3. Efficiency Comparison on M3-Bench Datasets
Figure 4. Hyper-parameter analysis across different thresholds and weights. (a) Impact of τ on accuracy across query types. (b) Impact of of δ on knowledge consolidation and merge (c) Impact of α on accuracy and efficiency.
Configuration
Description
Web
Robot
Baseline
M3-Agent
48.9
30.7
w/o Symbolic
+Neuro +DAG
50.0
31.7
w/o Neuro
+Symbolic +DAG
51.7
33.1
Full NS-Mem
+Neuro +Symbolic +DAG
53.6
34.7
Table 4. Ablation Study on M3-Bench Datasets
Metric
Dataset
Static
Dynamic
Δ
Accuracy (%)
Robot
33.8
34.7
+0.9
Web
53.0
53.6
+0.6
Avg. Rounds
Robot
3.52
3.38
-0.14
Web
2.92
2.84
-0.08
Table 5. Incremental Update Evaluation.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
M
Memory system M=(Lepi,Lsem,Llogic,E)
Lepi,Lsem,Llogic
Episodic, Semantic, and Logic layers
O={o1,o2,…}
Multimodal observation stream
ot=(otv,ota,ots)
Observation at time t (visual, audio, text)
Q,A
Query space and answer space
q∈Q
A user query in natural language
Appendix
Table 6. Key notations used throughout this paper.
Tencent Youtu Lab · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, EIT, Ningbo, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China +1