S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering

Encheng Su; Jianyu Wu; Jinouwen Zhang; Qiucheng Yu; Chen Tang; Pengze Li; Lintao Wang; Aoran Wang

cs.CLApr 10, 2026

S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering

Authors: Encheng Su, Jianyu Wu, Jinouwen Zhang, Qiucheng Yu, Chen Tang, Pengze Li, Lintao Wang, Aoran Wang,

Organizations: University of Science and Technology of China · Shanghai Jiao Tong University · Shanghai AI Laboratory · City University of Hong Kong ·

Abstract

Long-horizon memory question answering often requires sparse evidence from heterogeneous histories, including events, object states, visual observations, temporal relations, and causal steps. Existing memory interfaces expand reader context, retrieve semantically related chunks, or expose graph neighborhoods, but they are not explicitly designed to select compact evidence for a fixed reader. We propose Structured Spatiotemporal Scene--Event Memory (S3Mem), a query-time memory interface that writes textual, visual, and agent-use histories into structured scene--event units and routes compact evidence packs to the reader. Its router scores candidate units, query anchors, and anchor--support links, enabling both single-hop selection and short multi-hop evidence chains without reader fine-tuning or test-time training. Across LoCoMo, EMemBench Visual Games, and AMA-Bench, S3Mem provides a strong score--token trade-off, with the clearest gains on localized event, state, temporal, causal, or provenance evidence. On LoCoMo, S3Mem reaches $0.48$ F1 and $0.40$ BLEU with (1{,}073) evidence tokens per question, about $15.8\times$ fewer than the LoCoMo reference. On EMemBench Visual Games, it obtains the best F1 and second-best accuracy with only $189$ tokens.On AMA-Bench, it is not the highest-scoring method, but remains competitive while using the fewest reader-visible evidence tokens.

Figures & tables

Figure 2: Overview of the $\mathrm{S}^{3}$ Mem pipeline. Left: visual, textual, and agent-use trajectories provide heterogeneous long-horizon memory sources. Middle: raw records are written into structured scene–event units, and candidate units or unit pairs are scored by support, anchor, and edge routing heads. Right: query anchors guide evidence selection, after which the packer constructs a compact single-hop unit or multi-hop anchor–support chain for the frozen answer-time reader.

Method	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	Tokens
	MultiHop ( $N{=}282$ )		Temporal ( $N{=}321$ )		OpenDomain ( $N{=}96$ )		SingleHop ( $N{=}841$ )		Adversarial ( $N{=}446$ )		Overall ( $N{=}1986$ )
LoCoMo ( Maharana et al., 2024 )	0.28	0.18	0.09	0.06	0.16	0.15	0.62	0.54	0.53	0.51	0.44	0.39	16,910
ReadAgent ( Lee et al., 2024 )	0.15	0.10	0.04	0.03	0.09	0.08	0.12	0.10	0.07	0.06	0.10	0.08	805
MemoryBank ( Zhong et al., 2024 )	0.06	0.05	0.02	0.02	0.06	0.05	0.08	0.07	0.04	0.04	0.06	0.05	569
MemGPT ( Packer et al., 2023 )	0.30	0.23	0.17	0.13	0.12	0.12	0.60	0.53	0.35	0.34	0.41	0.36	16,987
A-Mem ( Xu et al., 2025 )	0.33	0.24	0.39	0.31	0.17	0.16	0.48	0.43	0.36	0.36	0.41	0.35	1,216

Table 1: QA performance on the LoCoMo benchmark with GPT-4o. Overall scores are weighted by the number of questions per category (

N{=}1986

). Best results in bold , second-best underlined .

Method	Recall	Causal Inf.	State Updating	State Abstraction	Average	tok/q
AriGraph Anokhin et al. (2024)	0.47	0.58	0.36	0.29	0.43	2,207
HippoRAG2 Gutiérrez et al. (2025)	0.46	0.51	0.39	0.24	0.41	3,633
A-Mem Xu et al. (2025)	0.42	0.54	0.38	0.19	0.40	2,579
GraphRAG Edge et al. (2024)	0.39	0.47	0.36	0.17	0.36	1,274
mem0 Chhikara et al. (2025)	0.32	0.56	0.32	0.19	0.35	20,932
$\mathrm{S}^{3}$ Mem	0.36	0.58	0.36	0.21	0.39	735

Table 2: AMA-Bench results. Best results in bold , second-best underlined .

Method	Acc	F1	tok/q
A-Mem Xu et al. (2025)	0.42	0.32	6,145
mem0 Chhikara et al. (2025)	0.33	0.09	2,752
HippoRAG2 Gutiérrez et al. (2025)	0.34	0.36	2,938
AriGraph Anokhin et al. (2024)	0.34	0.34	1,293
GraphRAG Edge et al. (2024)	0.12	0.12	779
$\mathrm{S}^{3}$ Mem	0.36	0.39	189

Table 3: Results on EMemBench Visual Games. Best results are shown in bold and second-best results are underlined . tok/q denotes the average number of evidence tokens exposed to the reader per question.

Variant	Representation	Acc	tok/q
$\mathrm{S}^{3}$ Mem	Full unit	0.3495	454
Plain chunk	Text chunk	0.2517	392
Event-only	Event field	0.1163	167
Object-only	Object field	0.0953	332

Table 4: Representation ablation on EMemBench External. The full structured unit outperforms plain chunks and single-field variants.

Reader model	GraphRAG	LightMem	HippoRAG	$\mathrm{S}^{3}$ Mem
DeepSeek-v4-pro DeepSeek-AI (2026)	35.1	31.0	47.6	49.6
Gemini3Flash Google DeepMind (2025)	34.3	33.0	38.1	40.1
GLM5.1 GLM-5-Team et al. (2026)	42.5	48.8	53.0	49.6

Table 5: Reader-model sensitivity on EMemBench text external. Each cell reports the official score under the corresponding reader.

Appendix Section	Content Description
App. A	Implementation details for structured scene–event units, candidate proposal, router scoring, and evidence packing.
App. B	Router implementation, feature construction, weak-label generation, candidate labeling, and a training-instance example.
App. C	Weak-label construction, router training targets, and training–inference separation.
App. D	Train/test separation, no-gold inference boundary, target adaptation policy, and leakage-control rules.
App. E	Benchmark conversion, scoring protocols, token accounting, and baseline provenance.
App. F	Supplemental results and diagnostic analyses that clarify, but do not replace, the main benchmark tables.

Table 6: Overview of appendix sections.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Rank	Step	Action	$y^{\mathrm{sup}}$	$y^{\mathrm{anc}}$	Cost
1	3	mix picture	1	1	33
2	4	mix picture	1	1	33
3	1	look at orange	0	0	29
4	5	mix door to kitchen	0	0	37
6	39	focus on hallway	0	0	109

Appendix

Table 7: Example router training instance. The question requires counting two occurrences of the same action within a step range. Weak support and anchor labels are assigned to the two matching memory units, while high-overlap distractors become hard negatives.

Source	Raw history type	Router labels
Crafter	visual/game trajectories	support / anchor / edge
Jericho	text-game trajectories	support / anchor / edge
ScienceWorld	science task trajectories	support / anchor / edge
ALFWorld	embodied instruction traces	support / anchor / edge

Appendix

Table 8: Weak-supervision sources for router training. The constructed labels supervise evidence selection rather than answer generation.

Long-Term Conversational Memory

Long-term conversational memory helps AI assistants retain, update, and use information from past conversations across sessions.

See all 267 papers

StructMem: Structured Memory for Long-Horizon Behavior in LLMs
Apr 23, 2026Buqiang Xu, Yijun Chen, Jizhan Fang +5Multi-Hop QA Temporal Reasoning
Zhejiang University · Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph · Ant Group
EM^2Mem: Event-Centric Multimodal Memory for Large Language Models
Sep 1, 2026Yijun Chen, Yaqi Zheng, Yanya Li +11Multimodal RAG Efficient Multimodal Inference
Zhejiang University · South China University of Technology · Lenovo Group Limited
Temporal Evidence Forests for Long-Horizon Multi-Party Conversational Memory
Oct 9, 2026Junyou Guo, Chongyang Tao, Yan He +2LLM Agent Memory Conversational Memory
School of Computer Science and Engineering, Beihang University, Beijing, China · Alibaba Group

Apr 23, 2026J/K move · Enter open · S save

StructMem: Structured Memory for Long-Horizon Behavior in LLMs

Buqiang Xu, Yijun Chen, Jizhan Fang, Ruobin Zhong+4

Zhejiang University · Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph · Ant Group

Multi-Hop QA Temporal Reasoning Persistent Memory for Language Models Hierarchical Memory Long-Term Conversational Memory

Open paper AskPDF arXiv

Long-term conversational agents need memory systems that capture relationships between events, not merely isolated facts, to support temporal reasoning and multi-hop question answering. Current approaches face a fundamental trade-off: flat memory is efficient but fails to model relational structure, while graph-based memory enables structured reasoning at the cost of expensive and fragile construction. To address these issues, we propose StructMem, a structure-enriched hierarchical memory framework that preserves event-level bindings and induces cross-event connections. By temporally anchoring dual perspectives and performing periodic semantic consolidation, StructMem improves temporal reasoning and multi-hop performance on LoCoMo, while substantially reducing token usage, API calls, and runtime compared to prior memory systems, see https://github.com/zjunlp/LightMem .

Abstract

Long-Term Conversational Memory

Explore similar work

StructMem: Structured Memory for Long-Horizon Behavior in LLMs

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models

Temporal Evidence Forests for Long-Horizon Multi-Party Conversational Memory