4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

Yanhao Jia; Yi Zhu; Yiran Zhong; Weigao Sun; Feida Zhu; Pengkun Jiao; Sihan Yang; Ming Ma

cs.CVOct 23, 2026

4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

Authors: Yanhao Jia, Yi Zhu, Yiran Zhong, Weigao Sun, Feida Zhu, Pengkun Jiao, Sihan Yang, Ming Ma,

Organizations: Nanyang Technological University · Token Foundry, Alibaba · The Chinese University of Hong Kong · Institute of Neuroscience, Chinese Academy of Sciences ·

Abstract

Humans efficiently process multimodal streams spanning hours to years, building persistent, fine-grained spatiotemporal memories that support reasoning about and predicting long-horizon events. In contrast, existing benchmarks focus on short- and medium-context tasks, and most foundation models struggle to preserve fine-grained information over long horizons. To evaluate and improve long-horizon spatiotemporal reasoning, we introduce the LongVideo dataset and the 4DMemoryBench. LongVideo comprises about 900 hours of video and over 5,000 manually reviewed questions covering visual, textual, and spatiotemporal reasoning across diverse context lengths. We further develop the 4DMemory Harness, which integrates four complementary skills, 22+ specialized tool models, and a model-specific retrieval-augmented generation system. Evaluating 13 leading models, we find that standalone foundation models struggle with long-horizon tasks, while the 4DMemory Harness improves all of them by over 50%. Compared to previous benchmark, the same models exhibit a performance drop of more than 10% on 4DMemoryBench. We hope 4DMemory will facilitate the development of spatiotemporal agents capable of addressing complex, long-horizon tasks.

Figures & tables

Figure 1: Motivation for 4DMemory Agent. (a) Traditional models with limited active context may lose access to earlier observations, leaving relevant evidence unavailable for temporal reasoning. (b) 4DMemory Agent coordinates persistent memory and task-specific skills through a harness to retrieve evidence and reason over events and their relations.

Benchmark	Corpus (h)	Temporal extent (Max)	Multi-History Scope?	Harness Support?
Video-MME ( Fu et al., 2025 )	254	1 h	✗	✗
LongVideoBench ( Wu et al., 2024 )	NR	1 h	✗	✗
HourVideo ( Chandrasegaran et al., 2024 )	381	2 h	✗	✗
LVBench ( Wang et al., 2025a )	117	68.4 min	✗	✗
InfiniBench ( Ataallah et al., 2025 )	$>1{,}000$	52.6 min	✗	✗
EgoMonth ( Chen et al., 2026 )	301	4-5 hours a	✓	✓

Table 1: Comparison with representative long-video and multimodal-memory benchmarks. Corpus hours measure total source-video duration, whereas temporal extent describes the video input or history scale, not the minimum evidence needed to answer a question. The final column summarizes documented evaluation or harness functionality, rather than whether third-party agents could be integrated.

Figure 2: Comparison of previous Video-MME (top) and LVBench (bottom) scores with 4DMemory results for the same models. All models shown score lower on 4DMemory, with dashed arrows indicating the decrease in percentage points.

Figure 3: Overview of the LongVideo construction pipeline. Hours-level videos undergo audio-text screening, clip construction, and evidence refinement to obtain coherent clips with grounded captions. Visual evidence, audio captions, and transcripts are aligned to original source timestamps and organized into chronological multimodal evidence chains. QA pairs are then constructed from multiple evidence chains and checked for answer correctness, clip necessity, and potential shortcuts.

Figure 4: Illustration of 4DMemory Agent framework. (a) The harness builds persistent, question-independent memory of events, entities, and spatial relations, preserving source timestamps for reuse across questions. (b) A foundation model retrieves timestamped evidence and iteratively refines its queries and reasoning, with optional tools provided by a skill library.

	Invocation	Ordered		Order-relaxed		Overall
Model		3 judges	Opus-5	3 judges	Opus-5	3 judges	Opus-5
Gemini 3.1-Pro	99.81	58.38	63.82	58.38	63.82	58.27	63.69
GPT-5.6 Sol	100.00	70.14	71.29	70.46	71.61	70.46	71.61
Opus-4.8	98.70	71.96	73.22	72.76	74.02	71.81	73.05
MiMo-V2.5	100.00	58.59	60.84	64.06	66.36	64.06	66.36
Grok-4.6	100.00	72.04	74.11	75.55	77.64	75.55	77.64

Table 2: The overall performance on the 4DMemory Benchmark among 13 SOTA methods.

Model	Ordering	State	Counting	Trajectory	CM reason	CM bind
Gemini 3.1-Pro	68.85	50.18	30.91	42.86	55.43	61.18
GPT-5.6 Sol	72.49	67.16	44.35	57.14	70.11	72.37
Opus-4.8	75.40	66.42	45.97	50.93	75.54	76.97
MiMo-V2.5	71.76	55.90	36.02	43.48	65.22	57.24
Grok-4.6	83.19	69.74	45.70	54.66	78.80	81.58
Muse Spark-1.1	82.02	73.80	47.58	62.73	78.80	78.95

Table 3: Six selected standalone-video spatiotemporal subtypes. Columns denote multi-event ordering, state-change tracking, distributed counting, path and trajectory tracking, cross-modal reasoning, and cross-modal binding, respectively.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: The composition details of LongVideo dataset. The inner ring shows the distribution of QA pairs across fine-grained tasks, with blue, purple, and green representing visual, textual, and spatiotemporal tasks, respectively. The outer ring shows video content categories derived from TV episode story summaries and standalone video descriptions. Semantically related low-frequency categories are merged for readability. Percentages are normalized independently within each ring over all QA pairs and all videos, respectively, with each TV episode counted as one video.

Model	Episode ( $n=172$ )	Season ( $n=50$ )	Entire series ( $n=4$ )	Video ( $n=111$ )
Gemini-3.1	70.93 / 70.35	— / —	— / —	36.04 / 34.23
GPT-5.6	100.00 / 99.42	58.00 / 56.00	75.00 / 75.00	48.65 / 45.95
Opus-4.8	95.35 / 94.77	62.00 / 60.00	— / —	49.55 / 52.25
MiMo-2.5	68.02 / 68.02	58.00 / 56.00	50.00 / 25.00	42.34 / 39.64
Grok-4.6	27.33 / 27.33	82.00 / 80.00	100.00 / 100.00	54.95 / 54.95
Muse-1.1	95.35 / 94.77	80.00 / 76.00	100.00 / 100.00	52.25 / 50.45

Appendix

Table S1: Visual-task joint accuracy by temporal scope in the September 25, 2026 snapshot. Each cell reports Opus-only / three-judge accuracy (%). Video denotes standalone videos; the other columns denote television scopes.

\dagger

marks incomplete execution or judging, and — indicates no valid judgments.

Model	Episode ( $n=215$ )	Season ( $n=126$ )	Entire series ( $n=36$ )	Video ( $n=238$ )
Gemini-3.1	99.53 / 99.53	— / —	50.00 / 50.00	59.24 / 53.36
GPT-5.6	100.00 / 100.00	96.83 / 96.03	75.00 / 72.22	71.01 / 69.33
Opus-4.8	98.14 / 98.14	85.71 / 85.71	50.00 / 50.00	72.69 / 72.69
MiMo-2.5	100.00 / 100.00	93.65 / 92.86	91.67 / 91.67	60.08 / 60.50
Grok-4.6	100.00 / 100.00	100.00 / 100.00	100.00 / 100.00	76.47 / 75.21
Muse-1.1	100.00 / 100.00	100.00 / 99.21	100.00 / 100.00	75.21 / 74.79

Appendix

Table S2: Text-task joint accuracy by temporal scope in the September 25, 2026 snapshot. Each cell reports Opus-only / three-judge accuracy (%). Video denotes standalone videos; the other columns denote television scopes.

\dagger

marks incomplete execution or judging, and — indicates no valid judgments.

Model	Episode ( $n=140$ )	Season ( $n=59$ )	Entire series ( $n=24$ )	Video ( $n=4,039$ )
Gemini-3.1	90.00 / 90.00	- / -	37.50 / 37.50	58.65 / 55.31
GPT-5.6	87.86 / 87.86	37.29 / 38.98	12.50 / 12.50	69.20 / 67.96
Opus-4.8	77.86 / 77.86	66.10 / 66.10	41.67 / 41.67	71.58 / 69.94
MiMo-2.5	87.86 / 87.86	57.63 / 55.93	79.17 / 70.83	63.88 / 61.10
Grok-4.6	91.43 / 91.43	86.44 / 86.44	100.00 / 100.00	77.57 / 74.97
Muse-1.1	100.00 / 100.00	86.44 / 86.44	100.00 / 100.00	75.96 / 74.92

Appendix

Table S3: Spatiotemporal-task joint accuracy by temporal scope in the September 25, 2026 snapshot. Each cell reports Opus-only / three-judge accuracy (%). Video denotes standalone videos; the other columns denote television scopes.

\dagger

marks incomplete execution or judging, and — indicates no valid judgments.

Model	Difference ( $n=64$ )	Ratio ( $n=108$ )	Above mean ( $n=17$ )	Max/min ( $n=5$ )	Range/total ( $n=16$ )	Top two ( $n=16$ )
Gemini-3.1	87.50	60.19	—	—	—	—
GPT-5.6	100.00	99.07	5.88	60.00	87.50	81.25
Opus-4.8	92.19	96.30	64.71	40.00	56.25	50.00
MiMo-2.5	78.12	62.04	41.18	60.00	75.00	43.75
Grok-4.6	29.69	25.93	82.35	60.00	87.50	81.25
Muse-1.1	98.44	92.59	70.59	60.00	87.50	81.25

Appendix

Table S4: Television visual subtypes, pooled across temporal scopes. Entries are three-judge joint accuracies (%). Difference and Ratio compare person instances with instances of the most frequent non-person object; Above mean counts units whose person count exceeds the mean; the remaining columns give the maximum/minimum count ratio, range divided by total count, and the share contributed by the two highest-count units.

\dagger

and — follow Table S1 .

Model	First pair ( $n=76$ )	First order ( $n=64$ )	Extrema gap ( $n=14$ )	Half diff. ( $n=16$ )	Adj. change ( $n=16$ )	Reversals ( $n=17$ )
Gemini-3.1	85.53	95.31	—	—	—	—
GPT-5.6	77.63	100.00	71.43	6.25	81.25	5.88
Opus-4.8	77.63	78.12	71.43	50.00	50.00	76.47
MiMo-2.5	82.89	93.75	78.57	62.50	62.50	23.53
Grok-4.6	85.53	98.44	100.00	75.00	81.25	94.12
Muse-1.1	100.00	100.00	100.00	75.00	81.25	94.12

Appendix

Table S5: Television temporal subtypes over visual evidence, pooled across scopes. Entries are three-judge joint accuracies (%). First pair compares the first appearances of two object types; First order orders first appearances and localizes their clips; Extrema gap measures ordinal distance between person-count extrema; Half diff. compares counts in the two temporal halves; Adj. change measures the largest adjacent-unit count change; Reversals counts changes in trend direction.

Long-Horizon Agent Evaluation

Long-Horizon Agent Evaluation measures whether AI agents can plan, act, remember, and stay reliable across extended tasks and changing environments.

See all 358 papers

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video
Jun 4, 2026Shiqiang Lang, Jing Liu, Haoyang He +6Visual Spatial Reasoning Multimodal Large Language Models
Beijing University of Posts and Telecommunications · Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences +2
EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding
May 11, 2026Ziyang Wang, Yue Zhang, Shoubin Yu +6Multimodal Memory Egocentric Video Understanding
UNC Chapel Hill · NTU Singapore
Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning
Jul 1, 2026Yixin Ji, Fanghua Ye, Juntao Li +5Long-Video Understanding Memory-Augmented Video Understanding
1Soochow University · 2Tencent Hunyuan Multimodal Department

Jun 4, 2026J/K move · Enter open · S save

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

Shiqiang Lang, Jing Liu, Haoyang He, Peiwen Sun+5

Beijing University of Posts and Telecommunications · Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences+2

Visual Spatial Reasoning Multimodal Large Language Models Temporal Video Understanding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs. Long-horizon tasks such as autonomous driving and robotic navigation require more than recognizing the current view, as models must remember and retrieve previously observed spatial layouts, routes, viewpoint changes, and object states. To evaluate this capability, we introduce LongSpace-Bench, a room-tour video benchmark for long-horizon spatial memory, covering scene perception, spatial relations, and spatial memory. In this work, we further propose LongSpace, a memory framework for long-video spatial reasoning. LongSpace models long videos as sequential chunks, incorporates 3D structural cues into early decoder layers, and constructs layer-aware memory for question-guided retrieval. Experiments on multiple spatial reasoning benchmarks show that LongSpace improves long-video spatial understanding, further demonstrating explicit spatial memory as a key capability for long-horizon video MLLMs.

Abstract

Long-Horizon Agent Evaluation

Explore similar work

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning