SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA

Xinyi Zheng; Yunze Liu; Chi-Hao Wu; Fan Zhang; Hao Zheng; Wenqi Zhou; Walterio W. Mayol-Cuevas; Junxiao Shen

cs.CVJan 21, 2026

SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA

Authors: Xinyi Zheng, Yunze Liu, Chi-Hao Wu, Fan Zhang, Hao Zheng, Wenqi Zhou, Walterio W. Mayol-Cuevas, Junxiao Shen

Organizations: University of Bristol · Memories.ai Research

Abstract

SpatialMem is a memory-based system which integrates 3D geometry, semantics, and language with one another in one combined, queryable representation. Starting with casually recorded egocentric RGB video, SpatialMem recreates metrically scaled interior scenes, recognizes structural 3D anchors (walls, doors, windows) as the first-layer scaffold, and associates open-vocabulary object nodes with evidence patches, visual embeddings, and two-layer descriptions to 3D coordinates, enabling spatial-relation reasoning, language-directed retrieval, and QA without special sensors. We test at scale on public benchmarks. On VSI-Bench ( $288$ videos, $5{,}130$ questions) SpatialMem scores $37.5$ overall with RGB alone, and its margin is precisely where a metric memory should earn it (room size $41.4$ vs.\ $24.4$ with a same-size frame model). On the episodic-memory task of OpenEQA ( $953$ questions) the memory serialised to text is lower than the blind control, but as evidence-frame retrieval it recovers frame-level accuracy, matching uniform sampling at an equal eight-frame budget and pulling ahead as the budget tightens, with every answer grounded in linked 3D evidence. On $12$ ScanNet scenes evaluated using instance-level official annotations, the anchor-first hierarchy assigns the retrieved object to the proper structural parent where open-vocabulary 3D scene graphs essentially cannot ( $0.24$ compared to $0.01$ with matched input), a capability difference instead of an overall win. The queryable index has a cost of $14$ ,KB per scene without the linked evidence frames, and we make available the suites, memories and code.

Figures & tables

Figure 1: Schematic overview of the five-stage pipeline.

Figure 2: The pipeline applied to a real capture ( scene0645_00 ).

Figure 3: Tree-structured scene memory with anchors, objects, and two-layer text.

			Metric measurement				Configurational
System	3D mem	Train- free	Cnt $\uparrow$	Abs. Dist $\uparrow$	Obj Sz $\uparrow$	Room Sz $\uparrow$	Rel Dist $\uparrow$	Rel Dir $\uparrow$	Route $\uparrow$	Order $\uparrow$	Overall $\uparrow$
Zero-shot, published for context [ 24 ]
Frequency prior	–	✓	62.1	32.0	29.9	33.1	25.1	47.9	28.4	25.2	35.5
GPT-4o	–	✓	46.2	5.3	43.8	38.2	37.0	41.3	31.5	28.5	34.0
Gemini-1.5 Pro	–	✓	56.2	30.9	64.1	43.6	51.3	46.3	36.0	34.6	45.4
LLaVA-NeXT-Video-72B	–	✓	48.9	22.8	57.4	35.3	42.4	36.7	35.0	48.6	40.9

Table 1: VSI-Bench (

288

videos,

5{,}130

questions), scored with the benchmark’s own metric code.

System	Memory use	Poses given	Frames @query	Text	ObjRec	Attr	State	Loc	Spatial	Func	World	Overall $\uparrow$
Published for context, full EM-EQA set ( $1{,}636$ q): blind GPT-4 † 35.5, CG captions † [ 7 ] 34.4, GPT-4V@50 † 48.1, human 85.9 [ 13 ]
VL-KnG [ 15 ]	graph retrieval	–	—	✓	—	—	—	—	—	—	—	55.2
GraphPad [ 1 ]	keyframe retrieval	✓	$5$	✓	—	—	—	—	—	—	—	55.3
3D-Mem [ 25 ]	snapshot retrieval	✓	$3.1$	–	—	—	—	—	—	—	—	57.2
This work: full ScanNet portion, $953$ questions / $79$ scenes, identical questions, answerer and judge
Blind GPT-4o (control)	none	–	$0$	–	32.9	40.2	57.9	34.8	41.2	58.7	47.2	44.5

Table 2: OpenEQA, episodic-memory task , LLM-Match with a GPT-4o judge.

			Relation verif.		Object retrieval				Hierarchy		Descr.	Cost
Model	Input	Poses given	all $\uparrow$	matched $\uparrow$	SR@ $0.5\uparrow$	SR@ $1.0\uparrow$	mLE [m] $\downarrow$	R@ $3\uparrow$	Wall $\uparrow$	All $\uparrow$	Colour $\uparrow$	Build s/scene $\downarrow$
			(573)	(390)	(471)	(471)	(471)	(471)	(436)	(471)	(19)
GPT-4o	8 frames	–	0.61	0.615	returns no 3D instance						0.74	—
Gemini-2.5-Flash	8 frames	–	0.60	0.623							0.79	—
Qwen2.5-VL-72B	8 frames	–	0.59	0.610							0.74	—
GPT-4o-mini	8 frames	–	0.55	0.549							0.84	—

Table 3: 12-scene ScanNet suite , all tasks scored against official instance annotations.

		ScanNet suite ( $573$ relation / $471$ retrieval, external GT)						VSI-Bench ( $5{,}130$ )
Component removed or replaced	Module	Coverage	Ver. all	Ver. answ.	SR@ $0.5$	SR@ $1.0$	$\mathrm{Acc_{parent}}$	Entity cov.	Metric	Config.
reference : full RGB memory	—	0.764	0.433	0.611	0.176	0.236	0.149	56.3	36.5	38.5
Estimated geometry (vs. given posed depth)	geometry	$-0.108$	$-0.120$	$-0.051$	$-0.087$	$-0.123$	$-0.093$	—	—	—
Scale $-10\%$	geometry	$0.000$	$-0.004$	$-0.005$	$0.000$	$0.000$	$-0.005$	—	—	—
Scale $+10\%$	geometry	$0.000$	$-0.007$	$-0.010$	$-0.013$	$-0.019$	$-0.009$	—	—	—
w/o open-vocabulary object nodes	object layer	$\mathbf{-0.573}$	$\mathbf{-0.354}$	$+0.006$	$-0.149$	$-0.193$	$-0.115$	$\mathbf{-25.5}$	$-1.8$	$-4.3$

Table 4: Contribution of each memory layer , reported as the change each removal causes.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: High-level SpatialMem pipeline. Left to right: egocentric RGB input, geometry and metric alignment, anchor and object construction, memory tree and query execution using geometry and semantic reasoning.

Figure 5: The geometry back-end and metric alignment. We use a feed-forward RGB geometry module and align it to an upright, metric frame.

Figure 6: The SpatialMem pipeline overview. The memory tree (centre) orders structural anchors, objects and the contextual relations. On the left side: the offline guidance can be executed with the help of the anchor graph to establish user and goal locations and provide step-by-step graph guidance. On the right side: object retrieval processes a query in language, searches the tree, and returns an answer with 3D anchors and linked 2D evidence.

Component	Setting	Value / rule
Frame sampling	video sampling stride	10
Floor fitting	RANSAC inlier threshold	2–5 cm
Metric scaling	height prior	fixed $2.5$ m ceiling prior (no per-scene metadata)
3D box fitting	box type	axis-aligned bounding box (AABB)
Multi-view object association	grouping rule	center-distance matching
Duplicate suppression	merge range	approximately 25 cm

Appendix

Table 5: Shared implementation settings used in SpatialMem.

Figure 7: Relational query traversal using the memory tree: The parsed query graph is solved against Tier-1 anchors and Tier-2 objects, candidates are filtered by geometric predicates and description agreement and the result is constructed with connected 3D anchors and 2D evidence.

Scene	$\mathrm{Acc}_{\text{rel}}$	$\mathrm{SR}_{\text{obj}}$	$\mathrm{Acc}_{\text{parent}}$	$\mathrm{Acc}_{\text{path}}$
HT (hotel room)	0.96	1.00	0.83	0.83
Apt (multi-room)	0.99	1.00	0.79	0.79
Replica (public)	1.00	1.00	1.00	1.00

Appendix

Table 6: Per-scene results of SpatialMem (human-verified GT,

95\%

CI). Relation / parent / path have been limited by reconstruction (answered by geometry); VLM comparison has been grouped into the main paper because it contains small per-scene samples.

Task	resolved	not resolved	% resolved
Room size	41.4 [37.4,45.3]	–	100
Counting	33.7 [30.0,37.7]	0.0	74
Object size	52.3 [48.9,55.9]	51.9 [48.5,55.5]	48
Absolute distance	25.1 [20.8,29.3]	28.6 [25.6,31.3]	30

Appendix

Table 7: The VSI-Bench metric tasks split by whether the queried entities resolved to memory objects (

95\%

CI). Only room size is answered from stored geometry throughout; absolute distance mostly is not, and where it is, the geometry does not beat the estimate. That is why we do not rest the argument on that task.

System	matched (390)	all episodes (573)
GPT-4o	$0.615$ $[0.564,0.664]$	$0.613$
Gemini-2.5-Flash	$0.623$ $[0.574,0.669]$	$0.604$
Qwen2.5-VL-72B	$0.610$ $[0.559,0.659]$	$0.586$
GPT-4o-mini	$0.549$ $[0.495,0.600]$	$0.548$
ConceptGraphs	$0.623$ $[0.569,0.669]$	$0.611$
SpatialMem-RGBD	$\mathbf{0.667}$ $[0.621,0.718]$	$0.553$

Appendix

Table 8: Relation verification on the matched set (n=

390

;

95\%

bootstrap CI) beside the all-episode value (n=

573

). The lead our matched-input memory holds on the matched set is nominal: its confidence interval overlaps the intervals of the scene graph and the most powerful frame-based models.

Task	Full	w/o obj. nodes	$\Delta$	quest. resolved
Object count	24.9	13.9	$+11.0$	74%
Absolute distance	27.5	28.9	$-1.4$	30%
Object size	52.1	54.5	$-2.4$	55%
Room size	41.4	41.4	$\pm 0.0$	100%
Relative distance	36.1	32.5	$+3.6$	8%
Relative direction	44.5	44.5	$\pm 0.0$	14%

Appendix

Table 9: Object layer ablated on all

5{,}130

VSI-Bench questions.

Variant	Ver. all	Ver. answ.	SR@ $0.5$	SR@ $1.0$	$\mathrm{Acc}_{\text{parent}}$
Full, RGB memory (reference)	0.433	0.611	0.176	0.236	0.149
Full, matched-input memory	0.553	0.662	0.263	0.359	0.242
w/o open-vocabulary object nodes	0.079	0.616	0.028	0.042	0.034
w/o wall anchors	0.433	0.611	0.176	0.236	0.019
w/o structural layer (flat)	0.433	0.611	0.176	0.236	0.000
w/o relational hint in retrieval	0.433	0.611	0.174	0.238	0.157

Appendix

Table 10: Absolute suite scores for every ablation variant (

n{=}573

relation,

n{=}471

retrieval).

Figure 8: Step-by-step walkthrough of the navigation-style guidance demonstration: a command is parsed into an intent graph, resolved over the memory tree, planned with the help of an anchor-derived room/cell graph, and expressed as step-wise guidance. The given task is qualitative only; no quantitative statement uses these metrics.

Figure 9: Summary on the three egocentric/Replica captures (not the public benchmarks). Left: radar over relative-position reasoning and guidance; each axis runs from $0$ at the centre to $1$ at the rim. Right: query time vs. success rate. Since these ground-truth episodes are suggested by the system and then human-checked, the figure is merely illustrative and does not support any head-to-head claim; all quantitative claims are in Tables $1$ – $3$ of the main paper.

	Provenance			Ground truth				Cost
Evaluation	Scene source	Scenes built	Questions / episodes	Origin	Chance / prior	Indep. of recon.	Human verified	Metrics reported	Build / scene (median)
VSI-Bench [ 24 ] (Tab. 1 )	ScanNet, ScanNet++, ARKitScenes	$287/288$	$5{,}130$	benchmark’s released answers	$35.5$ freq. prior	✓	benchmark	$8$ task scores, macro mean	$118$ s
OpenEQA [ 13 ] EM-EQA (Tab. 2 )	ScanNet portion	$79/89$	$953$	human-written answers	$33.5$ blind LLM	✓	benchmark	LLM-Match, $7$ categories	$118$ s
ScanNet suite (Tab. 3 ), released	ScanNet, $12$ scenes	$12/12$	$573$ relation, $471$ retrieval, $387$ description	official aggregation.json + released instance boxes	$0.500$ balanced	✓	$504$ episodes, $98.6\%$ confirmed	Ver. all / matched, $\mathrm{SR}_{\text{obj}}$ @ $0.5$ / $1.0$ , $\mathrm{Acc}_{\text{parent}}$ , Colour	$118$ s RGB, $89$ s RGB-D
Real egocentric probe set (supp., qualitative), released	$2$ self-captured + Replica	$3/3$	$174$ verified	system-proposed, human-verified	—	–	$174$ episodes	$\mathrm{Acc}_{\text{rel}}$ , $\mathrm{Acc}_{\text{path}}$ , Loc@parent/path, guidance metrics (qualitative)	—

Appendix

Table 11: Each assessment in this paper: provenance, scale, ground truth, metrics, and construction cost. Independent marks ground truth that does not originate in any system’s reconstruction. Construction is done only once and is offline; the index column is the per-scene footprint a query touches, and each quantitative claim in Sec. 5 of the main paper is calculated using a metric listed here. The

12

suite scene identifiers and the per-corpus question lists are available in the Supplementary Material.

	Input		Representation
Method	Sensor	Poses given	Persistent 3D	Metric	Anchor- first	Open-vocab objects	Language descr.	Train- free	Index / scene	Evaluated on
Frame-sampling and instruction-tuned VLMs
GPT-4o, Gemini, Qwen2.5-VL	RGB video	–	–	–	–	✓	✓	✓	re-reads video	VSI-Bench, OpenEQA
VLM-3R [ 5 ]	RGB video	–	$\circ$ implicit	$\circ$ implicit	–	✓	✓	–	in weights	VSI-Bench
Open-vocabulary 3D maps
OpenScene [ 18 ]	posed RGB-D + cloud	✓	✓	✓	–	✓	–	–	dense features	ScanNet, Matterport3D

Appendix

Table 12: What persistent-memory systems store, and what they must be given to store it. ✓ present, – absent,

\circ

partial. Poses given marks methods taking as input ground-truth camera poses, depth and intrinsics. Metric marks a memory in which the distances and extents can be retrieved in metres. Anchor-first marks a hierarchy based on structural features (walls, doors, windows) instead of on segmented regions. Train-free marks methods which do not involve task-specific 3D or spatial training on top of off-the-shelf perception and language models. Index size refers to the per-scene footprint that a query needs to hit.

Task	SpatialMem	w/o open-vocab. object nodes	LLaVA-NeXT-Video-7B, re-run
Object count	$24.9$ $[21.9,27.9]$	$13.9$ $[11.5,16.4]$	$48.6$ $[45.2,52.0]$
Absolute distance	$27.5$ $[25.1,29.8]$	$28.9$ $[26.4,31.2]$	$13.9$ $[12.1,15.7]$
Object size	$52.1$ $[49.8,54.4]$	$54.5$ $[52.4,57.0]$	$47.6$ $[45.4,49.7]$
Room size	$41.4$ $[37.4,45.3]$	$41.4$ $[37.4,45.6]$	$24.4$ $[20.5,28.4]$
Relative distance	$36.1$ $[32.2,39.4]$	$32.5$ $[29.4,36.2]$	$43.9$ $[40.0,47.8]$
Relative direction	$44.5$ $[41.4,47.8]$	$44.5$ $[41.4,47.6]$	$42.6$ $[39.3,45.8]$

Appendix

Table 13: Bootstrap

95\%

confidence intervals per task (

1{,}000

resamples, seed

42

) of the three rows in Tab. 1 that we ran ourselves. Resampling is within task, so the overall interval propagates each task’s uncertainty through the macro mean.

System	Relation (all)	Relation (answerable)	$\mathrm{SR}_{\text{obj}}$	$\mathrm{Acc}_{\text{parent}}$
ConceptGraphs	$0.611$ $[0.572,0.651]$	$0.611$ $[0.572,0.651]$	$0.412$ $[0.369,0.456]$	$0.011$ $[0.002,0.021]$
SpatialMem-RGBD	$0.553$ $[0.510,0.595]$	$0.662$ $[0.620,0.706]$	$0.263$ $[0.223,0.306]$	$0.242$ $[0.206,0.285]$
SpatialMem	$0.433$ $[0.393,0.473]$	$0.611$ $[0.564,0.658]$	$0.176$ $[0.144,0.214]$	$0.149$ $[0.119,0.183]$
GPT-4o	$0.613$ $[0.572,0.653]$	$0.613$ $[0.572,0.653]$	–	–
Gemini-2.5-Flash	$0.604$ $[0.564,0.646]$	$0.604$ $[0.564,0.646]$	–	–
Qwen2.5-VL-72B	$0.586$ $[0.545,0.628]$	$0.586$ $[0.545,0.628]$	–	–

Appendix

Table 14: The confidence intervals of Tab. 3 (n=

573

relation, n=

471

retrieval episodes after the human audit;

95\%

bootstrap,

1{,}000

resamples, seed

42

). The two relation columns have a common denominator; the answerable column eliminates the episodes where the queried entity cannot be found in the memory, hence it is only defined on our two rows and is equal to the all-episode value in systems that do not have the ability to abstain.

System	described objs.	all audited
	( $n{=}19$ )	( $n{=}175$ )
Ours, Layer-1 (per-view)	0.68 [0.47,0.90]	–
Ours, Layer-2 (consolidated)	0.42 [0.21,0.63]	–
GPT-4o	0.74 [0.53,0.90]	0.56 [0.49,0.63]
Gemini-2.5-Flash	0.79 [0.58,0.95]	0.59 [0.52,0.66]
Qwen2.5-VL-72B	0.74 [0.53,0.90]	0.69 [0.63,0.76]

Appendix

Table 15: Colour recall in comparison with human labels,

95\%

bootstrap intervals. Unlike the five-object measurement on our egocentric captures, the frame-based models recover colour attributes better than our description layers on these scenes, and the cross-view consolidation does not improve on the raw per-view captions. We report this rather than relying on the smaller sample. The intervals are wide and overlapping at n=19, and the consolidation figure is obtained through a re-implementation of the consolidation prompt.

Model	Verify $\uparrow$	Consist $\uparrow$	SR ${}_{\text{obj}}\uparrow$	Color $\uparrow$
GPT-4o	0.50	0.12	1.00	0.13
Gemini-2.5-Flash	0.50	0.16	0.95	0.37
Qwen2.5-VL-72B	0.53	0.60	0.91	0.13
GPT-4o-mini	0.53	0.44	0.95	0.27
Qwen2-VL-72B	0.53	0.64	–	–
SpatialMem (Ours)	0.99	1.00	1.00	0.80

Appendix

Table 16: Spatial understanding vs. five frontier VLMs (pooled over the two egocentric captures and the public Replica room;

95\%

CI). Verify: balanced yes/no on viewpoint-invariant relations (chance

0.5

); Consist: cross-view consistency on lateral relations; SR

{}_{\text{obj}}

: object-presence detection; Color: colour-attribute recall. SpatialMem’s relation accuracy is reconstruction-limited (answered from geometry); baselines answer from frames. “-”: free-text answers not reliably parsed.

Memory-Augmented Video Understanding

Memory-augmented video understanding helps models retain and retrieve useful visual history for reasoning over long videos and live streams.

See all 85 papers

May 26, 2026

Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning

Video spatial reasoning requires accumulating viewpoint-dependent evidence over time while retaining information useful to the question being asked. Existing spatial video-language models improve geometric perception and long-range context modeling, but often treat memory as a generic temporal cache, which can introduce redundant or irrelevant evidence and weaken long-horizon reasoning. We propose Q-GeoMem, a question-guided geometric memory framework for video spatial reasoning. Q-GeoMem injects camera-conditioned geometry into visual tokens and maintains two complementary memories: a Fine-Grained Context Bank for recent dense features and camera states, and a Semantic-Geometric Evidence Bank for compact long-range evidence. For each candidate frame, a calibrated Q-Former estimates question relevance, while novelty and evidence utility are recomputed with respect to the active evidence bank. The resulting relevance-novelty utility controls capacity-based replacement and serves as an attention bias during memory reading. During reasoning, both memories are read before update and adaptively fused with the current frame representation. Extensive experiments across two in-domain and five out-of-distribution benchmarks, and controlled memory analyses show that Q-GeoMem achieves state-of-the-art performance in the evaluated settings and validate the effectiveness of question-guided geometric evidence selection.

Xianqiang Gao, Qizhi Chen, Delin Qu +5

University of Science and Technology of China · Shanghai AI Lab · Northwestern Polytechnical University +1

3D Spatial Reasoning Video Reasoning Memory-Augmented Video Understanding

Ask PDF

Oct 4, 2026

Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants

Egocentric assistants must connect what users say with what they see across long interaction histories. We formalize this challenge as Spatially grounded Conversational Reasoning (SpaCR): cross-scene, recall-oriented, and counterfactual spatial queries that combine user-stated facts with geometric evidence. Direct vision-language models incur high inference costs and context limits as histories grow, while keyframe selection and retrieval can omit objects or evidence needed for complete recall. We propose Spatially grounded Conversational Memory (SpaC-MEM), an object-centric working memory that uses 3D reconstruction and segmentation to ground conversational information in persistent physical objects. It compresses multimodal histories while preserving spatial evidence and allowing object-specific facts to be updated through dialogue. We also introduce Ego-SpaCR, a benchmark comprising 620 ScanNet video sessions augmented with 95 task-oriented conversations and 3,100 evaluation queries. SpaC-MEM achieves the highest overall answer accuracy among the evaluated methods and improves object recall while requiring substantially fewer reference input tokens than native-video baselines. Removing 3D spatial information substantially degrades performance, highlighting the importance of preserving spatial and conversational evidence together.

Jiazhou Liang, Liam Gallagher, Kiko Chen +4

University of Toronto

Conversational Memory Egocentric Video Understanding 3D Visual Grounding Multimodal Spatial Reasoning

Ask PDF

Aug 18, 2026

Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering

Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D

Hsiang-Wei Huang, Fu-Chen Chen, Li-Wu Tsao +7

University of Washington, United States · Amazon, United States · The University of Texas at Austin, United States

Embodied QA 3D VQA Query-Aware Video Frame Selection Memory-Augmented Video Understanding Open-Domain QA

Ask PDF

Abstract

Memory-Augmented Video Understanding

Explore similar work

Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning

Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants

Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering