Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos

Shravan Chaudhari; William Paul; Suchi Saria; Rama Chellappa; Homanga Bharadhwaj

cs.CVOct 7, 2026

Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos

Authors: Shravan Chaudhari, William Paul, Suchi Saria, Rama Chellappa, Homanga Bharadhwaj

Organizations: Johns Hopkins University · Johns Hopkins University Applied Physics Laboratory

Abstract

As we move through the world and carry out everyday tasks, we encounter objects that may become relevant only later. We are capable of recalling where we left something or what was inside a container, even without knowing we would need it later. Here, we study how an embodied assistant can build a similar memory from egocentric videos, by observing a person's day-to-day activities. We present Ledger, a persistent 3D object memory that combines object locations, their histories, and contextual descriptions. It associates observations across the recording and retains objects after they leave the view, including those the person never touches. It clusters each object's observations by resting locations and records a move only after repeated evidence, reducing the effect of localization noise. Short descriptions preserve details such as an object's contents or supporting surface. It saves these records to later answer spatial questions without having to access the original images or video. Our memory raises HD-EPIC accuracy from 29.7% to 42.6%, UCS-Bench accuracy from 33.8% to 38.5% and localizes Ego4D objects with a 0.99 m median error on returned predictions. Our analyses identify complementary roles for temporal persistence, contextual descriptions, and retrieval. Our study on 100 stitched streams of multiple scenes each further exposes failures in both retrieval and construction. Per-scene construction partially recovers the performance lost across scene changes compared to that of single scene streams.

Figures & tables

Panel of Figure 1 — Figure 1: Persistent object memory from egocentric video. Top: Ledger ’s memory 2 min 24 s into a scene in the kitchen: tracked objects (bubbles) persist after leaving view, arcs show recorded moves (bold: the tongs), and the white frame is the wearer’s camera, with its current view inset; the point cloud is for display only. Bottom: \scriptsize1⃝ A wearer or an assistant (physical or digital) works for hours across scenes while questions arrive. \scriptsize2⃝ It keeps a record for each object, with its 3D position, times seen, and a short description, on a map of its path. \scriptsize3⃝ Later questions are answered from these records alone, without replaying the video.

	Memory content				Construction conditions			Query time
Method	Handled objects	Untouched objects	Motion history	Path & places	RGB only	Scene changes	$\geq$ 20-min video	No frames at query	Language queries	3D location
ConceptGraphs [ 18 ]	✗	✓	✗	✗	✗	✗	N.A.	✓	✓	✓
3D-Mem [ 57 ]	✗	✓	✗	✗	✗	✗	N.A.	✗	✓	✓
ReMEmbR [ 2 ]	✗	✗	✗	✓	✗	N.A.	✓	✓	✓	$\circ$
VideoAgent [ 14 ]	$\circ$	✓	✗	✗	✓	✗	✗	✗	✓	✗
AMEGO [ 16 ]	✓	✗	$\circ$	✓	✓	✗	✓	✓	✗	✗

Table 1: What existing memories built from egocentric or embodied video keep, and what they need at query time. ✓ : supported and demonstrated;

\circ

: partly or only under conditions; ✗ : not supported; N.A.: not applicable. The memories differ in their objects’ coverage, tracking of motion history, input conditions for construction or handling of scene changes, and query time support for no frame access, language queries and 3D location requests.

Figure 2: Ledger overview. (a) Objects detected in sampled egocentric frames are localized in 3D and placed in a scene coordinate frame using supplied or estimated camera poses. (b) Persistent object records, formed from associated observations, contain timestamped positions, rest segments defined by repeated evidence of displacement, and contextual descriptions. (c) After query grounding and retrieval, a language model answers from the records rendered as text, without receiving images. When pose scale is available, scene coordinates are metric.

Figure 3: Three memories answer the same questions from their stored records alone, without frames. Q1–Q4 come from one UCS-Bench stream that joins three EPIC-Kitchens recordings; Q5 asks for a chair’s position in an Ego4D VQ3D clip, with each memory’s answer on the top-down map. Bottom: what each memory retrieves for Q2. Questions were chosen to show how the memories differ; Figure 8 (Appendix G.2 ) adds the video timeline.

Memories rebuilt on our frames, read by GPT-5.4
Method	Video at answer	3D Perception	Object Motion	Mean
LongVA [ 60 ]	yes	32.9	22.7	27.8
2025 challenge, 1st place [ 42 ]	yes	42.6	30.2	36.4
EgoAdapt, 2026 [ 8 ]	yes	64.9	61.6	63.3
Human (sample) [ 35 ]	yes	93.8	92.7	93.3
DirectMe, graph only [ 49 ]	no	30.6	30.8	30.7

Table 2: HD-EPIC accuracy (%) on all 2,400 3D Perception and Object Motion questions, prototype-averaged as in Perrett et al. [35] ; chance is 20.0. yes : video at answer time; no : text alone. Published rows use other answerers and protocols, and the 2026 entry is a challenge report (Appendix B.3 ). Ledger

{}_{\text{agent}}

: our memory read by ReMEmbR’s querying agent ( 4 ). Bold: best per evidence group, except the human reference; 95% intervals in Table 6 ; other answerers in Figure 4 b.

Figure 4: What each component carries, and what the answering model adds. (a) HD-EPIC: accuracy lost on each prototype when one component is removed ( † survives Holm correction, ∗ raw $p<0.05$ ; single runs, cells $\pm$ 7–10 points). (b) Accuracy without and with the memory for each answering model; GPT-5.6 answered with reasoning enabled, the others without. The UCS-Bench component and reader analyses are in Figure 7 .

				Memory only		Frames + memory
Category	$n$	Blind	Frames	DM	Ledger	DM, own	DM, routed	Ledger
Trajectory & Movement	648	34.9	45.4	37.3	38.6	41.8	45.5	49.1
Position & Orientation	701	23.8	39.7	26.2	33.4	37.4	37.4	43.9
Proximity & Reachability	538	45.4	51.7	40.0	49.3	47.2	48.5	51.3
Category & Quantity	884	33.9	42.9	21.4	36.0	35.6	42.0	43.1
Overall	2771	33.8	44.4	30.0	38.5	39.8	42.9	46.3

Table 3: UCS-Bench accuracy (%) on 2,771 timestamped questions; 48 frames where frames are given; chance 25.7. DM is DirectMe with its released graph builder and retrieval: “own” adds its graph to the frames on every question, as DirectMe does, and “routed” only on the spatial questions where we add our memory. Bold: best per row among the green and among the red arms. Subcategories and paired tests are in Appendix D.1 .

Memory	Graded / 164	Median $L_{2}$ (m)	Mean $L_{2}$ (m)	$<1$ m (%)	$<2$ m (%)
ReMEmbR (wearer position)	164	1.49	1.77	23	69
DirectMe (graph only)	164	1.91	1.98	18	52
OSNOM (adapted)	146	1.33	1.65	35	71
No memory: clip centroid	164	1.69	1.82	10	63
Ledger , YOLO-World	148	1.06	1.46	49	74
Ledger , SAM 3	153	0.99	1.41	51	76

Table 4: VQ3D validation split: memories built by each method on the same 44 clips and read by the same GPT-5.4 matcher from the object’s name; errors in metres over graded queries (Appendix C.6 ). AMEGO stores neither names nor 3D positions. ReMEmbR answers with the wearer’s position; OSNOM uses our detections and depth; DirectMe is read without frames. Our rows name the detector and use trajectory memory and EgoLoc poses, without SAM 3 consolidation.

Figure 5: Gain over the blind control by recording length, on the same questions in each bin (question-level accuracy, GPT-5.4). The band is the 95% interval of Ledger ’s gain. (a) HD-EPIC, 2,400 questions on 150 videos. (b) UCS-Bench, 3,264 questions on 206 single recordings; AMEGO and OSNOM cannot answer these. Per-bin tables and trend tests are in Appendix F .

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

			Qwen3.5-9B		GPT-5.4
Category	Prototype	$n$	Blind	Memory	Blind	Memory
3D Perception	object location	500	25.4	45.6	38.6	48.0
	contents retrieval	200	16.0	28.5	16.5	33.5
	fixture location	500	16.2	52.6	13.8	56.0
	fixture interaction counting	300	36.0	40.3	40.0	41.3
Object Motion	movement itinerary	500	11.8	26.2	34.8	38.6

Appendix

Table 5: HD-EPIC accuracy (%) on all seven evaluated prototypes. Memory scores are bold when they exceed the matched blind control.

Figure 6: Accuracy by question type. Left: HD-EPIC by prototype; published video models as printed in Xu et al. [53] (EgoAdapt reports no per-prototype scores). Right: UCS-Bench by category on the 2,771 questions of Table 3 .

Answerer	Evidence	3D Perception	Object Motion	Mean
Qwen3.5-9B	Blind	23.4 [21.2,25.7]	22.1 [19.0,25.1]	22.8 [20.9,24.7]
Qwen3.5-9B	Memory	41.8 [39.3,44.4]	25.4 [22.3,28.4]	33.6 [31.6,35.5]
GPT-5.4	Blind	27.2 [25.0,29.6]	32.1 [28.9,35.4]	29.7 [27.7,31.7]
GPT-5.4	Memory	44.7 [42.0,47.3]	40.5 [37.2,44.3]	42.6 [40.5,44.8]

Appendix

Table 6: Prototype-averaged HD-EPIC accuracy (%) with 95% bootstrap intervals (2,000 resamples stratified by prototype, seed 0). Memory improves mean accuracy over the matched blind control by 10.8 points

[8.2,13.3]

for Qwen3.5-9B and 13.0 points

[10.4,15.5]

for GPT-5.4.

Stage	Parameter	Value
Sampling	interval / cap	4 s / 1,200 frames
Rectification	pinhole resolution	$1802\times 1802$ px
Tagging	model / token cap / vocabulary cap	Qwen3.5-9B / 130 / 80
Detection	model	YOLO-World v8x-worldv2
	input / confidence / top- $k$	1408 px / 0.05 / 40
3D lifting	per-frame / per-video / per-tag budget	12 / $\geq 400$ / $\max(25,N/4)$

Appendix

Table 7: HD-EPIC construction and answering settings.

Model	Images to answerer	3D Perception	Object Motion
Llama 3.2, text only [ 35 ]	no	22.3	25.5
Gemini 1.5 Pro, text only [ 35 ]	no	21.5	27.7
VideoLLaMA 2 [ 11 ]	yes	25.7	28.5
LongVA [ 60 ]	yes	32.9	22.7
LLaVA-Video [ 61 ]	yes	27.3	18.9
Gemini 1.5 Pro [ 44 ]	yes	32.5	20.8

Appendix

Table 8: Published HD-EPIC category scores and our prototype-averaged scores (%). Published baselines were not rerun; models, visual inputs, and inference budgets differ. The team labels used in the challenge reports are inconsistent between papers, so rows are identified by paper: the 40.9 / 29.9 entry is the method of Yang et al. [56] (2nd place 2025), and the 42.6 / 30.2 entry is the scene- and knowledge-graph system of Taluzzi et al. [42] (1st place 2025, 44.21 overall).

Prototype	$n$	Video- LLaMA 2	LLaVA- Video	Gemini 1.5 Pro	2025 2nd	Xu et al. 2026	GPT blind	Ours + Qwen	Ours + GPT
Fixture interaction counting	300	17.7	16.3	35.3	29.0	46.0	40.0	40.3	41.3
Fixture location	500	18.8	21.8	20.8	34.2	48.2	13.8	52.6	56.0
Object location	500	31.0	30.6	32.4	49.8	64.0	38.6	45.6	48.0
Object contents retrieval	200	35.5	40.5	41.5	50.5	58.5	16.5	28.5	33.5
Movement itinerary	500	11.0	9.8	18.0	14.2	49.0	34.8	26.2	38.6
Movement counting	200	44.0	20.0	13.0	44.5	51.0	31.0	35.5	45.0

Appendix

Table 9: HD-EPIC accuracy per prototype (%) for the published entries that report it (as printed in Xu et al. [53] ) and for our memory. Bold marks the best score per prototype. EgoAdapt reports category means only.

Prototype	$n$	Blind	DirectMe	ReMEmbR	OSNOM	AMEGO	Ours
Fixture location	500	14	26	21	44	13	56
Object location	500	39	37	39	42	40	48
Object contents retrieval	200	16	20	30	21	22	34
Fixture interaction counting	300	40	39	25	35	47	41
Movement itinerary	500	35	16	14	19	27	39
Movement counting	200	31	37	32	16	34	45

Appendix

Table 10: HD-EPIC accuracy per prototype (%, rounded) for the memories rebuilt on our setup. Bold marks the best memory per prototype.

Prototype	$n$	Full	Blind	$-$ Descriptions	$-$ Persistence	$-$ Triangulation
3D Perception
Fixture location	447	56.4	12.8	$+0.7$	$+1.6$	$+2.5$
Object location	500	48.0	38.6	$+8.4^{\dagger}$	$-2.0$	$-0.4$
Object contents retrieval	200	33.5	16.5	$+8.0^{*}$	$+1.0$	$+3.5$
Fixture interaction counting	270	42.2	39.6	$+7.8^{\dagger}$	$+1.5$	$+1.5$
Object Motion

Appendix

Table 11: Accuracy lost on each HD-EPIC prototype when one component is removed from the memory (full

-

ablated, points; 2,317 questions from 133 videos). † Significant after Holm correction over the 21 tests; ∗ raw

p<0.05

. One run per arm, so single cells vary by about

\pm

7–10 points.

Component removed	Without	With	Gain (points)
Persistence segmentation	37.3	42.8	$+5.5$ [+3.1,+7.9]
Segment descriptions	37.9	42.8	$+4.9$ [+2.9,+6.9]
Multi-view triangulation	40.5	42.8	$+2.2$ [+0.2,+4.4]

Appendix

Table 12: HD-EPIC component ablations at the category level (questions and runs of Table 11 ; prototype-averaged mean of the two categories). Differences are paired, computed before rounding, with 95% stratified bootstrap intervals.

			3D Perception				Object Motion
Change from the final system	Mean	$\Delta$	obj. loc.	contents	fixt. loc.	fixt. count	itinerary	mov. count	stationary
Final system (GPT-5.4)	42.6	–	48.0	33.5	56.0	41.3	38.6	45.0	38.0
Answering model: GPT-5.6	45.0	$+2.4^{*}$	$+0.4$	$+7.0^{*}$	$+2.2$	$+0.7$	$+5.2^{*}$	$+0.5$	$+1.0$
no memory, GPT-5.4	29.7	$-13.0^{*}$	$-9.4^{*}$	$-17.0^{*}$	$-42.2^{*}$	$-1.3$	$-3.8$	$-14.0^{*}$	$-7.5$
no memory, GPT-5.6	27.2	$-15.4^{*}$	$-20.0^{*}$	$-14.5^{*}$	$-36.8^{*}$	$-1.0$	$-21.0^{*}$	$-25.5^{*}$	$+8.5$
Lift depth: FastVGGT	44.7	$+2.1$	$+2.2$	$-5.0$	$+5.2^{*}$	$-1.7$	$+7.8^{*}$	$+1.5$	$+2.5$

Appendix

Table 13: HD-EPIC design ablations: one change from the final system at a time, all 2,400 questions, prototype-averaged accuracy.

\Delta

is against the final system; the prototype columns give the change on each prototype. Markers and single-run noise as in Table 11 , with Holm correction over this table’s tests. § See the text for the pose rescaling.

Retrieval	Qwen3.5-9B	GPT-5.4
Blind control	19.5	–
3D centre distance	29.2	36.6
2D area among candidates within 1 m	33.9	36.6
Union of both sets	30.4	36.3
3D centre, triangulated positions	–	40.4

Appendix

Table 14: Earlier 339-question retrieval analysis (accuracy, %).

Evidence	Accuracy (%)
Predicted memory, triangulated positions	40.4
Ground-truth queried-object trajectory, positions only	52.2
Ground-truth trajectory and supporting-fixture names	77.6
Any of four single-candidate answers, retrieved objects	65.5
Any of four single-candidate answers, non-retrieved objects	64.9

Appendix

Table 15: Earlier 339-question oracle analysis. Ground-truth supporting-fixture labels add information beyond correcting coordinates.

Rule	Mean count	Exact	Within $\pm 1$	Bias
Threshold, $\tau=0.3$ m	4.9	13.5%	36.0%	+2.3
Threshold, $\tau=0.8$ m	2.3	22.5%	57.3%	$-0.3$
Persistence, $\tau=0.3$ m, $k=3$	2.2	28.1%	67.4%	$-0.3$

Appendix

Table 16: Earlier movement-count analysis on 89 questions whose queried object is in memory (ground-truth mean count: 2.5).

Detector	Memory	Poses	Median	Mean	$<1$ m	$<2$ m	CS
SAM 3	Raw	EgoLoc	0.877	1.399	53	77	99
	Raw	FastVGGT	1.064	1.504	45	76	99
	Trajectory	EgoLoc	0.984	1.388	52	78	99
	Trajectory	FastVGGT	1.057	1.449	45	77	99
	Consolidated	EgoLoc	0.955	1.360	53	78	99
	Consolidated	FastVGGT	1.057	1.421	45	76	99

Appendix

Table 17: VQ3D localization on returned candidates. Error is in metres. Fractions and conditional success (CS) are percentages. Different rows may grade different queries. “Consolidated” adds mask tracking to trajectory memory.

Detector / level	Poses	$n$	$\Delta$ median [95% CI]	$p$	$<$ 0.5 m (%)	Mean $L_{2}$ , all (m)
SAM 3 / raw	EgoLoc	71	$-0.09$ $[-0.18,-0.01]$	0.002	24 $\to$ 31	1.38 $\to$ 1.34
SAM 3 / raw	FastVGGT	67	$-0.03$ $[-0.12,+0.01]$	0.029	22 $\to$ 28	1.53 $\to$ 1.49
SAM 3 / trajectory	EgoLoc	76	$-0.11$ $[-0.18,-0.03]$	0.004	21 $\to$ 32	1.46 $\to$ 1.42
SAM 3 / trajectory	FastVGGT	67	$-0.03$ $[-0.10,+0.02]$	0.064	24 $\to$ 30	1.54 $\to$ 1.51
SAM 3 / tracker	EgoLoc	70	$-0.12$ $[-0.19,-0.03]$	0.003	27 $\to$ 37	1.36 $\to$ 1.32
SAM 3 / tracker	FastVGGT	69	$-0.03$ $[-0.10,+0.02]$	0.112	25 $\to$ 32	1.43 $\to$ 1.40

Appendix

Table 18: VQ3D, triangulated against monocular positions, paired on the same retrieved object.

n

: queries whose object passes the triangulation guards;

\Delta

: median per-query change in

L_{2}

(m; negative is closer) with bootstrap 95% interval;

p

: Wilcoxon signed-rank. The last two columns cover all graded queries, where objects that fail the guards keep the monocular position.

Method	Succ (%)	Succ ∗ (%)	$L_{2}$ (m)	Angle	QwP (%)
Ego4D baseline [ 17 ]	1.22	30.77	5.98	1.60	1.83
Ego4D ∗ , EgoLoc poses [ 30 ]	73.78	91.45	2.05	0.82	80.49
EgoLoc [ 30 ]	80.49	98.14	1.45	0.61	82.32
EgoLoc-v1 [ 31 ]	81.13	98.10	1.45	0.55	84.73
EAGLE [ 4 ]	84.77	98.54	1.18	0.42	85.68
Embodied VideoAgent [ 15 ]	85.37	92.72	1.86	n/a	92.07

Appendix

Table 19: VQ3D validation split (164 queries, 44 clips), published methods and ours. Benchmark metrics as in Mai et al. [30] : Succ over all queries, Succ ∗ over queries with a prediction,

L_{2}

in metres, angular error in radians, QwP = queries with poses. Published rows are as reported. Our rows answer a text query from the memory, not the benchmark’s image crop.

Memory	Graded	Succ	Succ ∗	Median	Mean	$<1$ m	$<2$ m
AMEGO	N.C.	N.C.	N.C.	N.C.	N.C.	N.C.	N.C.
ReMEmbR (wearer position)	164	100.0	100.0	1.49	1.77	23	69
DirectMe (graph only)	164	99.4	99.4	1.91	1.98	18	52
OSNOM (adapted)	146	89.0	100.0	1.33	1.65	35	71
No memory: clip centroid	164	100.0	100.0	1.69	1.82	10	63
SAM 3 / raw / EgoLoc	143	86.6	99.3	0.95	1.41	52	77

Appendix

Table 20: All memories through one harness (164 validation queries). Succ and Succ ∗ as in Table 19 . The no-memory reference passes every success check, so

L_{2}

and the sub-metre share are the informative columns.

Memory	Median $L_{2}$	Mean $L_{2}$	$<1$ m (%)	$<2$ m (%)
Ledger (SAM 3, trajectory, EgoLoc)	1.00	1.44	50	76
Ledger (SAM 3, trajectory, FastVGGT)	1.05	1.49	44	74
OSNOM (adapted)	1.33	1.65	34	72
ReMEmbR	1.48	1.74	22	71
DirectMe (graph only)	1.88	1.93	17	54
Clip centroid (no memory)	1.69	1.81	9	64

Appendix

Table 21: The 138 queries answered by every method, which guards against methods answering different subsets.

Detector	s/frame	Detections/clip	Box IoU	3D distance (m)	Median $L_{2}$ (m)
SAM 3	4.4–11.1	1,281	–	–	0.984
YOLO-World	0.051	424	0.928	0.439	1.089
YOLOE-26	0.047	488	0.950	0.433	1.318
YOLOE-11	0.049	471	0.961	0.415	1.404

Appendix

Table 22: Detector comparison with the same vocabulary, sampling, lifting, and pose settings. Localization uses trajectory memory and EgoLoc poses. Box IoU and 3D distance measure agreement with SAM 3 where both produce detections. SAM 3 is not ground truth. Timing covers detection only.

Tag source	GPT judge	Qwen judge	Tags/clip
RAM++	53	39	53
Gemma-3-4B	76	64	297
Qwen3.5-9B $\cup$ RAM++	84	68	–
Canonical union	86	70	112
GPT vision, full	87	76	177
Gemma $\cup$ Qwen $\cup$ RAM++	87	75	–

Appendix

Table 23: Reported semantic vocabulary coverage (%) of the 164 query objects. These are automatic label-coverage estimates, not measured object-instance recall. Vocabulary sizes are means per clip when available.

			Text only				Frames +
Subcategory	$n$	Blind	Ours	DM graph	DM blind	Frames	Ours	DM, own	DM, routed
Quantity Change Tracking	624	34.5	33.3	17.5	21.3	37.8	38.8	30.9	37.7
My Trajectory	465	35.9	41.3	38.3	43.2	45.8	48.8	41.5	44.1
Relative Position & Orientation	392	24.0	32.4	29.1	22.2	37.8	40.6	35.7	36.2
Distance Comparison	313	41.9	44.7	38.3	36.1	48.9	48.2	43.1	43.8
Ego-centric Position	309	23.6	34.6	22.7	23.6	42.1	48.2	39.5	38.8

Appendix

Table 24: UCS-Bench accuracy (%) by subcategory on the 2,771 questions of Table 3 , with its abbreviations.

Dataset	$n$	Blind	Memory	48 frames	Frames + memory
EgoLife (189 recordings)	2,569	33.6	38.4	44.1	45.8
TeleEgo (22 recordings)	203	37.4	39.4	47.3	52.2
HourVideo (21 of 167 recordings)	358	39.7	41.1	53.1	49.4
All three	3,130	34.5	38.8	45.4	46.6

Appendix

Table 25: UCS-Bench accuracy (%) by source dataset. HourVideo covers only the 21 of 167 recordings whose memory could be built.

Evidence	All	Objects	Actions	People
Blind	34.6	37.2	28.0	36.1
Memory, text	33.3	33.6	29.8	37.0
Memory, text + counting aid (tracks, most visible together)	26.9	25.0	25.0	34.5
48 uniform frames at 448 px	37.9	33.9	41.1	45.4
48 frames chosen from memory keyframes	37.4	33.6	36.9	49.6
24 frames at 896 px, half from the last 24 s, + 6 keyframes	39.5	38.4	42.3	37.8

Appendix

Table 26: Quantity Change Tracking on EgoLife and TeleEgo (625 questions). Question types by keyword: objects present (336), the wearer’s actions (168), people (119).

Evidence	single	stitched	$\Delta$	single	stitched	$\Delta$	single	stitched	$\Delta$
	All (3,244)			Same scene (1,959)			Scene change (1,285)
None (blind)	33.2	33.5	$+0.3$	33.0	32.9	$-0.2$	33.4	34.4	$+1.0$
48 frames	47.8	44.2	$-3.6^{***}$	44.0	41.2	$-2.7^{*}$	53.7	48.6	$-5.1^{***}$
DM graph	30.1	29.6	$-0.5$	29.7	29.7	$-0.1$	30.7	29.6	$-1.2$
DM frames + graph (own)	42.9	40.9	$-2.0^{*}$	38.9	39.9	$+1.0$	48.9	42.4	$-6.5^{***}$
Ledger memory	39.0	36.4	$-2.5^{**}$	38.4	37.9	$-0.5$	39.8	34.2	$-5.7^{***}$

Appendix

Table 27: Paired accuracy (%) on individual recordings and 100 stitched streams (3,244 questions). Same scene joins different days in one house; scene change joins different places.

\Delta

: stitched minus single;

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

(McNemar). The last row emulates per-scene construction with detected boundaries (Appendix E.4 ).

Subset	$n$	Blind	Frames	Ours mem.	Ours fr.+mem.	DM graph	DM fr.+graph	DM fr.+graph, routed
All	3244	$+0.3$	$-3.6^{***}$	$-2.5^{**}$	$-3.3^{***}$	$-0.5$	$-2.0^{*}$	$-3.4^{***}$
First recording	1503	$+0.7$	$-0.2$	$-0.9$	$+0.7$	$-1.7^{*}$	$-0.9$	$-0.3$
After one earlier scene	1558	$-0.4$	$-5.6^{**}$	$-3.0^{*}$	$-6.0^{**}$	$-0.3$	$-2.6$	$-6.1^{**}$
After two earlier scenes	183	$+3.3$	$-15.3^{**}$	$-11.5^{*}$	$-12.0^{*}$	$+7.7^{*}$	$-6.0$	$-6.0$
Scene-change streams	1285	$+1.0$	$-5.1^{***}$	$-5.7^{***}$	$-6.8^{***}$	$-1.2$	$-6.5^{***}$	$-6.1^{***}$
Same-scene streams	1959	$-0.2$	$-2.7^{*}$	$-0.5$	$-0.9$	$-0.1$	$+1.0$	$-1.6$

Appendix

Table 28: Stitched

-

single (points) by the position of the question’s recording in the stream and by stream type. Stars as in Table 27 ; cells whose recorded

p

is rounded to 0.00 are marked ∗∗ .

Memory over the stitched stream	Accuracy	vs. single-scene build ( $p$ )
Single-recording build (reference)	39.8	$+5.7$ ( $<0.001$ )
Single-scene build over the stream	34.2	–
+ retrieval restricted to the current scene, true cuts	35.3	$+1.1$ (0.37)
+ retrieval restricted, cuts from the place-partition detector	35.0	$+1.0$ (0.34)
Per-scene construction, true cuts	38.4	$+4.3$ ( $<0.01$ )
Per-scene construction, detected cuts	37.0	$+2.9$ (0.05)

Appendix

Table 29: Scene-change streams, memory-only arm, 1,285 paired questions: which change recovers the loss. “True cuts” use the known seam times; “detected cuts” come from the video.

Prototype	Control	Event lines	$\Delta$ [95% CI]	AMEGO
Object location	48.8	50.8	$+2.0$ $[-1.6,+5.6]$	39.8
Object contents retrieval	30.5	35.0	$+4.5$ $[-1.5,+10.5]$	22.0
Fixture location	56.2	58.0	$+1.8$ $[-1.0,+4.6]$	12.8
Fixture interaction counting	41.3	40.7	$-0.7$ $[-4.3,+3.0]$	47.0
Movement itinerary	40.4	42.2	$+1.8$ $[-2.4,+5.8]$	26.8
Movement counting	42.5	43.0	$+0.5$ $[-4.5,+5.5]$	34.5

Appendix

Table 30: Event lines on HD-EPIC (2,400 questions, GPT-5.4): accuracy (%) per prototype for the memory re-answered in the same session (control) and with event lines;

\Delta

is paired, with 95% bootstrap intervals. AMEGO (adapted) is shown for reference.

Figure 7: UCS-Bench stitched streams, GPT-5.4. (a) Accuracy lost on each question category when one component is removed from the memory with event lines (single runs, 2,246 questions answered by every arm; Table 31 ). (b) The same two records read by our single-shot reader and by ReMEmbR’s querying agent (2,777 paired questions; Table 32 ).

Memory	All	MyTraj	ObjTraj	EgoPos	RelPos	Dist	Reach	Recall	Count
$n$	2,246	348	138	195	329	198	218	268	552
Full (objects + event lines)	39.0	35.6	39.9	37.4	40.7	47.0	57.8	42.5	28.3
$-$ event lines	37.8	37.1	36.2	36.4	37.1	43.9	54.1	36.9	31.2
$-$ object descriptions	37.6	38.2	39.1	38.5	33.7	44.9	55.5	39.2	28.4
$-$ 3D geometry	38.6	37.6	41.3	24.1	38.9	48.0	56.9	46.3	29.3
$-$ observation history	38.4	39.9	42.0	29.7	33.7	49.5	56.9	39.9	30.4

Appendix

Table 31: Leave-one-out on the stitched streams (2,246 questions answered by every arm). Category columns: my trajectory, object trajectory, egocentric position, relative position, distance comparison, reachability, object recall, counting.

Record $\rightarrow$ reader	All	MyTraj	ObjTraj	EgoPos	RelPos	Dist	Reach	Recall	Count
$n$	2,777	423	156	270	409	288	257	307	667
ReMEmbR captions $\rightarrow$ R1	36.0	38.5	37.8	29.6	32.3	41.7	55.3	32.9	30.6
ReMEmbR captions $\rightarrow$ R2 (= ReMEmbR)	41.1	45.9	50.0	28.1	37.9	42.7	48.6	44.3	37.9
Our event lines only $\rightarrow$ R1	38.2	40.2	39.7	26.7	33.5	46.2	58.0	41.4	31.8
Our objects + ReMEmbR captions $\rightarrow$ R1	38.5	39.5	36.5	35.6	38.9	45.8	55.6	39.7	29.1
Ledger (objects + event lines) $\rightarrow$ R1	39.9	37.1	40.4	36.7	40.6	45.8	58.8	45.6	30.0

Appendix

Table 32: Record against reader on the stitched streams (2,777 questions answered by every arm). R1 is our single-shot reader; R2 is ReMEmbR’s retrieval agent. Blind on these streams is 33.2.

Arm	All	MyTraj	ObjTraj	EgoPos	RelPos	Dist	Reach	Recall	Count
$n$	3,300	511	191	305	480	318	308	374	813
Qwen3.5-9B, blind	32.3	39.5	46.6	22.3	28.5	38.1	49.0	34.2	20.9
Ledger + event lines, Qwen3.5-9B	36.2	38.0	35.6	36.1	34.8	42.8	53.6	38.0	26.3
DirectMe graph + keyframes, Qwen3-VL-8B	28.0	33.3	33.5	24.9	27.1	36.5	46.8	26.2	15.5
Ledger + event lines, GPT-5.4	39.5	36.2	38.7	37.0	39.8	45.9	58.1	44.7	30.6
ReMEmbR, GPT-5.4	40.8	45.0	50.3	27.5	38.5	42.5	49.4	43.6	37.0

Appendix

Table 33: Answering models on the stitched streams (3,300 questions answered by every arm).

Evidence	At its moment	Delayed	$\Delta$	$\Delta$ net of blind
Blind	32.4	37.9	$+5.5$	–
48 frames	44.0	39.6	$-4.4$	$-9.9$
DirectMe graph	25.9	28.3	$+2.5$	$-3.0$
ReMEmbR	40.6	42.7	$+2.1$	$-3.4$
Ledger , objects only	34.1	32.8	$-1.4$	$-6.9$
Ledger + event lines	35.1	36.6	$+1.4$	$-4.1$

Appendix

Table 34: Delayed questions (763): accuracy at the question’s own moment and when re-asked at the end of the stream, with the change net of the blind control.

Length	Videos	Questions	single-shot	agent	ReMEmbR	AMEGO	DirectMe	OSNOM	Blind
			Ledger
$<$ 5	45	284	48.6	35.6	29.6	35.2	33.8	38.4	38.7
5–10	22	266	48.5	39.1	32.7	31.6	27.1	32.7	30.5
10–20	37	769	45.1	33.2	27.8	31.3	29.6	31.1	28.9
20–40	34	710	42.4	30.1	23.4	31.1	30.3	32.7	28.0
40–75	12	371	41.8	37.2	29.9	31.3	26.4	30.5	27.0

Appendix

Table 35: HD-EPIC accuracy (%, over questions) by recording length (minutes).

Length	Videos	Questions	single-shot	agent	ReMEmbR	DirectMe	Blind	48 frames
			Ledger
$<$ 2	16	236	36.0	41.5	37.7	31.4	33.1	61.4
2–6	23	370	38.1	43.5	42.4	28.6	32.4	50.0
6–11	18	321	45.8	49.5	46.1	29.9	34.9	55.1
11–13	123	2,050	38.6	40.4	40.8	29.3	32.6	44.5
13–20	26	287	37.3	44.3	41.5	36.2	35.2	48.1

Appendix

Table 36: UCS-Bench accuracy (%) by recording length (minutes), single recordings.

Length	Clips	Queries	single-shot	agent	ReMEmbR	DirectMe	OSNOM	Clip centroid
			Ledger
$\approx$ 5	16	66	56.1	59.1	22.7	10.6	34.8	7.6
$\approx$ 8	26	93	43.0	45.2	23.7	21.5	30.1	11.8
16	2	5	20.0	0.0	20.0	40.0	0.0	20.0
All	44	164	47.6	49.4	23.2	17.7	31.1	10.4

Appendix

Table 37: VQ3D: queries localized within 1 m (%) by clip length (minutes).

Arm	HD-EPIC	UCS-Bench	UCS-Bench, EPIC-Kitchens only
Ledger , single-shot reader	$-0.072$ (0.028)	$+0.013$ (0.69)	$+0.021$ (0.74)
Ledger ${}_{\text{agent}}$	$-0.054$ (0.12)	$-0.007$ (0.82)	$+0.022$ (0.72)
ReMEmbR	$-0.060$ (0.099)	$+0.016$ (0.63)	$+0.086$ (0.17)
DirectMe	$-0.072$ (0.044)	$+0.013$ (0.71)	$+0.010$ (0.89)
AMEGO (adapted)	$-0.076$ (0.029)	–	–
OSNOM (adapted)	$-0.084$ (0.016)	–	–

Appendix

Table 38: Change in the log-odds of a correct answer per doubling of recording length, with its

p

-value in parentheses.

Stage	What is lost	Questions affected	Evidence
Sampling (one frame per 4 s)	objects seen only briefly; views for triangulation	all	2 s sampling did not help a ten-video pilot (App. B.6 )
Tagging	objects outside the per-video vocabulary never enter memory	any about those objects	the best tag set misses about one VQ3D query object in ten (Table 23 )
3D lifting	depth along the viewing ray	metric localization	the error lies almost entirely along the ray (App. B.7 )
Association	one object split into several tracks	counting	the track count is exact for 11% of UCS-Bench count questions (App. D.2 )
Record content	continuous rest intervals; hand events	stationary localization, fixture-interaction counting	rest-segment starts are near chance; hand tracklets score higher (App. B.8 , Table 10 )
Construction	labels and metric scale at a scene cut; tractability with length	after a scene change; hour-long recordings	RQ5; 21 of 167 HourVideo memories built (App. D )

Appendix

Table 39: Where evidence is lost, by pipeline stage: what each stage loses, the questions it affects, and the measurement behind it.

Figure 8: Three memories answer questions about a multi-scene egocentric stream from their stored records alone. Scenes 1–3 form one UCS-Bench stream that joins three EPIC-Kitchens recordings; DirectMe, ReMEmbR, and Ledger each build one memory over the stream and answer Q1–Q4 from it as text, without frames. Scene 4 is a separate Ego4D VQ3D clip placed on the same timeline for illustration, with memories built from that clip alone; Q5 asks for the chair’s position in metres, and the map (right) shows each memory’s answer. Bottom: the evidence each memory retrieves for Q2. Ledger ’s detector labelled the object a coffee maker, but the segment description records what the wearer’s hands were holding; DirectMe’s graph has no node for the object, and ReMEmbR’s captions near that time do not name it. Questions were selected for illustration; aggregate results are in Tables 3 and 27 .

Memory	Read at answer time	Acc. (%)	Input tok	Output tok	Calls
Blind	question and options only	29.7	412	184	1.0
ReMEmbR	captions, retrieval agent	29.8	25,201	207	4.1
OSNOM (adapted)	tracked object map	30.2	8,762	111	1.0
DirectMe	scene graph, its own prompt	30.7	2,362	5	1.0
Ledger , Qwen3.5-9B	object memory	33.6	1,938	381	1.0
AMEGO (adapted)	hand–object interaction log	34.7	792	124	1.0

Appendix

Table 40: Answer-time cost on HD-EPIC (2,400 questions, GPT-5.4 unless stated). Accuracy is prototype-averaged, as in Table 2 . No arm here reads frames.

Evidence	Read at answer time	Acc. (%)	Input tok	Output tok	Calls
Blind	question and options only	33.8	452	149	1.0
DirectMe	scene graph	29.9	1,937	2	1.0
Ledger	object memory	38.5	2,044	211	1.0
Frames	48 frames	44.4	9,615	50	1.0
Frames, DM routed	frames; graph on spatial q.	42.9	10,406	49	1.0
Frames, DM own	frames and graph always	39.8	11,377	48	1.0

Appendix

Table 41: Answer-time cost on UCS-Bench (2,772 EgoLife and TeleEgo questions, GPT-5.4). Green marks arms whose answerer sees no frames, red those that do. Frame arms read 48 uniformly sampled frames up to the question time. This set has one question more than Table 3 , so DirectMe’s memory-only accuracy reads 29.9 here against 30.0 there.

Explore similar work

Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants
Oct 4, 2026Jiazhou Liang, Liam Gallagher, Kiko Chen +4Spatial Memory Conversational Memory
University of Toronto
OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation
Sep 23, 2026Jack B. Jedlicki, Tanguy Dieudonné, Heng YangSpatial Memory Long-Horizon Manipulation
Harvard University · ETH Zürich
R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video
Aug 11, 2026Ke Ma, Yamin Mao, Weiming Li +5Long Video Question Answering Egocentric Video
College of Design and Innovation Tongji University Shanghai, China · Samsung R&D Institute China - Beijing Beijing, China · Shanghai Jiao Tong University Shanghai, China +2

Oct 4, 2026, cs.AIJ/K move · Enter open · S save

Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants

Jiazhou Liang, Liam Gallagher, Kiko Chen, David Guo+3

University of Toronto

Spatial Memory Conversational Memory

Open paper AskPDF arXiv

Egocentric assistants must connect what users say with what they see across long interaction histories. We formalize this challenge as Spatially grounded Conversational Reasoning (SpaCR): cross-scene, recall-oriented, and counterfactual spatial queries that combine user-stated facts with geometric evidence. Direct vision-language models incur high inference costs and context limits as histories grow, while keyframe selection and retrieval can omit objects or evidence needed for complete recall. We propose Spatially grounded Conversational Memory (SpaC-MEM), an object-centric working memory that uses 3D reconstruction and segmentation to ground conversational information in persistent physical objects. It compresses multimodal histories while preserving spatial evidence and allowing object-specific facts to be updated through dialogue. We also introduce Ego-SpaCR, a benchmark comprising 620 ScanNet video sessions augmented with 95 task-oriented conversations and 3,100 evaluation queries. SpaC-MEM achieves the highest overall answer accuracy among the evaluated methods and improves object recall while requiring substantially fewer reference input tokens than native-video baselines. Removing 3D spatial information substantially degrades performance, highlighting the importance of preserving spatial and conversational evidence together.

Abstract

Explore similar work

Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants

OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation

R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video