Spatial Memory for AR Glasses: Hierarchical Visual Acquisition and Source-Linked Recall

Tianxing Zhu; Dan Huang

cs.AINov 3, 2026

Spatial Memory for AR Glasses: Hierarchical Visual Acquisition and Source-Linked Recall

Authors: Tianxing Zhu, Dan Huang

Organizations: *Independent Researcher, Shanghai, China

Abstract

We present a spatial-memory system that enables optical see-through AR glasses to recall earlier physical encounters. The dual-camera prototype combines hierarchical visual acquisition with source-linked storage and staged retrieval. Quality screen-ing, global and regional appearance novelty, and temporal support select monochrome observations for detailed RGB cap-ture. Semantic observations retain links to their source images, allowing retrieval to answer indexed historical questions direct-ly or inspect visual details omitted from scene descriptions. The architecture separates capture acceptance, interpretation out-come, and support for a particular question. This distinction motivates two core control mechanisms: bounded reassessment of familiar views when earlier processing leaves an evidence gap, and retention guided by source dependencies. In a 131.35 s egocentric replay, the implemented acquisition policy selected six RGB views from 648 monitoring observations, while feature extraction ran on 97.7% of observations. These results demon-strate selective RGB acquisition and identify frequent local inference as a remaining cost. Exposure duration and cross-view timing further constrain the evidence preserved from se-lected encounters. The system provides a practical acquisition-to-recall architecture for persistent AR assistance, with explicit boundaries between captured evidence, indexed interpretations, and supported historical claims.

Figures & tables

Channel	Optical or sensor characteristics	Role and design consequence
Near-eye display	Manufacturer-specified binocular MicroLED/diffractive waveguide; 640 $\times$ 480; 30° FOV [ 4 ]	Concise recall within a limited display field
Monochrome camera	Device-reported 640 $\times$ 480; 1.92 $\times$ 1.44 mm sensor; 0.71 mm focal length; f/2.0; nominal sensor-axis FOV 107.0° $\times$ 90.8°	Broad monitoring view for global and regional selection
RGB camera	Sony IMX681 (manufacturer [ 4 ] ); device-reported 4032 $\times$ 3024 native array; 4.032 $\times$ 3.024 mm sensor; 1.90 mm focal length; f/2.25; nominal sensor-axis FOV 93.4° $\times$ 77.0°	Detailed source evidence with a different viewing geometry and exposure state

Table 1: Optical channels and their roles in the memory system.

Figure 1: Hierarchical capture and evidence feedback.

Distinction	Representation	Consequence for recall
Observation versus interpretation	Stable source identity with a linked description	A revised description remains tied to the original encounter
Acquisition time versus processing time	Capture-time ordering independent of completion	Each response retains the time of its source encounter
Class sighting versus persistent identity	Entity labels and appearance tied to a source	Similar objects can remain identity-ambiguous
Historical versus current state	Time-qualified relations	Reports the observed location and time
Indexed fact versus original detail	Search structure linked to its image	Retrieval can escalate when a question exceeds the description
Processing outcome versus query support	Outcome and quality attributes assessed in query context	A completed record may still be insufficient for a claim

Table 2: Representational distinctions preserved by the memory hierarchy.

Figure 2: Memory organization and staged recall. Indexed observations support local historical lookup; general questions use catalog selection and image-backed answering.

Available evidence state	Control action	Reason
Interpretation pending	Wait and suppress duplicate work	The acquisition has not yet resolved into a semantic outcome
Transient interpretation failure	Consider bounded reuse of the saved image	A suitable saved image can support another interpretation attempt
Visual evidence inadequate and a clearer view recurs	Admit a bounded repeat acquisition	New detail may repair an existing evidence deficit
Committed evidence supports the relevant claim	Preserve the source and suppress redundant work	Additional acquisition has low immediate evidential value
Relevant source excluded by search	Expand or rerank retrieval within its budget	The relevant visual evidence already exists in memory
Past encounter never acquired	Preserve the historical gap	Historical claims require evidence from the relevant encounter

Table 3: Outcome-dependent control actions.

Figure 3: Recorded encounter and source-detail walkthrough. (a) The VGA observation with the four overlapping regional encoder crops. Both novelty scores exceed their thresholds, and the global branch admits this candidate. (b) A countertop crop of the paired RGB reference frame, acquired 29.4 ms after the VGA frame and 18.4 ms after the replay request. The cameras provide distinct fields of view. (c) An excerpt from the existing offline audit and an enlarged source detail: the mug handle is visible on the image-left side, a relation absent from the excerpt. The question illustrates source inspection using the recorded reference pair.

RGB time (s)	VGA-to-RGB offset (ms)	Request-to-RGB offset (ms)	RGB exposure (ms)
6.95	11.13	0.48	10.00
14.35	22.49	11.48	10.00
27.33	29.39	18.41	19.99
56.53	67.33	56.90	19.99
91.06	45.64	35.39	19.99
119.40	25.88	15.19	10.00

Table 4: Imaging conditions of the six selected RGB reference acquisitions.

Figure 4: Temporal trace of the 131.35 s causal replay. (a) Logged global and regional novelty with the configured thresholds. Gaps mark observations without feature extraction. The shaded initial-coverage interval uses the empty-bank initialization $\boldsymbol{G_{t}=R_{t}=1}$ . (b) Feature checks and luminance bypasses. (c) Six paired RGB acquisitions, colored by the admitting branch.

Augmented Reality

See all 20 papers

Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos
Oct 7, 2026Shravan Chaudhari, William Paul, Suchi Saria +2Memory-Augmented Video Understanding Egocentric Video Understanding
Johns Hopkins University · Johns Hopkins University Applied Physics Laboratory
Keepsake: Selective Spatial Memory for Long-Horizon Video Generation
Oct 5, 2026Abdul Mohaimen Al Radi, Kunyang Li, Yuzhang Shang +2Camera-Controlled Video Generation Long-Horizon Video Generation
Institute of Artificial Intelligence, University of Central Florida
What Spatial Memory Must Store: Occlusion as the Test for Language-Agent Memory
Jun 9, 2026Doeon Kwon, Junho BangOcclusion Reasoning Spatial Reasoning
Space Zero, Inc.

Oct 7, 2026J/K move · Enter open · S save

Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos

Shravan Chaudhari, William Paul, Suchi Saria, Rama Chellappa+1

Johns Hopkins University · Johns Hopkins University Applied Physics Laboratory

Memory-Augmented Video Understanding Egocentric Video Understanding 3D Spatial Reasoning Episodic Memory

Open paper AskPDF arXiv

As we move through the world and carry out everyday tasks, we encounter objects that may become relevant only later. We are capable of recalling where we left something or what was inside a container, even without knowing we would need it later. Here, we study how an embodied assistant can build a similar memory from egocentric videos, by observing a person's day-to-day activities. We present Ledger, a persistent 3D object memory that combines object locations, their histories, and contextual descriptions. It associates observations across the recording and retains objects after they leave the view, including those the person never touches. It clusters each object's observations by resting locations and records a move only after repeated evidence, reducing the effect of localization noise. Short descriptions preserve details such as an object's contents or supporting surface. It saves these records to later answer spatial questions without having to access the original images or video. Our memory raises HD-EPIC accuracy from 29.7% to 42.6%, UCS-Bench accuracy from 33.8% to 38.5% and localizes Ego4D objects with a 0.99 m median error on returned predictions. Our analyses identify complementary roles for temporal persistence, contextual descriptions, and retrieval. Our study on 100 stitched streams of multiple scenes each further exposes failures in both retrieval and construction. Per-scene construction partially recovers the performance lost across scene changes compared to that of single scene streams.

Abstract

Augmented Reality

Explore similar work

Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos

Keepsake: Selective Spatial Memory for Long-Horizon Video Generation

What Spatial Memory Must Store: Occlusion as the Test for Language-Agent Memory