Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents

Tianyidan Xie; Shenyi Wang; Qiang Tang; Mingjie Wang; Zhicheng Qiu; Xuanfu Li; Zhan Xu; Jian Yang

cs.CVSep 4, 2026

Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents

Authors: Tianyidan Xie, Shenyi Wang, Qiang Tang, Mingjie Wang, Zhicheng Qiu, Xuanfu Li, Zhan Xu, Jian Yang, +2 more

Organizations: Nanjing University · University of British Columbia · Zhejiang Sci-Tech University · Huawei Technologies Co., Ltd. · Tianjin University

Abstract

Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is \textbf{Linguistic Trajectory Encoding} (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the \textbf{Spatial Memory Benchmark} (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves $45.3\%$ success in semantic trajectory retrieval and $48.7\%$ in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: $31.9\%$ and $34.4\%$ ). LTE achieves trajectory compression by factors of $8.7\times$ to $26.1\times$ with sub-second query latency on $24$ ,h video. On Ego4D natural-language queries, the system reaches $28.75\%$ / $55.10\%$ R@1/R@5, $+15.80$ / $+31.30$ pts over EgoVLPv2.

Figures & tables

Figure 1 : Our memory system at a glance. A 24 h egocentric day is encoded into per-object Linguistic Trajectory Encoding (LTE) records, each braided from three channels: linguistic captions ( $\mathcal{C}$ , green), sparse spatial anchors ( $\mathcal{A}_{\text{spatial}}$ , blue), and visual anchors ( $\mathcal{A}_{\text{visual}}$ , terracotta). An octree provides room-level spatial pruning. Natural-language queries traverse only the relevant channel(s) and recover both the location and the moment, in $0.43$ s, instead of $98$ s on $24$ h video.

Figure 2 : LTE-centric memory architecture. (1) Perception (left) converts egocentric video into structured per-frame observations. (2) The memory layer (centre) records dynamic objects with Linguistic Trajectory Encoding (captions $\mathcal{C}$ , sparse spatial anchors $\mathcal{A}_{\text{spatial}}$ , and visual anchors $\mathcal{A}_{\text{visual}}$ ) and uses an octree as a static spatial index for region pruning. (3) Five complementary views feed a query router that dispatches to four tasks: NLQ and VQ2D from the Ego4D benchmark suite, and STR and LOR introduced in this paper as part of SMB.

Method	IoU=0.3		IoU=0.5
Method	R@1	R@5	R@1	R@5
Supervised
EgoVLPv2 ( Pramanick et al., 2023 )	12.95	23.80	7.91	16.11
GroundNLQ ( Hou et al., 2023 )	27.20	54.42	18.91	39.98
EgoVideo ( Pei et al., 2024 )	28.65	53.30	19.73	40.42
OSGNet ( Feng et al., 2025 )	32.56	59.82	22.74	46.35

Table 1 : Ego4D NLQ validation set. Take-away: the LTE-based system reaches

28.75\%/55.10\%

R@1/R@5 at IoU=0.3, improving substantially over video-language baselines (EgoVLPv2:

12.95/23.80

) and approaching task-specific supervised models. Structured object timelines and event indexing close most of the gap to supervised methods on temporal localisation.

Method	stAP	tAP	Succ.	Rec.
Supervised
VQLoC	0.22	0.31	55.9	47.1
PRVQL	0.27	0.35	57.9	47.9
Zero-shot
RELOCATE	0.33	0.41	58.0	50.5
Ours	0.36	0.43	59.5	51.2

Table 2: Ego4D VQ2D. Take-away: object-centric visual anchors enable direct identity matching, yielding the strongest zero-shot performance on all four metrics.

Method	stAP	tAP	Succ.	Rec.
Supervised
VQLoC	0.22	0.31	55.9	47.1
PRVQL	0.27	0.35	57.9	47.9
Zero-shot
RELOCATE	0.33	0.41	58.0	50.5
Ours	0.36	0.43	59.5	51.2

Table 2: Ego4D VQ2D. Take-away: object-centric visual anchors enable direct identity matching, yielding the strongest zero-shot performance on all four metrics.

Method	STR (%)	LOR (%)
VLM baselines (clip-scanning)
Q3VL-8B+GD	21.5	25.1
Q3VL-235B+GD	31.9	34.4
Structured-memory baselines
KFMem (3D-Mem-style)	19.8	33.8
VideoAgent ( Fan et al., 2024 )	24.7	30.5

Table 3: Spatial Memory Benchmark. STR/LOR success rate (%) on

300

queries each. Q3VL-8B/235B+GD denote Qwen3-VL augmented with Grounding-DINO; KFMem is a 3D-Mem-style structured-memory baseline. Take-away: the LTE-based system delivers

+13.4

STR /

+14.3

LOR over the strongest VLM baseline.

Figure 3 : Qualitative examples across the four tasks. Each row compares our system against the strongest competing method with one success and one diagnostic failure/partial case. Green =success, amber =partial, red =failure. Take-away: most failures originate upstream of LTE rather than from the memory representation; tracking ID-switches alone account for $34\%$ of STR errors (Appendix E ).

Dur.	#Obj.	Dense	Total	LTE	Compr.	Ours	Q3VL-235B+GD
Dur.	#Obj.	(MB)	(MB)	(MB)	Compr.	(s/q)	(s/q)
$2$ h	$\sim$ 18	71	45	8.2	$8.7\times$	0.15	8.2
$6$ h	$\sim$ 35	214	73	16.1	$13.3\times$	0.24	24.5
$12$ h	$\sim$ 52	428	98	23.5	$18.2\times$	0.32	49.1
$24$ h	$\sim$ 78	856	134	32.8	$26.1\times$	0.43	98.3

Table 4: Efficiency on a single A800 GPU. Memory footprint and per-query latency vs. video duration. Compression ratio compares LTE-only storage against a dense per-frame trajectory store. Take-away: memory grows sub-linearly and query latency stays sub-second up to

24

h, while the VLM baseline scales linearly and crosses

1.5

min/query.

	Absolute (%)				$\Delta$ vs. Ours
Configuration	NLQ	VQ2D	STR	LOR	NLQ	VQ2D	STR	LOR
Ours (full)	55.10	59.5	45.3	48.7	–	–	–	–
w/o text captions	53.40	58.9	33.5	48.7	$-1.70$	$-0.6$	$-11.8$	$0.0$
w/o spatial anchors	54.30	57.1	41.2	48.7	$-0.80$	$-2.4$	$-4.1$	$0.0$
w/o visual anchors	54.70	53.4	43.9	48.7	$-0.40$	$-6.1$	$-1.4$	$0.0$
w/o LTE (all removed)	52.31	51.8	28.8	48.7	$-2.79$	$-7.7$	$-16.5$	$0.0$

Table 5: Fine-grained ablation. Removing one component at a time across NLQ R@5 (

\%

), VQ2D Success (

\%

), SMB-STR Success (

\%

), and SMB-LOR Success (

\%

\Delta

is the change relative to the full system. Take-away: text captions drive STR (

-11.8

), visual anchors drive VQ2D (

-6.1

), the octree drives LOR (

-3.5

); LOR is invariant to LTE.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Lookback Window	Count	With Spatial Hint	Avg. Duration
$2$ hours	$75$	$32$ ( $42.7\%$ )	$3.2$ – $4.8$ s
$6$ hours	$75$	$29$ ( $38.7\%$ )	$3.5$ – $5.0$ s
$12$ hours	$75$	$31$ ( $41.3\%$ )	$3.1$ – $4.6$ s
$24$ hours	$75$	$28$ ( $37.3\%$ )	$3.8$ – $5.2$ s
Total	$300$	$120$ ( $40.0\%$ )	$3.4$ – $4.9$ s

Appendix

Table 6 : Long-Horizon Object Retrieval query distribution by lookback horizon.

Component	Model	Role
Object detection	SAM3 (ViT-H)	Instance segmentation
Object tracking	SAM3 tracker	Cross-frame association
Point cloud	ViPE (ViT-L)	3D reconstruction
VLM (scene)	Qwen3-VL-8B-Instruct	Room-level semantic labeling, captions
LLM (parsing)	Qwen3-8B	Query parsing, event extraction
Text embedding	Qwen3-Embedding-8B	Semantic similarity

Appendix

Table 7 : Model specifications.

Component	$2$ h video	$24$ h video	Parallelizable?
Offline memory construction
SAM3 detection + tracking	$2.8$ h	$33.6$ h	Yes (per-segment)
ViPE 3D reconstruction	$1.9$ h	$22.8$ h	Yes (independent)
VLM scene labeling	$0.4$ h	$3.2$ h	Yes (per-window)
VLM object captioning	$0.8$ h	$8.5$ h	After tracking
LTE construction + indexing	$0.05$ h	$0.3$ h	After captioning

Appendix

Table 8 : End-to-end timing. Offline memory construction (one-time) and online per-query latency.

Method	$2$ h	$6$ h	$12$ h	$24$ h	Avg.
Q3VL-8B+GD	$32.0$	$26.7$	$22.7$	$18.9$	$25.1$
Q3VL-235B+GD	$39.1$	$36.5$	$32.8$	$29.3$	$34.4$
Ours	$\mathbf{51.2}$	$\mathbf{49.3}$	$\mathbf{47.8}$	$\mathbf{46.8}$	$\mathbf{48.7}$
$\Delta$ (Ours vs. Q3VL-235B)	$+12.1$	$+12.8$	$+15.0$	$+17.5$	$+14.3$

Appendix

Table 9 : Long-Horizon Object Retrieval success rate (%) by lookback horizon.

Method	STR@ $0.3$	STR@ $0.5$	LOR@ $0.3$	LOR@ $0.5$
Q3VL-235B+GD	$31.9$	$22.4$	$34.4$	$25.7$
Ours	$\mathbf{45.3}$	$\mathbf{34.8}$	$\mathbf{48.7}$	$\mathbf{39.2}$
$\Delta$	$+13.4$	$+12.4$	$+14.3$	$+13.5$

Appendix

Table 10 : SMB performance at stricter IoU thresholds (%).

Query Type	Q3VL-235B+GD	Ours	$\Delta$
Single object queries	$34.2$	$48.1$	$+13.9$
Two-object queries	$28.3$	$40.7$	$+12.4$
With spatial hint only	$33.5$	$46.8$	$+13.3$
With temporal hint only	$31.2$	$44.2$	$+13.0$
With both hints	$35.8$	$49.5$	$+13.7$
Overall	$31.9$	$\mathbf{45.3}$	$+13.4$

Appendix

Table 11 : STR success rate (%) by query type.

Configuration	STR	LOR	NLQ R@5	VQ2D Succ.
Full LTE	$45.3$	$48.7$	$55.10$	$59.5$
Text captions only	$36.2$	$48.3$	$53.80$	$52.4$
Spatial anchors only	$31.5$	$48.5$	$52.90$	$52.1$
Visual anchors only	$30.1$	$48.4$	$52.50$	$56.8$
No LTE	$28.8$	$48.7$	$52.31$	$51.8$

Appendix

Table 12 : Keep-only-one LTE variants. Success/recall (%) on each task.

Error Category	STR	LOR
Tracking failure (ID switch, lost track)	$34$	$28$
Caption ambiguity (imprecise description)	$28$	$12$
Spatial localization error (point cloud drift)	$18$	$24$
Query parsing error	$12$	$18$
Object occlusion (partial/full)	$8$	$18$

Appendix

Table 13 : Failure case categorization on SMB (%).

Explore similar work

Sep 23, 2026cs.CV

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter change. Yet current agents struggle to maintain such memory reliably. Our analysis traces this limitation to four key deficiencies: weak fine-grained visual memory, unreliable dynamic world-state tracking, failing to record world state revealed by interaction outcomes, and limited generalization from prior experience. However, existing benchmarks do not directly assess these memory capabilities during long-horizon embodied interaction. To address this gap, we introduce EmbodiedMemory-Bench (EMem-Bench), comprising 2,554 interactive episodes across four task families. EMem-Bench requires agents to build and update memory from interaction history, then use it to complete a later task by acting in the environment. We further present Embodied-Memorizer (EMem), an external memory system that organizes embodied experience into spatial, event, and scene memories. We also train EMem-8B, an 8B policy that manages and uses these memories. We evaluate a diverse range of open-source and proprietary MLLMs and representative multimodal memory systems. Results show that current models remain weak and uneven across the four challenges. Under matched backbones, EMem achieves the best overall performance among the evaluated memory systems and improves both open-source and proprietary models, while EMem-8B further improves over its backbone. Project page: https://zju-omniai.github.io/EmbodiedMemoryBench/

Lizhou Liang, Xinyu Zhong, Miao Pan +7

Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences

Embodied Agents Interaction History Visual Memory

Ask PDF

Jun 2, 2026cs.RO

eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents

We present eMEM (Embodied Memory), a hybrid graph-based memory system for embodied agents operating in physical environments. Current agent memory architectures, such as Generative Agents, MemGPT, and A-MEM, treat memory as text streams or knowledge graphs, but embodied agents require memory that is simultaneously searchable by meaning, space, and time. eMEM fills this gap with a multi-index architecture (SQLITE for structured storage, hnswlib for approximate nearest neighbour semantic search, and an R-tree for spatial queries) unified behind a single graph model. A tiered consolidation pipeline transforms raw perceptual observations into compressed summaries, mirroring hippocampal-neocortical consolidation in biological systems. Ten agent-facing recall tools expose memory retrieval primitives, including concept-to-location resolution and cross layer recall, as first-class operations for LLM tool calling. The system is fully embedded and runs in-process alongside the agent. In addition we introduce eMEM-Bench v1, a benchmark we construct over ProcTHOR-10K scenes for embodied memory evaluation. The benchmark is organised explicitly around eight cognitive-psychology paradigms (DRM lures, pattern separation, pattern completion, source monitoring, context-dependent retrieval, long-horizon interference, serial position, and a foil augmented retention curve), each chosen so that the result is interpretable against the broader memory-systems literature in humans and prior agent-memory systems; a level of diagnostic that surface-task benchmarks like LoCoMo or OpenEQA cannot provide. eMEM scores 80.8 weighted mean over 988 probes, with a flat retention curve at ceiling from 1 h to 1 yr of simulated delay on room-unique items. We show that a pure RAG baseline (the flat_rag ablation) loses 30 pt on context dependent retrieval and 29 pt on DRM lure rejection, isolating the contribution of multi-layer storage and consolidation respectively. We release both the system and the benchmark code.

A. Haroon Rasheed, Maria Kabtoul

MIND · Automatika Robotics Inria, France · CHROMA

Embodied Agents Factual Recall

Ask PDF

Jun 17, 2026cs.AI

WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents

To assist humans over extended periods in real homes, embodied agents must remember user routines, world states, and past interactions. Existing long-term memory benchmarks mainly evaluate language-centric retrieval and question answering, while embodied benchmarks often focus on short-horizon task execution without testing long-term memory use in dynamic environments. We introduce WorldLines, a project-driven benchmark for long-horizon embodied household assistance. It constructs temporally extended household traces with dialogues, actions, execution feedback, object and device state changes, and converts them into evidence-linked samples for Memory QA and Embodied Task Planning. We further propose ObsMem, an observer-grounded memory framework that maintains visibility-aware memories and action-native state trails for state-aware decisions. Experiments reveal persistent challenges in partial observability, overwritten world states, and translating long-term memory into embodied plans, while ObsMem offers a stronger reference architecture for this setting.

Yehang Zhang, Jianchong Su, Haojian Huang +7

HKUST(GZ) · Knowin · HKUST

Long-Horizon Agents Long-Horizon Task Planning

Ask PDF