Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents

Tianyidan Xie; Shenyi Wang; Qiang Tang; Mingjie Wang; Zhicheng Qiu; Xuanfu Li; Zhan Xu; Jian Yang

cs.CVSep 4, 2026

Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents

Authors: Tianyidan Xie, Shenyi Wang, Qiang Tang, Mingjie Wang, Zhicheng Qiu, Xuanfu Li, Zhan Xu, Jian Yang, +2 more

Organizations: Nanjing University · University of British Columbia · Zhejiang Sci-Tech University · Huawei Technologies Co., Ltd. · Tianjin University

Abstract

Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is \textbf{Linguistic Trajectory Encoding} (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the \textbf{Spatial Memory Benchmark} (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves $45.3\%$ success in semantic trajectory retrieval and $48.7\%$ in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: $31.9\%$ and $34.4\%$ ). LTE achieves trajectory compression by factors of $8.7\times$ to $26.1\times$ with sub-second query latency on $24$ ,h video. On Ego4D natural-language queries, the system reaches $28.75\%$ / $55.10\%$ R@1/R@5, $+15.80$ / $+31.30$ pts over EgoVLPv2.

Figures & tables

Figure 1 : Our memory system at a glance. A 24 h egocentric day is encoded into per-object Linguistic Trajectory Encoding (LTE) records, each braided from three channels: linguistic captions ( $\mathcal{C}$ , green), sparse spatial anchors ( $\mathcal{A}_{\text{spatial}}$ , blue), and visual anchors ( $\mathcal{A}_{\text{visual}}$ , terracotta). An octree provides room-level spatial pruning. Natural-language queries traverse only the relevant channel(s) and recover both the location and the moment, in $0.43$ s, instead of $98$ s on $24$ h video.

Figure 2 : LTE-centric memory architecture. (1) Perception (left) converts egocentric video into structured per-frame observations. (2) The memory layer (centre) records dynamic objects with Linguistic Trajectory Encoding (captions $\mathcal{C}$ , sparse spatial anchors $\mathcal{A}_{\text{spatial}}$ , and visual anchors $\mathcal{A}_{\text{visual}}$ ) and uses an octree as a static spatial index for region pruning. (3) Five complementary views feed a query router that dispatches to four tasks: NLQ and VQ2D from the Ego4D benchmark suite, and STR and LOR introduced in this paper as part of SMB.

Method	IoU=0.3		IoU=0.5
Method	R@1	R@5	R@1	R@5
Supervised
EgoVLPv2 ( Pramanick et al., 2023 )	12.95	23.80	7.91	16.11
GroundNLQ ( Hou et al., 2023 )	27.20	54.42	18.91	39.98
EgoVideo ( Pei et al., 2024 )	28.65	53.30	19.73	40.42
OSGNet ( Feng et al., 2025 )	32.56	59.82	22.74	46.35

Table 1 : Ego4D NLQ validation set. Take-away: the LTE-based system reaches

28.75\%/55.10\%

R@1/R@5 at IoU=0.3, improving substantially over video-language baselines (EgoVLPv2:

12.95/23.80

) and approaching task-specific supervised models. Structured object timelines and event indexing close most of the gap to supervised methods on temporal localisation.

Method	stAP	tAP	Succ.	Rec.
Supervised
VQLoC	0.22	0.31	55.9	47.1
PRVQL	0.27	0.35	57.9	47.9
Zero-shot
RELOCATE	0.33	0.41	58.0	50.5
Ours	0.36	0.43	59.5	51.2

Table 2: Ego4D VQ2D. Take-away: object-centric visual anchors enable direct identity matching, yielding the strongest zero-shot performance on all four metrics.

Method	stAP	tAP	Succ.	Rec.
Supervised
VQLoC	0.22	0.31	55.9	47.1
PRVQL	0.27	0.35	57.9	47.9
Zero-shot
RELOCATE	0.33	0.41	58.0	50.5
Ours	0.36	0.43	59.5	51.2

Table 2: Ego4D VQ2D. Take-away: object-centric visual anchors enable direct identity matching, yielding the strongest zero-shot performance on all four metrics.

Method	STR (%)	LOR (%)
VLM baselines (clip-scanning)
Q3VL-8B+GD	21.5	25.1
Q3VL-235B+GD	31.9	34.4
Structured-memory baselines
KFMem (3D-Mem-style)	19.8	33.8
VideoAgent ( Fan et al., 2024 )	24.7	30.5

Table 3: Spatial Memory Benchmark. STR/LOR success rate (%) on

300

queries each. Q3VL-8B/235B+GD denote Qwen3-VL augmented with Grounding-DINO; KFMem is a 3D-Mem-style structured-memory baseline. Take-away: the LTE-based system delivers

+13.4

STR /

+14.3

LOR over the strongest VLM baseline.

Figure 3 : Qualitative examples across the four tasks. Each row compares our system against the strongest competing method with one success and one diagnostic failure/partial case. Green =success, amber =partial, red =failure. Take-away: most failures originate upstream of LTE rather than from the memory representation; tracking ID-switches alone account for $34\%$ of STR errors (Appendix E ).

Dur.	#Obj.	Dense	Total	LTE	Compr.	Ours	Q3VL-235B+GD
Dur.	#Obj.	(MB)	(MB)	(MB)	Compr.	(s/q)	(s/q)
$2$ h	$\sim$ 18	71	45	8.2	$8.7\times$	0.15	8.2
$6$ h	$\sim$ 35	214	73	16.1	$13.3\times$	0.24	24.5
$12$ h	$\sim$ 52	428	98	23.5	$18.2\times$	0.32	49.1
$24$ h	$\sim$ 78	856	134	32.8	$26.1\times$	0.43	98.3

Table 4: Efficiency on a single A800 GPU. Memory footprint and per-query latency vs. video duration. Compression ratio compares LTE-only storage against a dense per-frame trajectory store. Take-away: memory grows sub-linearly and query latency stays sub-second up to

24

h, while the VLM baseline scales linearly and crosses

1.5

min/query.

	Absolute (%)				$\Delta$ vs. Ours
Configuration	NLQ	VQ2D	STR	LOR	NLQ	VQ2D	STR	LOR
Ours (full)	55.10	59.5	45.3	48.7	–	–	–	–
w/o text captions	53.40	58.9	33.5	48.7	$-1.70$	$-0.6$	$-11.8$	$0.0$
w/o spatial anchors	54.30	57.1	41.2	48.7	$-0.80$	$-2.4$	$-4.1$	$0.0$
w/o visual anchors	54.70	53.4	43.9	48.7	$-0.40$	$-6.1$	$-1.4$	$0.0$
w/o LTE (all removed)	52.31	51.8	28.8	48.7	$-2.79$	$-7.7$	$-16.5$	$0.0$

Table 5: Fine-grained ablation. Removing one component at a time across NLQ R@5 (

\%

), VQ2D Success (

\%

), SMB-STR Success (

\%

), and SMB-LOR Success (

\%

\Delta

is the change relative to the full system. Take-away: text captions drive STR (

-11.8

), visual anchors drive VQ2D (

-6.1

), the octree drives LOR (

-3.5

); LOR is invariant to LTE.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Lookback Window	Count	With Spatial Hint	Avg. Duration
$2$ hours	$75$	$32$ ( $42.7\%$ )	$3.2$ – $4.8$ s
$6$ hours	$75$	$29$ ( $38.7\%$ )	$3.5$ – $5.0$ s
$12$ hours	$75$	$31$ ( $41.3\%$ )	$3.1$ – $4.6$ s
$24$ hours	$75$	$28$ ( $37.3\%$ )	$3.8$ – $5.2$ s
Total	$300$	$120$ ( $40.0\%$ )	$3.4$ – $4.9$ s

Appendix

Table 6 : Long-Horizon Object Retrieval query distribution by lookback horizon.

Component	Model	Role
Object detection	SAM3 (ViT-H)	Instance segmentation
Object tracking	SAM3 tracker	Cross-frame association
Point cloud	ViPE (ViT-L)	3D reconstruction
VLM (scene)	Qwen3-VL-8B-Instruct	Room-level semantic labeling, captions
LLM (parsing)	Qwen3-8B	Query parsing, event extraction
Text embedding	Qwen3-Embedding-8B	Semantic similarity

Appendix

Table 7 : Model specifications.

Component	$2$ h video	$24$ h video	Parallelizable?
Offline memory construction
SAM3 detection + tracking	$2.8$ h	$33.6$ h	Yes (per-segment)
ViPE 3D reconstruction	$1.9$ h	$22.8$ h	Yes (independent)
VLM scene labeling	$0.4$ h	$3.2$ h	Yes (per-window)
VLM object captioning	$0.8$ h	$8.5$ h	After tracking
LTE construction + indexing	$0.05$ h	$0.3$ h	After captioning

Appendix

Table 8 : End-to-end timing. Offline memory construction (one-time) and online per-query latency.

Method	$2$ h	$6$ h	$12$ h	$24$ h	Avg.
Q3VL-8B+GD	$32.0$	$26.7$	$22.7$	$18.9$	$25.1$
Q3VL-235B+GD	$39.1$	$36.5$	$32.8$	$29.3$	$34.4$
Ours	$\mathbf{51.2}$	$\mathbf{49.3}$	$\mathbf{47.8}$	$\mathbf{46.8}$	$\mathbf{48.7}$
$\Delta$ (Ours vs. Q3VL-235B)	$+12.1$	$+12.8$	$+15.0$	$+17.5$	$+14.3$

Appendix

Table 9 : Long-Horizon Object Retrieval success rate (%) by lookback horizon.

Method	STR@ $0.3$	STR@ $0.5$	LOR@ $0.3$	LOR@ $0.5$
Q3VL-235B+GD	$31.9$	$22.4$	$34.4$	$25.7$
Ours	$\mathbf{45.3}$	$\mathbf{34.8}$	$\mathbf{48.7}$	$\mathbf{39.2}$
$\Delta$	$+13.4$	$+12.4$	$+14.3$	$+13.5$

Appendix

Table 10 : SMB performance at stricter IoU thresholds (%).

Query Type	Q3VL-235B+GD	Ours	$\Delta$
Single object queries	$34.2$	$48.1$	$+13.9$
Two-object queries	$28.3$	$40.7$	$+12.4$
With spatial hint only	$33.5$	$46.8$	$+13.3$
With temporal hint only	$31.2$	$44.2$	$+13.0$
With both hints	$35.8$	$49.5$	$+13.7$
Overall	$31.9$	$\mathbf{45.3}$	$+13.4$

Appendix

Table 11 : STR success rate (%) by query type.

Configuration	STR	LOR	NLQ R@5	VQ2D Succ.
Full LTE	$45.3$	$48.7$	$55.10$	$59.5$
Text captions only	$36.2$	$48.3$	$53.80$	$52.4$
Spatial anchors only	$31.5$	$48.5$	$52.90$	$52.1$
Visual anchors only	$30.1$	$48.4$	$52.50$	$56.8$
No LTE	$28.8$	$48.7$	$52.31$	$51.8$

Appendix

Table 12 : Keep-only-one LTE variants. Success/recall (%) on each task.

Error Category	STR	LOR
Tracking failure (ID switch, lost track)	$34$	$28$
Caption ambiguity (imprecise description)	$28$	$12$
Spatial localization error (point cloud drift)	$18$	$24$
Query parsing error	$12$	$18$
Object occlusion (partial/full)	$8$	$18$

Appendix

Table 13 : Failure case categorization on SMB (%).

Explore similar work

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks
Sep 23, 2026Lizhou Liang, Xinyu Zhong, Miao Pan +7Embodied Agents Interaction History
Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences
eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents
Jun 2, 2026A. Haroon Rasheed, Maria KabtoulEmbodied Agents Factual Recall
MIND · Automatika Robotics Inria, France · CHROMA
WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents
Jun 17, 2026Yehang Zhang, Jianchong Su, Haojian Huang +7Long-Horizon Agents Long-Horizon Task Planning
HKUST(GZ) · Knowin · HKUST

Sep 23, 2026, cs.CVJ/K move · Enter open · S save

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

Lizhou Liang, Xinyu Zhong, Miao Pan, Xiaohe Zhou+6

Zhejiang University · Central South University · Institute of Software, Chinese Academy of Sciences

Embodied Agents Interaction History Visual Memory

Open paper AskPDF arXiv

Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter change. Yet current agents struggle to maintain such memory reliably. Our analysis traces this limitation to four key deficiencies: weak fine-grained visual memory, unreliable dynamic world-state tracking, failing to record world state revealed by interaction outcomes, and limited generalization from prior experience. However, existing benchmarks do not directly assess these memory capabilities during long-horizon embodied interaction. To address this gap, we introduce EmbodiedMemory-Bench (EMem-Bench), comprising 2,554 interactive episodes across four task families. EMem-Bench requires agents to build and update memory from interaction history, then use it to complete a later task by acting in the environment. We further present Embodied-Memorizer (EMem), an external memory system that organizes embodied experience into spatial, event, and scene memories. We also train EMem-8B, an 8B policy that manages and uses these memories. We evaluate a diverse range of open-source and proprietary MLLMs and representative multimodal memory systems. Results show that current models remain weak and uneven across the four challenges. Under matched backbones, EMem achieves the best overall performance among the evaluated memory systems and improves both open-source and proprietary models, while EMem-8B further improves over its backbone. Project page: https://zju-omniai.github.io/EmbodiedMemoryBench/

Abstract

Explore similar work

EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents

WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents