Who Schedules Agent Memory? A Sidecar Approach to Long-Horizon Tasks

Qijia Zhuang; Rui Liu; Yuxiang Ren

cs.AIOct 31, 2026

Who Schedules Agent Memory? A Sidecar Approach to Long-Horizon Tasks

Authors: Qijia Zhuang, Rui Liu, Yuxiang Ren

Organizations: Nanjing University · Nanjing University, Nanjing, China · Northeastern University · Northeastern University, China

Abstract

As large language model agents increasingly undertake complex, long-horizon tasks, memory has become an important component for supporting sustained decision-making and action. Existing methods primarily focus on memory storage, updating, and retrieval, but often overlook how memory scheduling affects task execution: the main agent must identify memory needs, initiate retrieval, and wait for results while advancing the task, increasing both scheduling burden and execution time. To address this problem, we propose MemSidecar, which decouples memory scheduling from task execution through two independently selectable modes. Hooks directly delivers bound memory to the main agent when execution events match predefined conditions. Foresight uses an independent agent to predict potential queries, retrieve results in parallel, and store them in a cache, delivering matching and valid results when actual requests arrive. Hooks and Foresight achieve scores of 41.23% and 44.34% on MCP-Persona, respectively, while effectively reducing execution costs and latency and improving the efficiency of long-horizon agent execution.

Figures & tables

Figure 1. Three failure modes in the XH hotel-booking task.

Figure 2. MemSidecar framework. Global Memory organizes knowledge and user preferences reusable across tasks; Workspace Memory organizes workspace-specific skills, experience, and work state. Hooks directly delivers bound memory when execution state or a pending action satisfies its condition; Foresight predicts queries from the task instruction and prepares a task-level cache in the background. The main agent uses the delivered context for task decisions.

Method	Shopping		Travel		Progressive Web Search		Reasoning		Cost per task $\downarrow$
	SR $\uparrow$	PS $\uparrow$	SR $\uparrow$	PS $\uparrow$	SR $\uparrow$	PS $\uparrow$	SR $\uparrow$	PS $\uparrow$	Store (K)	Retrieve (K)	Time (s)
Memory systems
Mem0	1.33	48.78	0.00	1.02	11.31	12.07	26.67	47.95	39.53	6.43	789
A-Mem	0.67	44.78	0.00	0.48	9.95	9.99	21.67	42.95	55.34	7.31	1063
LightMem	1.33	51.22	0.00	1.50	13.57	14.99	28.33	52.95	23.72	7.54	715
Agent harnesses

Table 1. Main results on MemoryArena.

Memory systems
Method	Checkpoint score (%) $\uparrow$								Cost per task $\downarrow$
	CP	CM	SM	EM	Lark	Rednote	Mixed	Overall	Store (K)	Retrieve (K)	Time (s)
Mem0	32.55	18.00	33.14	43.14	28.64	25.54	6.25	29.49	8.02	3.40	181.1
A-Mem	27.86	14.67	28.49	39.22	23.64	21.20	6.25	25.00	5.32	4.05	164.6
LightMem	35.16	20.67	35.47	45.10	31.59	27.72	18.75	32.18	3.81	1.94	139.8
Agent harnesses

Table 2. Main results on MCP-Persona.

Mode	Configuration	MCP-Persona			MemoryArena
		Overall (%) $\uparrow$	Store (K) $\downarrow$	Retrieve (K) $\downarrow$	SR (%) $\uparrow$	Store (K) $\downarrow$	Retrieve (K) $\downarrow$
Hooks	Full	41.23	3.03	1.72	13.41	19.61	6.20
	w/o W	36.46 (-4.77)	3.30 (+0.27)	1.92 (+0.20)	10.70 (-2.71)	21.29 (+1.68)	6.78 (+0.58)
	w/o G	38.67 (-2.56)	3.21 (+0.18)	1.79 (+0.07)	9.84 (-3.57)	20.81 (+1.19)	6.81 (+0.61)
Foresight	Full	44.34	3.11	2.29	15.41	17.85	6.07
	w/o W	38.12 (-6.22)	3.37 (+0.25)	2.59 (+0.31)	10.98 (-4.42)	19.32 (+1.47)	6.60 (+0.53)

Table 3. Dual-layer memory ablations.

Figure 3. MCP-Persona stage performance and retrieval intervention. (a) Early-, middle-, and late-stage checkpoint pass rates. (b) Original and extra-retrieval configurations of Mem0 and DSH, with standard Foresight as a reference; bars report Overall (%), Retrieve tokens (thousands), and runtime (seconds).

Figure 4. Execution trace of memory scheduling across stages in MCP-Persona task 153.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8

Condition	Overall (%) $\uparrow$	Retrieve (K) $\downarrow$	Time (s) $\downarrow$
Untrained	44.34	2.29	109.7
SFT	46.27	2.58	119.8
SFT+RL	49.17	2.82	113.7

Appendix

Table B1. ForesightAgent training comparison on MCP-Persona.

Condition	MCP-Persona				MemoryArena
	Overall (%) $\uparrow$	Store (K) $\downarrow$	Retrieve (K) $\downarrow$	Time (s) $\downarrow$	SR (%) $\uparrow$	Store (K) $\downarrow$	Retrieve (K) $\downarrow$	Time (s) $\downarrow$
only Hooks	41.23	3.03	1.72	82.5	13.41	19.61	6.20	535
only Foresight	44.34	3.11	2.29	109.7	15.41	17.85	6.07	617
Hooks+filtering	42.13	3.03	1.65	91.6	14.12	19.61	5.74	585
Hooks+Foresight	46.13	3.18	2.48	122.3	16.12	21.14	7.29	653

Appendix

Table B2. Single-mode, filtering, and joint-delivery comparison.

Method	MCP-Persona			MemoryArena
	Overall (%) $\uparrow$	Retrieve (K) $\downarrow$	Time (s) $\downarrow$	SR (%) $\uparrow$	Retrieve (K) $\downarrow$	Time (s) $\downarrow$
Mem0	35.70	2.65	148.6	10.27	6.87	747
DSH	39.16	2.91	136.8	12.55	8.13	703
Hooks	41.23	1.72	82.5	13.41	6.20	535
Foresight	44.34	2.29	109.7	15.41	6.07	617

Appendix

Table B3. Scheduling with a shared backend.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 382 papers

Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents
Jun 10, 2026Hao-Lun Hsu, Nikki Lijing Kuang, Boyi Liu +2Hierarchical Memory Long-Horizon Agent Tasks
Duke University · Snowflake AI Research
ActiveMem: Dynamic Latent Memory Trees for Long-Horizon Agents
Sep 27, 2026Song-Li Wu, Jingyi Wang, Zhaocheng Du +1LLM Agent Memory Memory-Augmented Language Model Agents
Tsinghua University · Huawei Noah’s Ark Lab
Pulse: Long-Horizon Tasks Need Streaming Personalization
Oct 31, 2026Qijia Zhuang, Jialiang Liang, Yuxiang RenLong-Horizon LLM Agents Memory-Augmented Language Model Agents
Nanjing University · Xinjiang University

Jun 10, 2026J/K move · Enter open · S save

Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents

Hao-Lun Hsu, Nikki Lijing Kuang, Boyi Liu, Zhewei Yao+1

Duke University · Snowflake AI Research

Hierarchical Memory Long-Horizon Agent Tasks Agent Memory Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large language model (LLM) agents struggle with long-horizon tasks due to their inherent statelessness, requiring all task-relevant information to be encoded in growing input contexts. The resulting degraded reasoning quality, increased inference cost, and higher latency necessitate efficient working memory mechanisms. However, existing approaches either rely on lossy compression or similarity-based retrieval, which often fail to capture temporal structure and causal dependencies required for multi-step agentic tasks. In this work, we present HORMA, a Hierarchical Organize-and-Retrieve Memory Agent that organizes experience into a file-system-like hierarchical structure, where summarized entities are linked to the corresponding raw trajectories, enabling efficient access without losing detailed information. HORMA decomposes working memory into two stages: structured memory construction and navigation-based retrieval. The construction module iteratively refines how experiences are structured by distinguishing between failures caused by missing information and those caused by misleading or overloaded context. The navigation module retrieves task-relevant context by traversing the hierarchy using a lightweight agent trained with reinforcement learning to select minimal yet sufficient context, thereby reducing latency along the critical execution path. Across ALFWorld, LoCoMo, and LongMemEval, HORMA improves task performance under constrained context budgets while requiring at most 22.17% of the baseline token usage in long conversation tasks. Compared to existing methods, it consistently achieves better efficiency-performance trade-offs and generalizes effectively to unseen tasks.

Abstract

LLM Agent Memory

Explore similar work

Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents

ActiveMem: Dynamic Latent Memory Trees for Long-Horizon Agents

Pulse: Long-Horizon Tasks Need Streaming Personalization