LIMBO: Lifelong Inference-Time Memory and Budget Optimization for LLM Agents

Siddharth Sharma; Nilesh Prasad Pandey; Onat Gungor; Tajana Rosing

cs.LGSep 12, 2026

LIMBO: Lifelong Inference-Time Memory and Budget Optimization for LLM Agents

Authors: Siddharth Sharma, Nilesh Prasad Pandey, Onat Gungor, Tajana Rosing

Organizations: University of California, San Diego, CA, USA · West Virginia University, Morgantown, WV, USA

Abstract

As LLM agents become integrated into increasingly complex workflows, they must continually acquire new capabilities while retaining competence on previously learned tasks. Lifelong agents address this through experience replay, injecting past interactions into the prompt to leverage prior experience during inference. However, replay is not free: every replayed trajectory competes with retrieval, reasoning, tool use, and verification for the same limited prompt and compute budget, making effective resource allocation essential. Existing approaches allocate these resources using fixed replay policies, regardless of whether replay is beneficial for the current task. We identify this as inference-time memory allocation, a distinct problem class for lifelong agents, and introduce LIMBO: the first online framework to our knowledge that treats memory as a controllable inference-time resource and jointly optimizes memory strategy and inference budget for each incoming task. Unlike prior approaches that fix the replay policy or require model weights, teacher supervision, or offline retraining, LIMBO learns this allocation online in a single pass, explicitly balancing task performance and inference cost without modifying the underlying agent. Across three LLM backbones on LifelongAgentBench, LIMBO achieves better cost-accuracy tradeoffs than state-of-the-art memory-augmented baselines and nearly matches all strongest such baselines at up to ~83% lower inference cost (~53% on average). LIMBO adapts its policy across models and environments without retraining, demonstrating that effective allocation can be learned online rather than manually specified.

Figures & tables

Panel of Figure 1 — Fig. 1: Replay cost–accuracy tradeoff on DBBench. Relative to no replay, LIMBO improves accuracy by 1.4 points while increasing cost by only 1.10 $\times$ (9.8%). In contrast, increasing the replay budget from no replay to $k=8$ raises cost by 3.6 $\times$ for a 1.5-point gain, while increasing it further from $k=8$ to $k=16$ incurs another 1.7 $\times$ cost increase for only 0.35 additional points.

Panel of Figure 2 — Fig. 2: LIMBO : Online memory allocation loop. The controller selects a replay action and compute budget before each task, then updates the memory bank and bandit statistics using the observed reward and cost after execution.

Primitive	Replay	Tokens	Rounds	Tools
no-replay-256	none	256	2	4
no-replay-512	none	512	3	6
full-512	full	512	3	6
retrieved-256	retrieved	256	2	4
retrieved-512	retrieved	512	3	6
retrieved-768	retrieved	768	3	6

TABLE I: Retrieval gated budget primitives used in DBBench experiments.

Model	Env	No replay	Strongest baseline	LIMBO	Cost
		(Acc./Cost)	(Acc./Cost)	(Acc./Cost)	reduction
Qwen 2.5 7B	DB	73.0 / $0.046	Fixed $k{=}16$ : 74.85 / $0.290	74.4 / $0.050	$-82.8\%$
Qwen 2.5 7B	OS	46.4 / $0.104	Fixed trim $k{=}1$ : 51.5 / $0.343	51.4 / $0.226	$-34.1\%$
Llama 3.1 8B	DB	20.8 / $0.071	Fixed $k{=}16$ : 70.0 / $0.316	66.2 / $0.144	$-54.4\%$
Llama 3.1 8B	OS	40.6 / $0.234	Fixed $k{=}1$ : 44.4 / $0.304	43.0 / $0.224	$-26.3\%$
GPT-4o mini	DB	66.4 / $0.283	LongLLMLingua: 69.0 / $1.654	67.4 / $0.292	$-82.3\%$

TABLE II: Overall cost–accuracy summary across three LLM backbones and two environments on LifelongAgentBench (500 tasks each). The strongest fixed-memory baseline is the highest-accuracy fixed-memory method available in each setting. Cost reduction is LIMBO ’s cost relative to that baseline. All LIMBO points use the configuration described in Section IV-A ; per-cell method points appear in Figure 3 . All values are three-seed means (seeds

42

43

44

Panel of Figure 5 — Fig. 3: Cost–accuracy plots across three LLM backbones (columns) and two environments (rows). Markers denote no replay (circles), fixed replay (squares), retrieval (triangles), LongLLMLingua (diamonds), GSC (pentagons), and LIMBO (stars). LIMBO generally occupies a low-cost, competitive-accuracy region, while the highest-accuracy fixed-replay configurations often incur substantially greater inference cost.

Panel of Figure 6 — Fig. 4: LIMBO ’s learned action mix across the six (model, environment) settings, each a three-seed mean (seeds $42$ , $43$ , $44$ ). The same controller, features, and hyperparameters produce sharply different policies per cell. Qwen DB and GPT-4o mini DB stay near no-replay ( $\approx 78\%$ of tasks); Llama DB shifts $85\%$ of tasks to retrieved replay to recover from a $20.8\%$ no-replay collapse; all three OS environments concentrate on retrieved and full replay, with mix proportions that vary by backbone. No fixed replay policy is uniformly cost-effective, and each cell retains task-level variation rather than collapsing to a single arm.

Panel of Figure 7 — Fig. 5: LIMBO ’s online adaptation over the 500-task Qwen OS stream, averaged across three seeds ( $42$ , $43$ , $44$ ) in non-overlapping 50-task windows. (a) The memory-action mix shifts from an exploration-heavy early regime (window 1: $17.3\%$ no replay, $34.0\%$ full replay, $48.7\%$ retrieved) toward a stable replay-heavy regime (windows 2–10 mean: $3.8\%$ no replay, $23.5\%$ full replay, and $72.7\%$ retrieved replay). Compressed replay is selected on $0\%$ of tasks throughout and is omitted from the stack. (b) The $\alpha$ -scaled UCB uncertainty bonus for the selected action decays by $67\%$ from the first 100 tasks to the last 100 ( $0.347\to 0.115$ ) as per-action precision matrices $A_{a}$ accumulate observations. Cross-seed SDs (shaded band) stay below $0.017$ in every window, so the trajectory is a property of the controller rather than of any single run.

Inference-Time Compute Allocation

Inference-time compute allocation decides where, when, and how much computation models spend during prediction to improve quality under resource limits.

See all 77 papers

Remembering Has a Price: Learning Memory Worth under Resource Constraints
Oct 19, 2026Guannan Lai, Zi-Yang Bai, Jun-Peng Jiang +1Memory Management for LLM Agents Lifelong Learning Agents
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University
MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents
Oct 5, 2026Haozhen Zhang, Haodong Yue, Quanyu Long +6LLM Agent Memory LLM Memory
Nanyang Technological University · Tsinghua University · University of Illinois Urbana-Champaign
Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
Jul 15, 2026Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu +11Persistent Memory for Language Models Memory Management for LLM Agents
University of California Los Angeles · University of Washington · Northwestern University

Oct 19, 2026J/K move · Enter open · S save

Remembering Has a Price: Learning Memory Worth under Resource Constraints

Guannan Lai, Zi-Yang Bai, Jun-Peng Jiang, Han-Jia Ye

School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University

Memory Management for LLM Agents Lifelong Learning Agents

Open paper AskPDF arXiv

Lifelong agents continually accumulate and reuse experience to improve after deployment, yet in practice they often operate under tight generation, interaction, and memory budgets. Under such stringent joint constraints, we find that existing lifelong-memory methods can become ineffective, with several sophisticated approaches offering little advantage over simple retrieval or even no-memory agents. This reveals a more fundamental problem: resource scarcity makes each memory decision more consequential. When only a small amount of past experience can be retained or reused and opportunities for reasoning and recovery are limited, retrieving a relevant but ineffective memory can consume scarce context and hinder downstream execution. The key challenge is therefore not simply to retrieve relevant experience, but to determine which experiences are actually worth retaining and reusing. To address this challenge, we propose TokenWorth, a lightweight online memory framework that learns query-dependent memory worth from sparse execution feedback and applies it consistently to memory retrieval and bounded memory management. Across three LifelongAgentBench environments and both local and API-based deployments, TokenWorth achieves the highest task success rate while using the fewest output tokens among the evaluated methods. These results suggest that effective lifelong adaptation under resource constraints depends not on remembering more, but on learning what is worth remembering. Our code is publicly available at https://github.com/AIGNLAI/TokenWorth.

Abstract

Inference-Time Compute Allocation

Explore similar work

Remembering Has a Price: Learning Memory Worth under Resource Constraints

MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents