As LLM agents become integrated into increasingly complex workflows, they must continually acquire new capabilities while retaining competence on previously learned tasks. Lifelong agents address this through experience replay, injecting past interactions into the prompt to leverage prior experience during inference. However, replay is not free: every replayed trajectory competes with retrieval, reasoning, tool use, and verification for the same limited prompt and compute budget, making effective resource allocation essential. Existing approaches allocate these resources using fixed replay policies, regardless of whether replay is beneficial for the current task. We identify this as inference-time memory allocation, a distinct problem class for lifelong agents, and introduce LIMBO: the first online framework to our knowledge that treats memory as a controllable inference-time resource and jointly optimizes memory strategy and inference budget for each incoming task. Unlike prior approaches that fix the replay policy or require model weights, teacher supervision, or offline retraining, LIMBO learns this allocation online in a single pass, explicitly balancing task performance and inference cost without modifying the underlying agent. Across three LLM backbones on LifelongAgentBench, LIMBO achieves better cost-accuracy tradeoffs than state-of-the-art memory-augmented baselines and nearly matches all strongest such baselines at up to ~83% lower inference cost (~53% on average). LIMBO adapts its policy across models and environments without retraining, demonstrating that effective allocation can be learned online rather than manually specified.
Figures & tables
Fig. 1: Replay cost–accuracy tradeoff on DBBench. Relative to no replay, LIMBO improves accuracy by 1.4 points while increasing cost by only 1.10 × (9.8%). In contrast, increasing the replay budget from no replay to k=8 raises cost by 3.6 × for a 1.5-point gain, while increasing it further from k=8 to k=16 incurs another 1.7 × cost increase for only 0.35 additional points.
Fig. 2: LIMBO : Online memory allocation loop. The controller selects a replay action and compute budget before each task, then updates the memory bank and bandit statistics using the observed reward and cost after execution.
Primitive
Replay
Tokens
Rounds
Tools
no-replay-256
none
256
2
4
no-replay-512
none
512
3
6
full-512
full
512
3
6
retrieved-256
retrieved
256
2
4
retrieved-512
retrieved
512
3
6
retrieved-768
retrieved
768
3
6
TABLE I: Retrieval gated budget primitives used in DBBench experiments.
Model
Env
No replay
Strongest baseline
LIMBO
Cost
(Acc./Cost)
(Acc./Cost)
(Acc./Cost)
reduction
Qwen 2.5 7B
DB
73.0 / $0.046
Fixed k=16 : 74.85 / $0.290
74.4 / $0.050
−82.8%
Qwen 2.5 7B
OS
46.4 / $0.104
Fixed trim k=1 : 51.5 / $0.343
51.4 / $0.226
−34.1%
Llama 3.1 8B
DB
20.8 / $0.071
Fixed k=16 : 70.0 / $0.316
66.2 / $0.144
−54.4%
Llama 3.1 8B
OS
40.6 / $0.234
Fixed k=1 : 44.4 / $0.304
43.0 / $0.224
−26.3%
GPT-4o mini
DB
66.4 / $0.283
LongLLMLingua: 69.0 / $1.654
67.4 / $0.292
−82.3%
TABLE II: Overall cost–accuracy summary across three LLM backbones and two environments on LifelongAgentBench (500 tasks each). The strongest fixed-memory baseline is the highest-accuracy fixed-memory method available in each setting. Cost reduction is LIMBO ’s cost relative to that baseline. All LIMBO points use the configuration described in Section IV-A ; per-cell method points appear in Figure 3 . All values are three-seed means (seeds 42 , 43 , 44 ).
Fig. 3: Cost–accuracy plots across three LLM backbones (columns) and two environments (rows). Markers denote no replay (circles), fixed replay (squares), retrieval (triangles), LongLLMLingua (diamonds), GSC (pentagons), and LIMBO (stars). LIMBO generally occupies a low-cost, competitive-accuracy region, while the highest-accuracy fixed-replay configurations often incur substantially greater inference cost.
Fig. 4: LIMBO ’s learned action mix across the six (model, environment) settings, each a three-seed mean (seeds 42 , 43 , 44 ). The same controller, features, and hyperparameters produce sharply different policies per cell. Qwen DB and GPT-4o mini DB stay near no-replay ( ≈78% of tasks); Llama DB shifts 85% of tasks to retrieved replay to recover from a 20.8% no-replay collapse; all three OS environments concentrate on retrieved and full replay, with mix proportions that vary by backbone. No fixed replay policy is uniformly cost-effective, and each cell retains task-level variation rather than collapsing to a single arm.
Fig. 5: LIMBO ’s online adaptation over the 500-task Qwen OS stream, averaged across three seeds ( 42 , 43 , 44 ) in non-overlapping 50-task windows. (a) The memory-action mix shifts from an exploration-heavy early regime (window 1: 17.3% no replay, 34.0% full replay, 48.7% retrieved) toward a stable replay-heavy regime (windows 2–10 mean: 3.8% no replay, 23.5% full replay, and 72.7% retrieved replay). Compressed replay is selected on 0% of tasks throughout and is omitted from the stack. (b) The α -scaled UCB uncertainty bonus for the selected action decays by 67% from the first 100 tasks to the last 100 ( 0.347→0.115 ) as per-action precision matrices Aa accumulate observations. Cross-seed SDs (shaded band) stay below 0.017 in every window, so the trajectory is a property of the controller rather than of any single run.
Inference-time compute allocation decides where, when, and how much computation models spend during prediction to improve quality under resource limits.
Lifelong agents continually accumulate and reuse experience to improve after deployment, yet in practice they often operate under tight generation, interaction, and memory budgets. Under such stringent joint constraints, we find that existing lifelong-memory methods can become ineffective, with several sophisticated approaches offering little advantage over simple retrieval or even no-memory agents. This reveals a more fundamental problem: resource scarcity makes each memory decision more consequential. When only a small amount of past experience can be retained or reused and opportunities for reasoning and recovery are limited, retrieving a relevant but ineffective memory can consume scarce context and hinder downstream execution. The key challenge is therefore not simply to retrieve relevant experience, but to determine which experiences are actually worth retaining and reusing. To address this challenge, we propose TokenWorth, a lightweight online memory framework that learns query-dependent memory worth from sparse execution feedback and applies it consistently to memory retrieval and bounded memory management. Across three LifelongAgentBench environments and both local and API-based deployments, TokenWorth achieves the highest task success rate while using the fewest output tokens among the evaluated methods. These results suggest that effective lifelong adaptation under resource constraints depends not on remembering more, but on learning what is worth remembering. Our code is publicly available at https://github.com/AIGNLAI/TokenWorth.
Guannan Lai, Zi-Yang Bai, Jun-Peng Jiang +1
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University
Memory has become integral to the LLM agent ecosystem, supporting information retention and reuse across interactions. However, most existing agent memory systems construct memory in a query-agnostic manner, which can incur unnecessary preprocessing cost and discard details that later prove essential. Recent studies have begun shifting memory processing toward runtime adaptation, but typically specialize in particular operations or fixed processing schemes, leaving flexible control over performance, cost, and latency largely underexplored. To address this challenge, we present MemPilot, a flexible framework that orchestrates on-demand memory curation under different performance--cost--latency preferences. Specifically, we optimize a multi-step LLM policy via reinforcement learning to iteratively choose between retrieving from query-agnostic memory and delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy jointly controls evidence amount, curation instructions, model selection, and visual access, enabling fine-grained allocation of runtime computation. To optimize this policy under competing objectives, we adapt objective-wise advantage decoupling by separately estimating each objective's advantage before aggregation. Moreover, we introduce prefix-based marginal utility estimation for fine-grained credit assignment across multi-step rollouts. Experiments on five multimodal agent-memory benchmarks demonstrate favorable performance--cost--latency trade-offs across optimization preferences, with preference sweeps yielding broader frontiers than existing trade-off-aware baselines.
Haozhen Zhang, Haodong Yue, Quanyu Long +6
Nanyang Technological University · Tsinghua University · University of Illinois Urbana-Champaign
Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.
Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu +11
University of California Los Angeles · University of Washington · Northwestern University