Organizations: Southern University of Science and Technology · City University of Hong Kong · The Chinese University of Hong Kong, Shenzhen · Harbin Institute of Technology (Shenzhen)
Long-term LLM assistants must repeatedly decide what prior context to expose and which model should answer as an interaction evolves. Existing memory systems and model routers usually make these decisions separately, although memory construction changes the evidence, distractors, and difficulty of the resulting input. We introduce MARS, a memory-aware routing framework that uses session state as a shared control interface between memory construction and model selection. A Query and Topic Manager (QTM) estimates compact decision-relevant state that guides a fixed state-conditioned memory policy and an online router, enabling each turn to receive an appropriate memory view and a suitable model under a quality-memory-cost trade-off. We evaluate MARS on RealMem and MultiChallenge with 16 system configurations each, covering long-history memory-driven interaction and bounded sessions with structured context interference. On RealMem, MARS preserves 98.2% of full-context GPT-4o's QA while using only 14.5% of its generation API cost, including both generation and controller calls for MARS, and achieves the strongest measured memory-recall and retrieval scores among compared systems. On MultiChallenge, MARS trails the strongest static system overall but approaches its performance at lower recorded cost on a mechanism-focused subset. Together, these results support a favorable quality-memory-cost trade-off for long-history, memory-dependent interaction.
Figures & tables
Figure 1: Illustrative motivation for memory-aware routing in a multi-turn session. MARS uses a shared QTM-derived session state to coordinate task-relevant memory construction and online model routing. The surrounding panels illustrate missing constraints, stale memory, model mismatch, and costly overuse. A–C denote candidate models; outputs are schematic and costs qualitative, with “lower” relative to the overuse configuration.
Figure 2: RealMem persona-macro QA versus generation-controller API cost for key systems. Online feedback and final evaluation are excluded.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Allowed values / representation
Memory
Router
Role in the decision interface
action
MATCH , NEW_TOPIC
yes
yes
Segments the turn as continuation of the active thread or a topic switch.
query_type
STATIC , DYNAMIC , TEMPORAL , PROACTIVE
yes
yes
Distinguishes self-contained, evolving, temporal, and proactive turns.
Encodes interaction phase for memory exposure and model selection.
taxonomy.*
need_memory , memory_span , ambiguity, and risk metadata
yes
yes
Records decision-relevant memory and ambiguity state without raw topic identity.
retrieval_scope
LOCAL , GLOBAL
yes
yes
Selects active-thread evidence or broader historical retrieval.
top_k_hint
3 , 5 , 8
yes
yes
Gives the memory constructor a retrieval-budget hint.
Appendix
Table 5: QTM state schema used in the RealMem implementation. The main-text abstract fields map to this concrete schema: for example, memory need is represented primarily through taxonomy.need_memory ; memory view is represented through lens_recommendation , retrieval_scope , and top_k_hint .
QTM condition
Active state
Recent turns
Historical retrieval
Intended behavior
taxonomy.need_memory=0 or lens_recommendation=0
omitted or minimal
optional short window
none
Avoid unnecessary context expansion for self-contained turns.
action=MATCH , retrieval_scope=LOCAL
included
included
none or active-thread artifacts
Preserve the active thread for continuation, local edit, or verification turns.
phase=REVISE or phase=VERIFY
included
included
optional local evidence
Support revision and verification without drifting from the current artifact.
retrieval_scope=GLOBAL
optional active summary
short recency window
ranked global candidates
Recover cross-session evidence for temporal or resumed-topic queries.
prefer_recent=1
included if active
recency-biased
recent candidates prioritized
Bias memory construction toward recent history when recency is decision-relevant.
lens_recommendation=3 or top_k_hint=8
included
included
expanded ranked evidence
Provide a richer memory view for synthesis, finalization, or high-difficulty turns.
Appendix
Table 6: Memory-view construction policy. QTM emits top- k hints in {3,5,8} for the main MARS policy. The three-persona ablation slice uses a matched top-10 historical retrieval budget across the main memory-construction variants to control exposure budget.
categorical one-hot plus normalized numeric features, including retrieval-budget, similarity, retrieval-count, and binary state flags
Warm-start policy
cold-start online policy for MARS ; the no-update ablation freezes this cold-start policy without online updates
Appendix
Table 8: Router hyperparameters and settings.
Judge output
Definition
Range
Scoring focus
answer_helpfulness
Helpfulness and directness of the candidate answer for the current query.
0–5
helpfulness
state_alignment
Consistency with active topic, session state, constraints, and memory context.
0–5
fidelity diagnostic
revision_application
Whether revision instructions are applied without restarting, overwriting, or drifting.
0–5
revision diagnostic
intermediate_product
Quality of intermediate artifacts or state updates when the turn is not final.
0–5
intermediate quality
routing_adequacy
Whether the selected model is adequate for the state and context.
0–5
routing diagnostic
final_integration
Whether the response integrates prior state and evidence into the final answer.
0–5
fidelity diagnostic
Appendix
Table 9: Turn-level judge outputs. Normalized rubric scores, coverage signals, and failure indicators form the three composite components in Eq. 17 , weighted by 0.45, 0.40, and 0.15. A benchmark-specific failure penalty is applied before clipping; generation cost is then subtracted.
Table 10: Cost-accounting components and the unit prices used for the reported runs. Prices are per 1M tokens where applicable.
Setting
Value
Evaluation subset
Three personas from the 10-persona evaluation pool
Persona sampling
random sample with seed 28
Personas
Ethan_Hunt , Kenta_Tanaka , Sarah_Miller
Evaluated turns
473
Historical retrieval budget
top-10
Model pool
Llama3B, Qwen2.5 72B, GPT-4o
Appendix
Table 16: RealMem three-persona ablation setup. The slice is used to test component behavior under a controlled retrieval budget.
Variant
Ablated dependency
Purpose
Full MARS ( mars_full )
none
Full memory-aware routing system.
Embedding-only semantic ( mars_embedctx_bm )
Replace QTM-guided memory construction with raw-query semantic top-10 retrieval
Tests whether QTM-guided context improves over embedding-only memory.
No QTM state/header ( mars_semctx_noactive )
Start from embedding-only semantic retrieval and remove the QTM state/header from the generator prompt
Isolates prompt-state exposure relative to the embedding-only variant; its retrieval policy is unchanged.
Recent-only memory ( mars_recentctx_bm )
Replace QTM/semantic retrieval with recent-only top-10 memory
Tests whether recency alone is sufficient.
No-QTM semantic ( mars_noqtm_semctx )
Disable QTM; use semantic memory and query-embedding router features
Stress-tests the framework without QTM state.
No-QTM recent ( mars_noqtm_recentctx )
Disable QTM; use recent-only memory and query-embedding router features
Tests the weakest no-QTM recency-based setting.
Appendix
Table 17: Ablation variants on the three-persona RealMem evaluation slice. All variants use deterministic generation and the same model pool; historical retrieval uses a matched top-10 budget unless otherwise noted.
Query / setting
Required memory
Baseline pattern
MARS pattern
Error type
Present-tense continuation constraint
Preserve a previously imposed grammar/style rule across turns.
Several full-context baselines lose the active rule.
Maintains the constraint and returns a passing final verdict.
Instruction retention
School memory recall under distractors
Recall that a schoolyard feature was located near a sanitation hazard.
Baselines deny the specific detail or lose path continuity.
Recovers the relevant detail and answers consistently.
Inference memory
Three-paragraph revision limit
Preserve a local document-format constraint while editing a later version.
Baselines drift on the paragraph-count requirement.
Keeps the active artifact state and satisfies the editing constraint.
Reliable version editing
Appendix
Table 22: Representative anonymized MultiChallenge patterns summarized from fixed final-evaluation outputs. They are illustrative rather than additional quantitative evidence.
Failure case
Relevant memory evidence
Static-system behavior
MARS behavior
Diagnosis
Travel-plan follow-up after frustration
July travel windows, five adult travelers, and comfort-focused flight constraints.
Gives generic or conflicting travel guidance.
Recovers the travel constraints and responds in the correct planning frame.
Relevant-memory miss
Broken workout-video link
A recurring 9 PM back-care routine and selected exercise playlist.
Gives generic troubleshooting or ignores the routine context.
Uses the routine and exercise context to provide a targeted next step.
Context ignored
Story-development continuation
The “Golden Hook” is locked in and the next component is character development.
Pivots to a generic plot-structure step.
Continues from the locked story state and focuses on the intended component.
Version-state drift
Appendix
Table 23: Representative anonymized RealMem static-configuration error patterns; diagnoses are illustrative and descriptive, not causal.
Item
Reproducibility information
Code and configs
An implementation package will be released with system configurations, prompt templates, and evaluation scripts.
RealMem split
RealMem evaluation pool with 10 personas and 1,415 queries; QA is normalized as official raw QA divided by 3.
MultiChallenge split
Converted MultiChallenge split with 245 scenarios and 1,480 turns; chunk size 3; target subset consists of Inference Memory and Reliable Version Editing .
Ablation slice
Three-persona RealMem evaluation slice with 473 turns (472 valid QA evaluations for the full variant); personas sampled with seed 28; matched top-10 historical retrieval budget.
Candidate model pools
Main RealMem runs use the configured MARS candidate pool reported in Appendix B.4 ; ablations use Llama3B, Qwen2.5 72B, and GPT-4o.
QTM and router settings
Main QTM uses qwen/qwen-2.5-7b-instruct ; its prompt, schema, guardrails, and state-to-memory mapping are fixed. The controller-swap diagnostic additionally evaluates Mistral-7B-Instruct-v0.3 with the same original prompt/schema. Only the router updates online in the main system.
Appendix
Table 25: Reproducibility checklist. The release package will provide the implementation, configurations, prompt templates, and evaluation scripts.
Long-term conversational agents must decide which turns to store in external memory, yet recent systems rely on autoregressive LLM generation at every turn to make that decision. We present MemRouter, a write-side memory router that decouples memory admission from the downstream answer backbone and replaces per-turn memory-management decoding with an embedding-based routing policy. MemRouter encodes each turn together with recent context, projects the resulting embeddings through a frozen LLM backbone, and predicts whether the turn should be stored using lightweight classification heads while training only 12M parameters. Under a controlled matched-harness comparison on LoCoMo, where the retrieval pipeline, answer prompts, and QA backbone (Qwen2.5-7B) are held identical, MemRouter outperforms an LLM-based memory manager on every question category (overall F1 52.0 vs 45.6, non-overlapping 95% CIs) while reducing memory-management p50 latency from 970ms to 58ms. Descriptive factorial averaging further shows that learned admission improves mean F1 by +10.3 over random storage, category-specific prompting adds +5.2 over a generic prompt, and retrieval contributes +0.7. These results suggest that write-side memory admission can be learned by a small supervised router, while answer generation remains a separate downstream component in long-horizon conversational QA.
Tianyu Hu, Weikai Lin, Weizhi Zhang +2
University of Central Florida · University of Rochester · University of Illinois at Chicago +1
Long-term memory enables LLM agents to leverage past interactions, but dialogue histories quickly exceed the context window, forcing agents to retrieve relevant subsets at query time. Because useful evidence is sparse and scattered across verbose conversations, retrieval faces a fundamental tension: broadening recall improves coverage but floods downstream reasoning with noise, while compressing memories at write time eases retrieval but irreversibly discards details that future queries may need. We introduce LazyMem, which resolves this tension by deferring all memory construction to query time. Given a retrieved candidate pool, a lightweight model processes it in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained with supervised fine-tuning followed by reinforcement learning, using a reward that jointly encourages the identification of relevant messages and the generation of compressions that are faithful to the source and useful for answering the query. On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline while using only 213 answer-context memory tokens, 21.0 times fewer than the baseline. It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline. Code is available at https://github.com/allacnobug/LazyMem.
Jing Yu, Yibo Zhao, Jiaming Zhang +1
School of Data Science and Engineering, East China Normal University
Large language model (LLM) agents have demonstrated the ability to solve multi-step tasks requiring planning, tool use, and external information access, yet growing execution histories increase inference cost and expose reasoning to outdated, irrelevant, or misleading information, potentially degrading reasoning quality. Existing memory approaches organize or compress execution histories but provide limited mechanisms for deciding which memories remain active. We introduce the, a hierarchical memory system that organizes execution into tasks, subtasks, and actions while assigning each memory a dynamic retention score. Event-based updates and selection-based decay revise these scores, allowing WMT to preserve useful information, fold completed trajectories, suppress low-utility content, and retain access to folded context. We evaluate WMT on GAIA-Text using Qwen3-8B, Gemma 4 E4B, and Llama-3.1-8B, with ablations and memory-poisoning experiments. Relative to linear memory, WMT improves accuracy by an average of 9.97 percentage points while reducing prompt-token usage by 32.8%. Memory-poisoning experiments show that WMT limits the persistence and propagation of unreliable information. Our results suggest that effective long-horizon agent memory depends less on storing more information than on deciding which information should remain active.
Quang Dao, Purvi Kathalkar, Kenneth Eaton
Rose-Hulman Institute of Technology · Georgia Institute of Technology · Georgia Tech Research Institute