MARS: Memory-Aware Routing for Session-Level LLM Assistants

Minghan Wang; Jinhang Zuo; Zhongxiang Dai; Zhiyong Wang; Fang Kong

cs.CLOct 16, 2026

MARS: Memory-Aware Routing for Session-Level LLM Assistants

Authors: Minghan Wang, Jinhang Zuo, Zhongxiang Dai, Zhiyong Wang, Fang Kong

Organizations: Southern University of Science and Technology · City University of Hong Kong · The Chinese University of Hong Kong, Shenzhen · Harbin Institute of Technology (Shenzhen)

Abstract

Long-term LLM assistants must repeatedly decide what prior context to expose and which model should answer as an interaction evolves. Existing memory systems and model routers usually make these decisions separately, although memory construction changes the evidence, distractors, and difficulty of the resulting input. We introduce MARS, a memory-aware routing framework that uses session state as a shared control interface between memory construction and model selection. A Query and Topic Manager (QTM) estimates compact decision-relevant state that guides a fixed state-conditioned memory policy and an online router, enabling each turn to receive an appropriate memory view and a suitable model under a quality-memory-cost trade-off. We evaluate MARS on RealMem and MultiChallenge with 16 system configurations each, covering long-history memory-driven interaction and bounded sessions with structured context interference. On RealMem, MARS preserves 98.2% of full-context GPT-4o's QA while using only 14.5% of its generation API cost, including both generation and controller calls for MARS, and achieves the strongest measured memory-recall and retrieval scores among compared systems. On MultiChallenge, MARS trails the strongest static system overall but approaches its performance at lower recorded cost on a mechanism-focused subset. Together, these results support a favorable quality-memory-cost trade-off for long-history, memory-dependent interaction.

Figures & tables

Figure 1: Illustrative motivation for memory-aware routing in a multi-turn session. MARS uses a shared QTM-derived session state to coordinate task-relevant memory construction and online model routing. The surrounding panels illustrate missing constraints, stale memory, model mismatch, and costly overuse. A–C denote candidate models; outputs are schematic and costs qualitative, with “lower” relative to the overuse configuration.

Figure 2: RealMem persona-macro QA versus generation-controller API cost for key systems. Online feedback and final evaluation are excluded.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Allowed values / representation	Memory	Router	Role in the decision interface
action	MATCH , NEW_TOPIC	yes	yes	Segments the turn as continuation of the active thread or a topic switch.
query_type	STATIC , DYNAMIC , TEMPORAL , PROACTIVE	yes	yes	Distinguishes self-contained, evolving, temporal, and proactive turns.
phase	EXPLORE , CONSTRAIN , REVISE , SYNTHESIZE , VERIFY	yes	yes	Encodes interaction phase for memory exposure and model selection.
taxonomy.*	need_memory , memory_span , ambiguity, and risk metadata	yes	yes	Records decision-relevant memory and ambiguity state without raw topic identity.
retrieval_scope	LOCAL , GLOBAL	yes	yes	Selects active-thread evidence or broader historical retrieval.
top_k_hint	3 , 5 , 8	yes	yes	Gives the memory constructor a retrieval-budget hint.

Appendix

Table 5: QTM state schema used in the RealMem implementation. The main-text abstract fields map to this concrete schema: for example, memory need is represented primarily through taxonomy.need_memory ; memory view is represented through lens_recommendation , retrieval_scope , and top_k_hint .

QTM condition	Active state	Recent turns	Historical retrieval	Intended behavior
taxonomy.need_memory=0 or lens_recommendation=0	omitted or minimal	optional short window	none	Avoid unnecessary context expansion for self-contained turns.
action=MATCH , retrieval_scope=LOCAL	included	included	none or active-thread artifacts	Preserve the active thread for continuation, local edit, or verification turns.
phase=REVISE or phase=VERIFY	included	included	optional local evidence	Support revision and verification without drifting from the current artifact.
retrieval_scope=GLOBAL	optional active summary	short recency window	ranked global candidates	Recover cross-session evidence for temporal or resumed-topic queries.
prefer_recent=1	included if active	recency-biased	recent candidates prioritized	Bias memory construction toward recent history when recency is decision-relevant.
lens_recommendation=3 or top_k_hint=8	included	included	expanded ranked evidence	Provide a richer memory view for synthesis, finalization, or high-difficulty turns.

Appendix

Table 6: Memory-view construction policy. QTM emits top-

k

hints in {3,5,8} for the main MARS policy. The three-persona ablation slice uses a matched top-10 historical retrieval budget across the main memory-construction variants to control exposure budget.

Feature group	Examples	Encoding	Purpose
QTM categorical state	topic relation, query type, memory need, retrieval scope, memory view, interaction phase	one-hot / categorical	Main session-state signal for model selection.
Memory-load features	candidate memory blocks, retrieved-history flag, estimated candidate-token count	normalized numeric	Estimates the load of the pre-packing candidate view.
Session-position features	turn depth, recent revision flag, finalization flag	numeric / binary	Distinguishes early continuation from later synthesis or verification.
Retrieval features	retrieval scope, number of candidates, retrieval score summaries	numeric / categorical	Indicates whether the response depends on recovered evidence.
Cost features	model-specific cost enters through reward rather than direct decision-time subtraction	reward-side scalar	Lets the router learn quality–cost utility from observed outcomes.

Appendix

Table 7: Router feature groups. The implementation uses structured state and pre-packing candidate statistics rather than raw topic identifiers.

Hyperparameter / setting	Final value
Candidate model pool $\mathcal{A}$	Main RealMem runs: Llama-3.2-3B, Mixtral-8x7B, Qwen-72B, GPT-4o; three-persona ablation pool: Llama3B, Qwen2.5-72B, GPT-4o.
Feature dimension	30
LinUCB exploration coefficient $\alpha$	0.8
Regularization / initialization	per-arm $A_{a}=\lambda_{\mathrm{reg}}I$ with $\lambda_{\mathrm{reg}}=1.0$ and $b_{a}=0$
Feature normalization	categorical one-hot plus normalized numeric features, including retrieval-budget, similarity, retrieval-count, and binary state flags
Warm-start policy	cold-start online policy for MARS ; the no-update ablation freezes this cold-start policy without online updates

Appendix

Table 8: Router hyperparameters and settings.

Judge output	Definition	Range	Scoring focus
answer_helpfulness	Helpfulness and directness of the candidate answer for the current query.	0–5	helpfulness
state_alignment	Consistency with active topic, session state, constraints, and memory context.	0–5	fidelity diagnostic
revision_application	Whether revision instructions are applied without restarting, overwriting, or drifting.	0–5	revision diagnostic
intermediate_product	Quality of intermediate artifacts or state updates when the turn is not final.	0–5	intermediate quality
routing_adequacy	Whether the selected model is adequate for the state and context.	0–5	routing diagnostic
final_integration	Whether the response integrates prior state and evidence into the final answer.	0–5	fidelity diagnostic

Appendix

Table 9: Turn-level judge outputs. Normalized rubric scores, coverage signals, and failure indicators form the three composite components in Eq. 17 , weighted by 0.45, 0.40, and 0.15. A benchmark-specific failure penalty is applied before clipping; generation cost is then subtracted.

Component	Role / model	Input price	Output price
QTM/controller	qwen/qwen-2.5-7b-instruct	$0.04 / 1M tokens	$0.10 / 1M tokens
Turn-level judge	GPT-4o-mini online feedback judge	$0.15 / 1M tokens	$0.60 / 1M tokens
Final RealMem evaluator	GPT-4o final benchmark judge where required	excluded from both views	excluded
Final MultiChallenge evaluator	GPT-4o path-aware final judge	excluded from both views	excluded
Generators	candidate pool $\mathcal{A}$	Llama3B: $0.05; Mixtral:$ 0.20; Qwen72B: $0.35; GPT-4o:$ 2.50 / 1M tokens	Llama3B: $0.10; Mixtral:$ 0.20; Qwen72B: $0.40; GPT-4o:$ 10.00 / 1M tokens
Memory search/add	Mem0 / A-MEM where applicable	no external embedding API; local compute unpriced	same

Appendix

Table 10: Cost-accounting components and the unit prices used for the reported runs. Prices are per 1M tokens where applicable.

Setting	Value
Evaluation subset	Three personas from the 10-persona evaluation pool
Persona sampling	random sample with seed 28
Personas	Ethan_Hunt , Kenta_Tanaka , Sarah_Miller
Evaluated turns	473
Historical retrieval budget	top-10
Model pool	Llama3B, Qwen2.5 72B, GPT-4o

Appendix

Table 16: RealMem three-persona ablation setup. The slice is used to test component behavior under a controlled retrieval budget.

Variant	Ablated dependency	Purpose
Full MARS ( mars_full )	none	Full memory-aware routing system.
Embedding-only semantic ( mars_embedctx_bm )	Replace QTM-guided memory construction with raw-query semantic top-10 retrieval	Tests whether QTM-guided context improves over embedding-only memory.
No QTM state/header ( mars_semctx_noactive )	Start from embedding-only semantic retrieval and remove the QTM state/header from the generator prompt	Isolates prompt-state exposure relative to the embedding-only variant; its retrieval policy is unchanged.
Recent-only memory ( mars_recentctx_bm )	Replace QTM/semantic retrieval with recent-only top-10 memory	Tests whether recency alone is sufficient.
No-QTM semantic ( mars_noqtm_semctx )	Disable QTM; use semantic memory and query-embedding router features	Stress-tests the framework without QTM state.
No-QTM recent ( mars_noqtm_recentctx )	Disable QTM; use recent-only memory and query-embedding router features	Tests the weakest no-QTM recency-based setting.

Appendix

Table 17: Ablation variants on the three-persona RealMem evaluation slice. All variants use deterministic generation and the same model pool; historical retrieval uses a matched top-10 budget unless otherwise noted.

Query / setting	Required memory	Baseline pattern	MARS pattern	Error type
Present-tense continuation constraint	Preserve a previously imposed grammar/style rule across turns.	Several full-context baselines lose the active rule.	Maintains the constraint and returns a passing final verdict.	Instruction retention
School memory recall under distractors	Recall that a schoolyard feature was located near a sanitation hazard.	Baselines deny the specific detail or lose path continuity.	Recovers the relevant detail and answers consistently.	Inference memory
Three-paragraph revision limit	Preserve a local document-format constraint while editing a later version.	Baselines drift on the paragraph-count requirement.	Keeps the active artifact state and satisfies the editing constraint.	Reliable version editing

Appendix

Table 22: Representative anonymized MultiChallenge patterns summarized from fixed final-evaluation outputs. They are illustrative rather than additional quantitative evidence.

Failure case	Relevant memory evidence	Static-system behavior	MARS behavior	Diagnosis
Travel-plan follow-up after frustration	July travel windows, five adult travelers, and comfort-focused flight constraints.	Gives generic or conflicting travel guidance.	Recovers the travel constraints and responds in the correct planning frame.	Relevant-memory miss
Broken workout-video link	A recurring 9 PM back-care routine and selected exercise playlist.	Gives generic troubleshooting or ignores the routine context.	Uses the routine and exercise context to provide a targeted next step.	Context ignored
Story-development continuation	The “Golden Hook” is locked in and the next component is character development.	Pivots to a generic plot-structure step.	Continues from the locked story state and focuses on the intended component.	Version-state drift

Appendix

Table 23: Representative anonymized RealMem static-configuration error patterns; diagnoses are illustrative and descriptive, not causal.

Item	Reproducibility information
Code and configs	An implementation package will be released with system configurations, prompt templates, and evaluation scripts.
RealMem split	RealMem evaluation pool with 10 personas and 1,415 queries; QA is normalized as official raw QA divided by 3.
MultiChallenge split	Converted MultiChallenge split with 245 scenarios and 1,480 turns; chunk size 3; target subset consists of Inference Memory and Reliable Version Editing .
Ablation slice	Three-persona RealMem evaluation slice with 473 turns (472 valid QA evaluations for the full variant); personas sampled with seed 28; matched top-10 historical retrieval budget.
Candidate model pools	Main RealMem runs use the configured MARS candidate pool reported in Appendix B.4 ; ablations use Llama3B, Qwen2.5 72B, and GPT-4o.
QTM and router settings	Main QTM uses qwen/qwen-2.5-7b-instruct ; its prompt, schema, guardrails, and state-to-memory mapping are fixed. The controller-swap diagnostic additionally evaluates Mistral-7B-Instruct-v0.3 with the same original prompt/schema. Only the router updates online in the main system.

Appendix

Table 25: Reproducibility checklist. The release package will provide the implementation, configurations, prompt templates, and evaluation scripts.

Memory Management for LLM Agents

See all 58 papers

MemRouter: Memory-as-Embedding Routing for Long-Term Conversational Agents
May 1, 2026Tianyu Hu, Weikai Lin, Weizhi Zhang +2Persistent Memory for Language Models Long-Term Conversational Memory
University of Central Florida · University of Rochester · University of Illinois at Chicago +1
LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory
Jul 17, 2026Jing Yu, Yibo Zhao, Jiaming Zhang +1Persistent Memory for Language Models Long-Term Conversational Memory
School of Data Science and Engineering, East China Normal University
Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agents
Aug 21, 2026Quang Dao, Purvi Kathalkar, Kenneth EatonLLM Agent Memory Memory Management for LLM Agents
Rose-Hulman Institute of Technology · Georgia Institute of Technology · Georgia Tech Research Institute

May 1, 2026J/K move · Enter open · S save

MemRouter: Memory-as-Embedding Routing for Long-Term Conversational Agents

Tianyu Hu, Weikai Lin, Weizhi Zhang, Jing Ma+1

University of Central Florida · University of Rochester · University of Illinois at Chicago+1

Persistent Memory for Language Models Long-Term Conversational Memory Agent Memory Management

Open paper AskPDF arXiv

Long-term conversational agents must decide which turns to store in external memory, yet recent systems rely on autoregressive LLM generation at every turn to make that decision. We present MemRouter, a write-side memory router that decouples memory admission from the downstream answer backbone and replaces per-turn memory-management decoding with an embedding-based routing policy. MemRouter encodes each turn together with recent context, projects the resulting embeddings through a frozen LLM backbone, and predicts whether the turn should be stored using lightweight classification heads while training only 12M parameters. Under a controlled matched-harness comparison on LoCoMo, where the retrieval pipeline, answer prompts, and QA backbone (Qwen2.5-7B) are held identical, MemRouter outperforms an LLM-based memory manager on every question category (overall F1 52.0 vs 45.6, non-overlapping 95% CIs) while reducing memory-management p50 latency from 970ms to 58ms. Descriptive factorial averaging further shows that learned admission improves mean F1 by +10.3 over random storage, category-specific prompting adds +5.2 over a generic prompt, and retrieval contributes +0.7. These results suggest that write-side memory admission can be learned by a small supervised router, while answer generation remains a separate downstream component in long-horizon conversational QA.

Abstract

Memory Management for LLM Agents

Explore similar work

MemRouter: Memory-as-Embedding Routing for Long-Term Conversational Agents

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agents