VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents

Yuhao Chen; Yi Xu; Xinyun Ding; Xiang Fang; Shuochen Liu; Luxi Lin; Qingyu Zhang; Ya Li

cs.AIMar 25, 2026

VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents

Authors: Yuhao Chen, Yi Xu, Xinyun Ding, Xiang Fang, Shuochen Liu, Luxi Lin, Qingyu Zhang, Ya Li,

Organizations: University of Science and Technology of China · iFLYTEK Research · Xiamen University · University of Chinese Academy of Sciences

Abstract

With the growing demand for intelligent in-vehicle experiences, vehicle-based agents are evolving from simple assistants to long-term companions. This evolution requires agents to continuously model multi-user preferences and make reliable decisions in the face of inter-user preference conflicts and changing habits over time. However, existing benchmarks are largely limited to single-user, static question-answer settings, failing to capture the temporal evolution of preferences and the multi-user, tool-interactive nature of real vehicle environments. To address this gap, we introduce VehicleMemBench, a multi-user long-context memory benchmark built on an executable in-vehicle simulation environment. The benchmark evaluates tool use and memory by comparing the post-action environment state with a predefined target state, enabling objective and reproducible evaluation without LLM-based or human scoring. VehicleMemBench includes 23 tool modules, and each sample contains over 80 historical memory events. Experiments show that powerful models perform well on direct instruction tasks but struggle in scenarios involving memory evolution, particularly when user preferences change dynamically. Even advanced memory systems struggle to handle domain-specific memory requirements in this environment. These findings highlight the need for more robust and specialized memory management mechanisms to support long-term adaptive decision-making in real-world in-vehicle systems. To facilitate future research, we release the data and code.

Figures & tables

Figure 1: Overview of the VehicleMemBench pipeline. The framework constructs multi-user long-term memory scenarios through persona generation, event chain construction, and temporal interleaving, followed by dialogue and QA generation. These are integrated into an executable in-vehicle simulation environment with tool-based interactions, enabling state-based evaluation of agent memory and tool use capabilities.

Model	Recursive Summarization				Key Value Store				Gold Memory
Model	ESM	F-F1	V-F1	Calls	ESM	F-F1	V-F1	Calls	ESM	F-F1	V-F1	Calls
MiniMax-M2.1	50.00	72.41	67.16	2.43	30.40	49.26	43.29	3.93	74.00	86.32	84.55	2.16
MiniMax-M2.5	52.20	74.54	69.48	2.61	32.60	49.67	45.12	3.81	77.20	87.88	86.08	2.25
Kimi-K2	53.00	79.83	72.64	3.34	43.60	71.31	64.73	5.99	82.80	91.59	89.68	2.29
Kimi-K2.5	56.40	79.46	75.56	3.23	44.00	70.66	63.76	6.40	81.00	91.32	89.10	2.29
Qwen3-Max	60.60	82.20	77.33	2.87	46.60	73.74	66.26	5.99	83.60	92.73	90.55	2.15

Table 1: Overall task performance on VehicleMemBench under different memory construction paradigms. The best results are highlighted in bold , and the second-best results are underlined .

Panel of Figure 2 — Figure 2: Model performance and memory efficiency on VehicleMemBench. ( Left ) Exact State Match across different backbone models under Gold Memory and Recursive Summarization settings. ( Right ) Trade-off between Memory Score and retrieval cost (memory tokens per retrieval) for different memory systems with Gemini-3-Pro and Qwen3-Max backbones.

Model	Memory System	Exact State Match					Overall
Model	Memory System	Pref. Conflict	Coref. Res.	Cond. Const.	State Shift	Err. Corr.	ESM	Mem Token
Gemini-3-Pro Preview	Gold Memory	92.62	92.78	87.25	90.00	88.71	90.60	93.29
	Rec. Sum.	67.79	61.86	62.75	62.22	69.35	64.80	303.25
	Key Value Store	64.93	52.38	52.13	49.38	67.27	57.37	29.25
	Memobase	32.21	24.74	30.39	32.22	24.19	29.40	993.49
	LightMem	55.70	68.75	59.80	61.11	61.29	60.72	170.19

Table 2: ESM and Mem Token of different memory systems on VehicleMemBench.

Panel of Figure 3 — Figure 3: Failure mode distribution across memory systems. ( Left ) Total number of error cases for each memory system, categorized into four error types. ( Right ) error composition under the Conditional Constraint setting, reported as percentages within each system.

Figure 4: Fine-grained error composition by reasoning type and memory system.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Groups	Personas	Queries	Chains	Events/inst.	Delay (d)	History (avg.)
50	150	500	1,500	81.78	23.16	2,690 lines / 92,819 tokens

Appendix

Table 3: Overview statistics of the released VehicleMemBench benchmark.

Reasoning type	Count	Share (%)	Target module	Count	Share (%)
Preference Conflict	149	29.8	Navigation	96	19.2
Conditional Constraint	102	20.4	Seat	85	17.0
Coreference Resolution	97	19.4	Light	64	12.8
State Shift	90	18.0	AirConditioner	64	12.8
Error Correction	62	12.4	InstrumentPanel	42	8.4
			Music	41	8.2

Appendix

Table 4: Distribution of executable queries by reasoning type and target module in the released benchmark. The target-module statistics are computed from the verified reference tool sequences.

Category	Module	# APIs	Representative APIs
Display & Info	HUD	3	switch, set_brightness_level, set_height_level
	CenterInformationDisplay	5	set_power, set_brightness, set_auto_brightness, set_language, set_time_format
	InstrumentPanel	8	set_theme, set_color, set_brightness, set_behavior_mode, set_language
	OverheadScreen	4	switch, set_brightness_level, set_language, set_time_format
Climate & Comfort	AirConditioner	6	set_temperature, set_fan_speed, set_air_direction, set_mode, set_circulation
	Seat	15	set_heating, set_massage, set_ventilation, set_position, set_headrest_height

Appendix

Table 5: Complete catalog of vehicle modules and executable APIs in VehicleMemBench. APIs follow the naming convention carcontrol_{module}_{action}.

Model	Exact State Match					Overall
Model	Pref. Conflict	Coref. Res.	Cond. Const.	State Shift	Err. Corr.	Overall
MiniMax-M2.1	16.78	12.37	17.65	15.56	6.45	14.60
MiniMax-M2.5	16.11	18.56	17.65	17.78	12.90	16.80
Kimi-K2	14.09	11.34	19.61	14.44	9.68	14.20
Kimi-K2.5	20.13	16.49	15.69	14.44	14.52	16.80
Qwen3-Max	20.81	15.46	16.67	16.67	12.90	17.20

Appendix

Table 6: Performance without memory. All models achieve very low Exact ESM, typically below 20, indicating that the task cannot be solved from the query alone and requires long-term memory.

Stage	Model	Temp.	Description
Profile enrichment	Gemini-3-Pro-Preview	0.7	Enriches seed personas with structured attributes and detects preference conflicts.
Event chain generation	Gemini-3-Pro-Preview	0.8	Generates structured event chains covering five reasoning types per user group.
Dialogue generation	GPT-4.1	0.7	Converts structured events into natural multi-turn conversations among occupants.
Answer generation	Gemini-3-Pro-Preview	0.0	Generates reference tool-call sequences and executes them in the simulator for verification.

Appendix

Table 7: Models and hyperparameters used in each stage of the data generation pipeline.

Family	Model	Thinking Mode
Google Gemini	Gemini-3-Pro-Preview	✓
OpenAI	GPT-5	✓
ByteDance	Doubao-Seed-1.6	✓
MiniMax	MiniMax-M2.5	✓
MiniMax	MiniMax-M2.1	✓
Zhipu GLM	GLM-5	✓

Appendix

Table 8: Models evaluated on VehicleMemBench.

Memory System	Type	Description
Recursive Summarization	Scenario-specific	Incrementally compresses daily dialogues into a single text summary of vehicle-related preferences; the final summary is injected into the system prompt at evaluation time.
Key Value Store	Scenario-specific	Maintains an explicit key-value store via memory add/memory remove/memory search tool calls; at evaluation time, the agent queries the store via memory list and memory search.
MemOS	General-purpose	Manages heterogeneous memories through structured memory units (MemCubes) with lifecycle scheduling.
Mem0	General-purpose	Automatically extracts, compresses, and persistently stores key user information for personalization.
LightMem	General-purpose	Lightweight framework organizing information into short-term and long-term memory stages with efficient compression.
Memobase	General-purpose	Incrementally structures and organizes user preferences across multi-turn conversations.

Appendix

Table 9: Memory systems evaluated on VehicleMemBench.

Parameter	Value
Decoding temperature	0.0 (greedy)
Maximum generation tokens	8,192
Maximum tool-call rounds per query	10
Tool choice	auto
API retry attempts on failure	3
History tokenizer (for statistics)	GPT-4o

Appendix

Table 10: Hyperparameters used for all evaluation experiments.

Panel of Figure 5 — Figure 5: Shortened prompt for structured event-chain generation.

Panel of Figure 6 — Figure 6: Shortened prompt for dialogue generation from structured events.

Panel of Figure 7 — Figure 7: Shortened prompt for recursive summary memory construction.

Panel of Figure 8 — Figure 8: Shortened prompt for key-value memory construction.

Panel of Figure 9 — Figure 9: Shortened prompt for memory-augmented online tool-use evaluation.

Memory Agent Benchmarks

Memory agent benchmarks evaluate how AI agents store, retrieve, update, and use information across interactions and changing tasks.

See all 164 papers

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users
Oct 2, 2026Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao +5Agent Memory Memory Agent Benchmarks
Nanyang Technological University · Stepfun · National University of Singapore +1
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
May 12, 2026Di Wu, Zixiang Ji, Asmi Kawatkar +4Computer-Use Agent Benchmarks Retrieval-Augmented Generation
University of California, Los Angeles
DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
Jun 22, 2026Wenya Xie, Shengming Zhou, Zelin Li +9Agent Memory LLM Agent Memory
University of Minnesota · University of North Carolina at Charlotte · Johns Hopkins University +2

Oct 2, 2026J/K move · Enter open · S save

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao, Fanyi Wang+4

Nanyang Technological University · Stepfun · National University of Singapore+1

Agent Memory Memory Agent Benchmarks Long-Term Conversational Memory Agent Memory Management LLM Agent Evaluation

Open paper AskPDF arXiv

Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves. Existing benchmarks primarily supervise user facts and preferences or experience reusable across users, leaving this relationship-specific agent memory implicit. Additionally, most prior works measure the model solely with final-answer QA over long interaction histories, making the assessment still incomplete and unreliable. To this end, we introduce DyadMem with the proposed new definition User-conditioned Relational Agent Memory (URAM). DyadMem jointly annotates user-side memory and URAM along the same multi-session trajectories, resulting in 6 memory categories. To summarize, it includes 3,065 episodes, 50,961 sessions, and 61,210 QA instances, with extensive session-level Capture and Update gold annotations, query-level Recall support, and two QA settings: Gold-Memory and Full-Pipeline. Across 16 open-weight and 4 proprietary models, Gold-Memory QA is consistently strong, yet Full-Pipeline QA drops sharply. Such a gap explicitly supports our fine-grained evaluation design. Additionally, several quantitative results further reveal low Capture recall, incomplete Recall, and unsafe-deletion issues arising from even the frontier LLMs. We further conduct a rigorous experiment to validate the effectiveness of our URAM and observe the positive effects for all 20 models. In summary, DyadMem is a dual-domain, full-pipeline memory benchmark with extensive annotation efforts for advancing the domain's development.

Abstract

Memory Agent Benchmarks

Explore similar work

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings