MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents

Yu Liu; Wenxiao Zhang; Cheng Hu; Cong Cao; Fangfang Yuan; Xinyu Wang; Jin B. Hong; Yanbing Liu

cs.CLAug 25, 2026

MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents

Authors: Yu Liu, Wenxiao Zhang, Cheng Hu, Cong Cao, Fangfang Yuan, Xinyu Wang, Jin B. Hong, Yanbing Liu

Organizations: Institute of Information Engineering, Chinese Academy of Sciences Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences Beijing, China · Department of Computer Science and Software Engineering, The University of Western Australia Perth, Australia

Abstract

Multimodal large language model (MLLM) agents are increasingly used as personal assistants for long-running tasks. Their utility depends on continuity: agents must retrieve and use earlier evidence across dialogue, files, and workspace state. However, agents can generate plausible answers even when access to that history has degraded, causing outcome-only evaluation to overestimate true evidence use. We present MIRAGE (Multimodal Interaction Retrieval, Attribution, and Grounding Evaluation), a controlled study of historical evidence use under conversation-state variation in multimodal personal agents. MIRAGE holds evidence objects, questions, and scoring fixed while varying only conversation state, and evaluates whether an agent can determine answerability, recover the correct source, and answer from it. Across seven frontier and open-weight multimodal backbones, we find that: 1) pre-compaction depth and post-compaction continuation form distinct, non-monotonic failure regimes rather than a single degradation curve; 2) open-weight models rely heavily on context continuity and are reluctant to spontaneously switch to tool-mediated retrieval when provenance fails; and 3) retrieval pressure improves source attribution in deep pre-compaction states for tool-compliant models, but consistently regresses after compaction, where stored evidence has already degraded. These findings show that historical evidence use should be evaluated under state variation, rather than inferred from outcome-only correctness.

Figures & tables

Figure 1. Same query, same answer, different grounding. After context compaction (4), one reasoning path retrieves the source document via a memory tool (7a); the other guesses from residual context (7b). Both produce the same final answer (8), making the grounding failure invisible to outcome-only evaluation.

Figure 2. MIRAGE evaluation pipeline. Left : a trunk conversation plants multimodal evidence objects, then filler turns deepen context through four checkpoints (S1-d0, S1-d50k, S1-d80k, S2). Center : the same probe question is issued at each state to a personal agent with tool access. Right : responses are evaluated on a three-stage grounding chain (answerability, source attribution, answer correctness), producing state-conditioned diagnostic profiles.

Model	State	AC	SC	VC	GC	HR	WS	PF
Frontier API (C0 only)
GPT-5 ( OpenAI, 2025 )	S1-d0	85.1	93.9	85.7	85.7	22.7	6.1	5
	S1-d50k	84.2	93.9	83.7	83.7	24.5	6.1	4
	S1-d80k	83.9	93.9	83.8	82.8	25.0	6.1	1
	S2	88.4	97.0	85.9	84.8	19.2	3.0	2
Claude-4.5-Haiku ( Anthropic, 2025 )	S1-d0	80.7	94.9	46.5	46.5	30.6	5.1	3

Table 1. Baseline ( C0 ) metrics across all seven backbones and four conversation states. Metric columns are percentages; PF is a count. The gap VC

-

GC reveals overestimation by outcome-only evaluation.

Figure 3. Mirage Index (MI; Section 3.5 ) across models and states. Higher values indicate a larger fraction of recorded answerable calls for which the model claims answerability without achieving grounded correctness.

Figure 4. Evidence-grounding transitions across states for all seven models. Numbers show counts over answerable probes shared across the three states; blue denotes grounded outputs, and pink denotes ungrounded or unparseable outputs.

					OG (per state)
Model	DS	$\Delta_{\mathrm{comp}}$	Ret	$\mathrm{MI}_{d80k}$	S1-d0	S1-d50k	S1-d80k	S2
GPT-5 ( OpenAI, 2025 )	3.4	$+$ 0.02	94.0	10.0	0.00	0.00	0.01	0.01
Claude-4.5-Haiku ( Anthropic, 2025 )	11.8	$-$ 0.05	73.9	39.0	0.00	0.00	0.01	0.04
Qwen3-VL-4B ( Bai et al., 2025 )	98.1	$+$ 0.33	1.9	26.0	0.07	0.13	0.04	0.10
Qwen3-VL-8B ( Bai et al., 2025 )	74.2	$+$ 0.26	21.7	78.0	0.02	0.06	0.29	0.02
Qwen3-VL-30B ( Bai et al., 2025 )	43.1	$-$ 0.03	51.4	9.0	0.02	0.01	0.03	0.02

Table 2. Cross-state diagnostic indicators under C0 . DS, Ret, and

\mathrm{MI}_{d80k}

are percentages;

\Delta_{\mathrm{comp}}

and OG are proportions.

Figure 5. Retrieval-path composition under C0 and Cm . Percentages use all recorded probe calls in each model–condition–state cell; blue/pink encode C0 / Cm , and solid/hatched encode

R_{\text{context}}

R_{\text{tool}}

. Frontier models have C0 only.

Figure 6. Effect of retrieval pressure, shown as

\Delta(\texttt{Cm}-\texttt{C0})

in percentage points. Top: GC/SC; bottom: AC/HR. Grey marks tool non-compliance or parse failure; “—” denotes unevaluable cells.

Case	Cond.	State	Ans.	Source	Answer	GC
Case 1: Mirage (Qwen3-VL-8B, “What is the username in the fitness profile?”)
	C0	S1-d0	YES	profile…json	leelin	1
	C0	S1-d80k	YES	NONE	leelin	0
	C0	S2	YES	profile…json	leelin	1
Case 2: Cm boundary (Qwen3-VL-30B, GDP query, gold: cqa_9f9df328 )
	C0	S1-d80k	YES	NONE	NONE	0

Table 3. Two probe-level case studies: Case 1 is an outcome-preserving mirage; Case 2 shows that Cm repairs S1-d80k but not S2 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Evidence set	S1-d0	S1-d50k	S1-d80k	S2
Qwen3-VL-4B	Original	71	32	3	52
Qwen3-VL-4B	Extension	46	14	16	23
Qwen3-VL-8B	Original	90	78	27	79
Qwen3-VL-8B	Extension	51	33	15	35
Qwen3-VL-30B	Original	82	72	47	53
Qwen3-VL-30B	Extension	39	23	12	25

Appendix

Table 4. SC across the original chart/UI evidence and six additional evidence types under C0 . The original set has 100 answerable probes and the extension has 55; values are percentages.

Answerer	Verbatim	Free form	Per fact citations
Qwen3-VL-4B	52 / 34	63 / 22	64 / 24
Qwen3-VL-8B	79 / 44	94 / 44	95 / 47
Qwen3-VL-30B	53 / 38	48 / 31	52 / 34

Appendix

Table 5. SC/GC at S2 under C0 for three GPT-5 summary formats. Verbatim is the paper default; each cell reports percentages over 100 answerable probes.

Backbone	ATM-Bench oracle	MIRAGE under state variation
Qwen3-VL-8B	QS 77.8 (Hard 47.3)	S1-d80k : joint-event MI 0.78; separate OG 0.29
GPT-5	QS 85.3 (Hard 74.7)	MI $\leq 0.11$ and OG $\leq 0.01$ across states

Appendix

Table 6. Existing-benchmark scores versus MIRAGE’s state-conditioned diagnostic gap for shared backbones.

Axis	Memory Arena	ATM-Bench	MIRAGE
Conversation state as sole controlled variable	no	no	yes
Per-claim source attribution (SC)	no	partial	yes
Pre/post-compaction regime separation	no	no	yes
Planted multimodal evidence	no	partial	yes
Diagnoses mechanism vs. outcome only	partial	partial	yes

Appendix

Table 7. Conceptual comparison with the closest memory benchmarks.

Backbone/family	Window	d80k	DS	Ret
GPT-5	400K	20%	3.4	94.0
Claude-4.5-Haiku	200K	40%	11.8	73.9
Gemma-3-27B	128K	63%	48.6	42.1
Qwen3-VL-4B	256K	31%	98.1	1.9
Qwen3-VL-8B	256K	31%	74.2	21.7
Qwen3-VL-30B	256K	31%	43.1	51.4

Appendix

Table 8. Depth normalization and within-family size trend.

Cell	Count	Rate	95% CI
Qwen3-VL-30B GC, S1-d80k	41/100	41.0%	[31.9, 50.8]
Qwen3-VL-4B GC, S1-d80k	1/100	1.0%	[0.2, 5.4]
Gemma-3-27B SC, S1-d80k	70/99	70.7%	[61.1, 78.8]
GPT-5 GC, S2	84/99	84.8%	[76.5, 90.6]

Appendix

Table 9. Representative single-run Wilson 95% confidence intervals.

Multimodal Agents

Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.

See all 387 papers

When History Helps and Hurts: Selective History Use across Multimodal Turns
Oct 8, 2026Shuoyang Sun, Kerui Gu, Hao Fang +2Multi-Turn Dialogue Evaluation Multimodal Model Evaluation
Harbin Institute of Technology, Shenzhen · AgiBot · Tsinghua Shenzhen International Graduate School, Tsinghua University
LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger
Jul 30, 2026Enjun Du, Hange Zhou, Chenxu Du +4Multimodal Grounding Multimodal QA
The Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · Tsinghua University +1
M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
Jun 5, 2026Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6Multimodal Understanding Multimodal Grounding
The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4

Oct 8, 2026J/K move · Enter open · S save

When History Helps and Hurts: Selective History Use across Multimodal Turns

Shuoyang Sun, Kerui Gu, Hao Fang, Shaoli Huang+1

Harbin Institute of Technology, Shenzhen · AgiBot · Tsinghua Shenzhen International Graduate School, Tsinghua University

Multi-Turn Dialogue Evaluation Multimodal Model Evaluation Multimodal QA

Open paper AskPDF arXiv

Reliable multimodal interaction depends on selective use of conversational history: an earlier question may remain relevant while its previous answer is outdated, whereas a current request may depend on historical evidence despite conflicting new observations. Existing multi-turn evaluations rarely separate these history-use demands from underlying question difficulty. To address this gap, we introduce ReTurn, a benchmark of 7,000 base tasks spanning visual and audio evidence for evaluating selective history use. For task-carrying history, Reconfirm/Reground require applying a historical question to current media while varying historical agreement; for evidence-carrying history, Retrieve/Rebind require answering a current question using historical media while varying current-media competition. Each pair preserves the target question, media, and answer. Tasks support open-ended and multiple-choice evaluation, with matched single-turn counterparts serving as answerability references. Across 13 omni-modal, vision-language, and audio-language models, median model-level open-ended accuracy falls from 93.7% with direct input to 72.3% in conversation. Behavioral probes show that high question recall can coexist with weaker task application, while competing media can redirect answers away from historical targets. Supervised adaptation yields only partial gains. ReTurn provides a controlled framework for assessing whether multimodal models select and use the historical information required by each request.

Abstract

Multimodal Agents

Explore similar work

When History Helps and Hurts: Selective History Use across Multimodal Turns

LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger

M3^33Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions