MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

Fangda Ye; Yuxin Hu; Pengxiang Zhu; Yibo Li; Ziqi Jin; Yao Xiao; Yibo Wang; Lei Wang

cs.AIMar 30, 2026

MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

Authors: Fangda Ye, Yuxin Hu, Pengxiang Zhu, Yibo Li, Ziqi Jin, Yao Xiao, Yibo Wang, Lei Wang, +14 more

Abstract

Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.

Figures & tables

Figure 1: Model performance comparison on 70 text-only deep research tasks across three dimensions.

Figure 2: Overview of query construction pipeline.

Figure 3: Overview of the query distribution. (a) Joint distribution of 12 domains and 10 task types. (b) Domain distribution. (c) Task type distribution. N = 100 (70 text-only, 30 multimodal).

Stage	Count	Retention
User-Derived Path
Internal testing patterns $\rightarrow$ rewritten queries	65	—
Auto-Generated Path
Trend-grounded generation	180	—
+ Search validation	152	84.4%
+ Deep-research necessity	96	63.2%

Table 1: Benchmark construction statistics. User-derived queries are fully rewritten from patterns observed during internal testing; auto-generated queries are produced by trend-grounded generation with three-stage filtering.

Metric	User-Derived	Auto-Generated
Fleiss’ $\kappa$ (validity)	0.83	0.79
Fleiss’ $\kappa$ (non-triviality)	0.78	0.74
Majority-vote precision	94.0%	90.0%
Unanimous agreement	86.0%	82.0%
Aggregated
Fleiss’ $\kappa$ (validity)	0.81

Table 2: Human verification results. Three annotators assess validity and non-triviality.

Figure 4: Overview of the evaluation pipeline.

Model	Text-Only				MultiModal				Overall
Model	Synthesis	Factuality	Process	Overall	Synthesis	Factuality	Process	Overall
Kimi-K2.5 Deep Research	75.7	65.4	64.2	68.4	–	–	–	–	–
Doubao Deep Research	64.2	64.9	53.1	60.7	–	–	–	–	–
Grok Deep Research	58.7	63.7	58.3	60.2	56.3	71.5	53.9	60.5	60.3
Qwen-3.5-Plus Deep Research	60.0	73.1	61.1	64.7	44.6	69.9	53.8	56.1	62.1
Manus-1.6-Max Wide Research	55.4	72.6	64.2	64.0	54.3	70.0	61.8	62.0	63.4

Table 3: Performance comparison of models with MiroEval.

Model	Synthesis						Factuality					Overall
Model	Cov.	Insight	Instr.	Clarity	Spec.	Avg	Right	Wrong	Conf.	Unk.	Ratio	Overall
Text-Only (70 Tasks)
Grok Deep Research	67.3	56.3	74.9	64.7	51.1	58.7	1924	368	–	699	63.7	61.2
Manus-1.6-Max Wide Research	61.2	54.8	67.9	65.6	48.1	55.4	1972	191	–	459	72.6	64.0
Doubao Deep Research	72.9	62.7	74.6	67.2	58.2	64.2	3890	780	–	1393	64.9	64.6
ChatGLM Agent	69.9	62.8	74.5	67.5	57.1	63.2	4096	580	–	981	68.6	65.9

Table 4: Combined evaluation on Synthesis quality and Factuality. Report is assessed across five dimensions (Coverage, Insight, Instruction-following, Clarity, and Query Specification). Factuality is measured by the average right ratio (scaled to

[0,100]

). Overall is the average of Report Avg and Factuality Ratio. Text-Only comprises 70 tasks and Multimodal comprises 30 tasks.

Figure 5: Relationship between synthesis quality, factuality, and statement-level precision across different systems. Left: synthesis quality vs. factuality score. Right: total number of generated statements vs. right ratio. Each point represents a system. The gray dashed lines denote linear regression fits, illustrating a weak positive correlation between synthesis quality and factuality, and a negative correlation between statement volume and precision.

Model	Intrinsic						Alignment				Overall
Model	Brdth	Depth	Refin	Critl	Effic	Avg	P $\to$ R	R $\to$ P	Contr	Avg	Overall
Text-Only (70 Tasks)
Doubao Deep Research	59.3	41.6	59.6	55.7	53.3	53.9	65.7	36.8	54.2	52.2	53.1
Grok Deep Research	50.9	49.4	61.0	54.6	64.7	56.1	74.6	42.2	64.6	60.4	58.3
Qwen-3.5-Plus Deep Research	74.4	64.1	75.0	74.1	63.2	70.2	59.6	39.7	56.9	52.1	61.1
Manus-1.6-Max Wide Research	62.8	58.4	60.6	53.5	68.8	60.8	75.1	51.3	76.3	67.6	64.2

Table 5: Process evaluation results. Intrinsic metrics assess the quality of the research process itself across five dimensions (Search Breadth, Analytical Depth, Progressive Refinement, Critical Thinking, and Efficiency). Alignment metrics measure the consistency between the research process and the final report (Findings

\to

Report coverage, Report

\to

Process traceability, and Contradiction detection). Overall is the weighted average of Intrinsic Avg and Alignment Avg.

Model	User-Derived				Auto-Generation				Overall
Model	Synthesis	Factuality	Process	Overall	Synthesis	Factuality	Process	Overall
Grok Deep Research	59.8	65.6	63.4	62.9	57.8	62.0	53.4	57.7	60.3
Doubao Deep Research	63.2	60.8	47.9	57.3	65.2	68.8	58.1	64.0	60.7
Manus-1.6-Max Wide Research	57.1	67.9	64.6	63.2	53.7	77.2	63.9	64.9	64.1
Qwen-3.5-Plus Deep Research	57.9	70.3	58.7	62.3	62.1	75.8	63.5	67.1	64.7
ChatGLM Agent	61.1	62.9	64.5	62.9	65.3	74.0	66.7	68.7	65.8

Table 6: Performance comparison of models on user derived query and auto generation query.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

System	Text-Only	Multimodal	Overall
OpenAI Deep Research	17,669	12,751	16,194
MiroThinker-H1	20,442	11,802	17,850
MiroThinker-1.7	21,138	11,293	18,185
MiroThinker-1.7-mini	21,823	–	–
Qwen-3.5-Plus Deep Research	24,299	9,081	19,734
Manus-1.6-Max Wide Research	10,263	5,585	8,860

Appendix

Table 7: Average report length across different systems.

Feature	Definition
Goal adherence	Whether the system maintains focus on all specified goals and constraints throughout a multi-step task without deviating from original objectives or silently dropping sub-tasks.
Repetition avoidance	Whether the system avoids repeating the same information or analysis across different sections of its output when the query contains multiple similar but distinct sub-tasks.
Planning	Whether the system can decompose a complex query into a coherent, logically ordered sequence of execution steps with clear dependencies between stages.
Search	Whether the system can formulate effective search queries and retrieve relevant external information, rather than relying solely on parametric knowledge or the provided attachments.
Report generation	Whether the system can organize retrieval results into a well-structured, logically coherent report (comparison tables, analytical summaries, or recommendation lists) that synthesizes information from multiple sources.
Factuality	Whether factual claims in the system’s output are accurate and verifiable against authoritative sources, with proper citation where appropriate.

Appendix

Table 8: Definitions of the 8 evaluation features.

ID	Difficulty	Target Features	Description
A	Easy	search, multimodal understanding	Extract 1–2 key points from the attachment, perform one round of retrieval for supplementary context, and generate a concise response. Requires attachments.
B	Medium	planning, search, report generation, factuality, repetition avoidance	Compare attachment data against at least 2 external public sources and produce a structured comparative analysis report. Requires attachments.
C	Hard	factuality, error correction, multimodal understanding	Embed contradictions between the query text and attachment content (e.g., numerical discrepancies, date misalignment). The system must discover the inconsistency through reading the attachment and/or retrieval. Requires high-density attachments.
D	Hard	error correction, goal adherence	Embed false premises or ambiguous expressions in the query. The system should identify the erroneous premise, correct it, and still complete the core task.
E	Medium / Hard	planning, search, report generation, goal adherence, repetition avoidance	Multi-step research query with no attachment dependency. Answers must be synthesized from multiple public sources through iterative retrieval.
F	Easy / Medium	multimodal understanding, report generation	Primary focus on attachment processing: structured extraction, summarization, format conversion, or cross-page synthesis. Retrieval is auxiliary only. Requires attachments.

Appendix

Table 9: The 6 rewrite strategies used for user-derived query curation. “Requires attachments” indicates the strategy is excluded for text-only queries during routing.

#	Topic	Subtopics
1	AI Policy & Regulation	EU AI Act implementation; US state AI laws; AI safety frameworks
2	Cybersecurity	Zero-day exploits; Agentic SOC; AI-powered social engineering
3	Finance & Macro	Central bank policy; Sovereign debt; Infrastructure investment
4	Crypto & Digital Assets	Stablecoin regulation; DeFi compliance; CBDC adoption
5	Healthcare & Pharma	Gene therapy trials; GLP-1 market dynamics; FDA regulatory shifts
6	International Trade	Global supply chain restructuring; Free trade agreements impact; Cross-border regulatory harmonization

Appendix

Table 10: Topic taxonomy for automated query generation: 12 topics, each with 3 subtopics.

#	Label	Scope
1	finance	Financial markets, investment analysis, banking, macroeconomics, corporate earnings, and economic data.
2	policy	Government policy, regulation, governance, and institutional rule-making at local, national, and international levels.
3	tech	Technology, software engineering, AI/ML, hardware, and internet products. Also serves as the default fallback label.
4	cybersecurity	Digital security threats, defenses, vulnerability research, threat intelligence, and security operations.
5	health	Healthcare, medicine, pharmaceuticals, clinical research, medical devices, and public health.
6	science	Natural sciences, academic research, engineering, mathematics, and research infrastructure.

Appendix

Table 11: The 11 canonical domain labels.

Model	Synthesis			Factuality			Process			Overall			Avg	Std
Model	R1	R2	R3	R1	R2	R3	R1	R2	R3	R1	R2	R3
OpenAI Deep Research	66.7	66.5	66.7	77.0	75.0	76.9	66.8	65.0	66.2	70.2	68.8	69.9	69.6	0.6
Gemini-3.1-Pro	66.4	66.3	66.9	73.7	70.0	72.7	64.1	63.0	62.8	68.1	66.4	67.5	67.3	0.6
MiroThinker-1.7	69.0	68.7	68.6	78.5	80.9	79.6	67.4	67.7	67.4	71.6	72.4	71.9	72.0	0.3
MiroThinker-H1	71.5	72.0	70.8	78.4	76.2	77.4	73.5	73.6	73.3	74.5	73.9	73.8	74.1	0.3

Appendix

Table 12: Intra-judge stability on MultiModal (30 tasks). Three independent runs with the same GPT judge configuration. Each dimension reports scores from Run 1 / Run 2 / Run 3, followed by the mean and standard deviation.

Model	Synthesis	Factuality	Process	Overall	$\Delta$ Rank
Cross-Judge: Text-Only (70 Tasks) — GPT Series (orig.) / Gemini Series (alt.) / $\Delta$
OpenAI Deep Research	73.8 / 90.2 / +16.4	83.3 / 89.3 / +6.0	73.1 / 90.2 / +17.1	76.7 / 89.9 / +13.2	0
Gemini-3.1-Pro	71.2 / 89.5 / +18.3	71.3 / 81.8 / +10.5	67.1 / 87.9 / +20.9	69.9 / 86.4 / +16.5	0
ChatGLM Agent	63.2 / 82.2 / +19.0	68.6 / 80.3 / +11.7	65.6 / 85.7 / +20.1	65.8 / 82.7 / +16.9	0
MiroThinker-1.7-mini	74.0 / 90.3 / +16.3	76.2 / 86.2 / +10.0	68.5 / 89.8 / +21.3	72.9 / 88.8 / +15.9	0
MiroThinker-1.7	74.3 / 90.8 / +16.5	79.4 / 87.6 / +8.2	72.7 / 91.0 / +18.3	75.5 / 89.8 / +14.3	0

Appendix

Table 13: Robustness to judge model choice and prompt variation. Each cell reports original / alternative /

\Delta

\Delta

Rank: rank change based on Overall score. Upper : Cross-judge consistency on Text-Only (70 tasks), GPT vs. Gemini. Lower : Prompt sensitivity on MultiModal (30 tasks), original vs. modified prompt with the same GPT judge.

System	Human	MiroEval	$\Delta$ Rank
MiroThinker-H1	1.8	1	$=$
OpenAI Deep Research	2.5	2	$=$
MiroThinker-1.7	2.8	3	$=$
Claude-Opus-4.6 Research	5.2	5	$\uparrow 1$
Gemini-3.1-Pro Deep Research	5.3	4	$\downarrow 1$
MiniMax-M2.5 Research	6.0	6	$=$

Appendix

Table 14: Comparison between human rankings and MiroEval rankings. Human rankings are averaged across 5 annotators.

\Delta

Rank: positive values indicate higher human ranking than MiroEval.

Explore similar work

Apr 16, 2026

DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation

Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR

^{3}

-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR

^{3}

-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR

^{3}

-Agent based on multiple state-of-the-art language models demonstrate that DR

^{3}

-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.

Qianqian Xie, Qingheng Xiong, He Zhu +16

Nanjing University · M-A-P · Nanjing University of Science and Technology +2

LLM Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks

Ask PDF

Sep 10, 2026

Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents

Deep research agents are increasingly capable of web search, tool use, multimodal evidence analysis, and information synthesis. However, existing benchmarks mainly evaluate medium-horizon exploration and rarely test whether agents can sustain long, dependency-heavy research processes. We introduce Mr. LHDR (Multimodal real-world Long-Horizon Deep Research), a benchmark for evaluating real-world deep research over long, irreducible chains of interdependent evidence across eight categories. Each question is constructed from a hidden Node-Relation graph and requires an average of 12.1 necessary intermediate conclusions with a mean dependency depth of 10.4 before reaching a short, unique, and verifiable answer. Questions incorporate multimodal evidence, including images, maps, PDFs, logos, charts, tables, and video frames, with at least one non-text element that changes the reasoning state. Mr. LHDR evaluates both final answers and the correctness of intermediate conclusions under annotated dependencies. We evaluate general models, deep research systems, and agent frameworks using Overall Accuracy (OA), Strict Accuracy (SA), Checklist Score (CS), and Dependency-Aware Checklist Score (DACS). Results show that even the strongest system achieves only 43.1% OA and 34.3% SA, indicating that final-answer accuracy substantially overestimates complete research success. Removing images reduces DACS by 12.6 points, demonstrating the importance of multimodal evidence, while SA consistently declines as reasoning chains become longer. These findings reveal sustained, dependency-consistent evidence integration, rather than isolated fact retrieval, as a key bottleneck for current deep research agents.

Minghao Guo, Meng Cao, Sui Zhao +10

Mohamed bin Zayed University of Artificial Intelligence · University of Science and Technology of China · Zhejiang University +1

Deep Research Agents Long-Horizon Agent Evaluation Multimodal Reasoning AI Agent Benchmarks

Ask PDF

Mar 26, 2026

From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

Deep Research Agents (DRAs) aim to answer complex questions by searching the web, checking evidence, and synthesizing conclusions across heterogeneous sources. We introduce a category-theoretic framework for evaluating and improving such agents. The framework treats deep research as a structured mapping from user intent to evidence-grounded conclusions, making retrieval traces, cross-source alignment, and final synthesis explicit. Guided by this view, we derive a mechanism-aware benchmark of 296 bilingual questions. The benchmark targets four structural skills central to real research: following multi-hop evidence chains, verifying claims across sources, re-ordering fragmented information, and rejecting unsupported assumptions. We evaluate 16 frontier systems with human verification and find that these structural tasks remain highly challenging: the best system reaches only 19.9% average accuracy. The results show that strong agents can sometimes reorganize evidence and detect false premises, but still struggle with long-horizon synthesis and intersection-heavy verification. Beyond evaluation, the same theory also leads to practical system improvements. We instantiate theory-guided interventions such as tracked search, which preserves retrieval traces, and category tools, which add explicit verification and synthesis steps. These interventions yield measurable gains in API-based deep research systems. Our work therefore provides both a challenging benchmark and concrete design guidance for building more reliable research agents.

Shuoling Liu, Zhiquan Tan, Kun Yi +6

The Hong Kong University of Science and Technology · E Fund Management Co., Ltd. · The Hong Kong Polytechnic University

Deep Research Agents AI Agent Benchmarks Long-Horizon Agent Evaluation

Ask PDF

Abstract

Explore similar work

DR3^{3}3-Eval: Towards Realistic and Reproducible Deep Research Evaluation

Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents

From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation