cs.AIMar 30, 2026
SaveMiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome
Abstract
Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.
Figures & tables
Figure 1: Model performance comparison on 70 text-only deep research tasks across three dimensions.
Figure 2: Overview of query construction pipeline.
Figure 3: Overview of the query distribution. (a) Joint distribution of 12 domains and 10 task types. (b) Domain distribution. (c) Task type distribution. N = 100 (70 text-only, 30 multimodal).
| Stage | Count | Retention |
| User-Derived Path | ||
| Internal testing patterns rewritten queries | 65 | — |
| Auto-Generated Path | ||
| Trend-grounded generation | 180 | — |
| + Search validation | 152 | 84.4% |
| + Deep-research necessity | 96 | 63.2% |
Table 1: Benchmark construction statistics. User-derived queries are fully rewritten from patterns observed during internal testing; auto-generated queries are produced by trend-grounded generation with three-stage filtering.
| Metric | User-Derived | Auto-Generated |
| Fleiss’ (validity) | 0.83 | 0.79 |
| Fleiss’ (non-triviality) | 0.78 | 0.74 |
| Majority-vote precision | 94.0% | 90.0% |
| Unanimous agreement | 86.0% | 82.0% |
| Aggregated | ||
| Fleiss’ (validity) | 0.81 | |
Table 2: Human verification results. Three annotators assess validity and non-triviality.
Figure 4: Overview of the evaluation pipeline.
| Model | Text-Only | MultiModal | Overall | ||||||
| Synthesis | Factuality | Process | Overall | Synthesis | Factuality | Process | Overall | ||
| Kimi-K2.5 Deep Research | 75.7 | 65.4 | 64.2 | 68.4 | – | – | – | – | – |
| Doubao Deep Research | 64.2 | 64.9 | 53.1 | 60.7 | – | – | – | – | – |
| Grok Deep Research | 58.7 | 63.7 | 58.3 | 60.2 | 56.3 | 71.5 | 53.9 | 60.5 | 60.3 |
| Qwen-3.5-Plus Deep Research | 60.0 | 73.1 | 61.1 | 64.7 | 44.6 | 69.9 | 53.8 | 56.1 | 62.1 |
| Manus-1.6-Max Wide Research | 55.4 | 72.6 | 64.2 | 64.0 | 54.3 | 70.0 | 61.8 | 62.0 | 63.4 |
Table 3: Performance comparison of models with MiroEval.
| Model | Synthesis | Factuality | Overall | |||||||||
| Cov. | Insight | Instr. | Clarity | Spec. | Avg | Right | Wrong | Conf. | Unk. | Ratio | ||
| Text-Only (70 Tasks) | ||||||||||||
| Grok Deep Research | 67.3 | 56.3 | 74.9 | 64.7 | 51.1 | 58.7 | 1924 | 368 | – | 699 | 63.7 | 61.2 |
| Manus-1.6-Max Wide Research | 61.2 | 54.8 | 67.9 | 65.6 | 48.1 | 55.4 | 1972 | 191 | – | 459 | 72.6 | 64.0 |
| Doubao Deep Research | 72.9 | 62.7 | 74.6 | 67.2 | 58.2 | 64.2 | 3890 | 780 | – | 1393 | 64.9 | 64.6 |
| ChatGLM Agent | 69.9 | 62.8 | 74.5 | 67.5 | 57.1 | 63.2 | 4096 | 580 | – | 981 | 68.6 | 65.9 |
Table 4: Combined evaluation on Synthesis quality and Factuality. Report is assessed across five dimensions (Coverage, Insight, Instruction-following, Clarity, and Query Specification). Factuality is measured by the average right ratio (scaled to ). Overall is the average of Report Avg and Factuality Ratio. Text-Only comprises 70 tasks and Multimodal comprises 30 tasks.
Figure 5: Relationship between synthesis quality, factuality, and statement-level precision across different systems. Left: synthesis quality vs. factuality score. Right: total number of generated statements vs. right ratio. Each point represents a system. The gray dashed lines denote linear regression fits, illustrating a weak positive correlation between synthesis quality and factuality, and a negative correlation between statement volume and precision.
| Model | Intrinsic | Alignment | Overall | ||||||||
| Brdth | Depth | Refin | Critl | Effic | Avg | P R | R P | Contr | Avg | ||
| Text-Only (70 Tasks) | |||||||||||
| Doubao Deep Research | 59.3 | 41.6 | 59.6 | 55.7 | 53.3 | 53.9 | 65.7 | 36.8 | 54.2 | 52.2 | 53.1 |
| Grok Deep Research | 50.9 | 49.4 | 61.0 | 54.6 | 64.7 | 56.1 | 74.6 | 42.2 | 64.6 | 60.4 | 58.3 |
| Qwen-3.5-Plus Deep Research | 74.4 | 64.1 | 75.0 | 74.1 | 63.2 | 70.2 | 59.6 | 39.7 | 56.9 | 52.1 | 61.1 |
| Manus-1.6-Max Wide Research | 62.8 | 58.4 | 60.6 | 53.5 | 68.8 | 60.8 | 75.1 | 51.3 | 76.3 | 67.6 | 64.2 |
Table 5: Process evaluation results. Intrinsic metrics assess the quality of the research process itself across five dimensions (Search Breadth, Analytical Depth, Progressive Refinement, Critical Thinking, and Efficiency). Alignment metrics measure the consistency between the research process and the final report (Findings Report coverage, Report Process traceability, and Contradiction detection). Overall is the weighted average of Intrinsic Avg and Alignment Avg.
| Model | User-Derived | Auto-Generation | Overall | ||||||
| Synthesis | Factuality | Process | Overall | Synthesis | Factuality | Process | Overall | ||
| Grok Deep Research | 59.8 | 65.6 | 63.4 | 62.9 | 57.8 | 62.0 | 53.4 | 57.7 | 60.3 |
| Doubao Deep Research | 63.2 | 60.8 | 47.9 | 57.3 | 65.2 | 68.8 | 58.1 | 64.0 | 60.7 |
| Manus-1.6-Max Wide Research | 57.1 | 67.9 | 64.6 | 63.2 | 53.7 | 77.2 | 63.9 | 64.9 | 64.1 |
| Qwen-3.5-Plus Deep Research | 57.9 | 70.3 | 58.7 | 62.3 | 62.1 | 75.8 | 63.5 | 67.1 | 64.7 |
| ChatGLM Agent | 61.1 | 62.9 | 64.5 | 62.9 | 65.3 | 74.0 | 66.7 | 68.7 | 65.8 |
Table 6: Performance comparison of models on user derived query and auto generation query.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| System | Text-Only | Multimodal | Overall |
| OpenAI Deep Research | 17,669 | 12,751 | 16,194 |
| MiroThinker-H1 | 20,442 | 11,802 | 17,850 |
| MiroThinker-1.7 | 21,138 | 11,293 | 18,185 |
| MiroThinker-1.7-mini | 21,823 | – | – |
| Qwen-3.5-Plus Deep Research | 24,299 | 9,081 | 19,734 |
| Manus-1.6-Max Wide Research | 10,263 | 5,585 | 8,860 |
Table 7: Average report length across different systems.
| Feature | Definition |
| Goal adherence | Whether the system maintains focus on all specified goals and constraints throughout a multi-step task without deviating from original objectives or silently dropping sub-tasks. |
| Repetition avoidance | Whether the system avoids repeating the same information or analysis across different sections of its output when the query contains multiple similar but distinct sub-tasks. |
| Planning | Whether the system can decompose a complex query into a coherent, logically ordered sequence of execution steps with clear dependencies between stages. |
| Search | Whether the system can formulate effective search queries and retrieve relevant external information, rather than relying solely on parametric knowledge or the provided attachments. |
| Report generation | Whether the system can organize retrieval results into a well-structured, logically coherent report (comparison tables, analytical summaries, or recommendation lists) that synthesizes information from multiple sources. |
| Factuality | Whether factual claims in the system’s output are accurate and verifiable against authoritative sources, with proper citation where appropriate. |
Table 8: Definitions of the 8 evaluation features.
| ID | Difficulty | Target Features | Description |
| A | Easy | search, multimodal understanding | Extract 1–2 key points from the attachment, perform one round of retrieval for supplementary context, and generate a concise response. Requires attachments. |
| B | Medium | planning, search, report generation, factuality, repetition avoidance | Compare attachment data against at least 2 external public sources and produce a structured comparative analysis report. Requires attachments. |
| C | Hard | factuality, error correction, multimodal understanding | Embed contradictions between the query text and attachment content (e.g., numerical discrepancies, date misalignment). The system must discover the inconsistency through reading the attachment and/or retrieval. Requires high-density attachments. |
| D | Hard | error correction, goal adherence | Embed false premises or ambiguous expressions in the query. The system should identify the erroneous premise, correct it, and still complete the core task. |
| E | Medium / Hard | planning, search, report generation, goal adherence, repetition avoidance | Multi-step research query with no attachment dependency. Answers must be synthesized from multiple public sources through iterative retrieval. |
| F | Easy / Medium | multimodal understanding, report generation | Primary focus on attachment processing: structured extraction, summarization, format conversion, or cross-page synthesis. Retrieval is auxiliary only. Requires attachments. |
Table 9: The 6 rewrite strategies used for user-derived query curation. “Requires attachments” indicates the strategy is excluded for text-only queries during routing.
| # | Topic | Subtopics |
| 1 | AI Policy & Regulation | EU AI Act implementation; US state AI laws; AI safety frameworks |
| 2 | Cybersecurity | Zero-day exploits; Agentic SOC; AI-powered social engineering |
| 3 | Finance & Macro | Central bank policy; Sovereign debt; Infrastructure investment |
| 4 | Crypto & Digital Assets | Stablecoin regulation; DeFi compliance; CBDC adoption |
| 5 | Healthcare & Pharma | Gene therapy trials; GLP-1 market dynamics; FDA regulatory shifts |
| 6 | International Trade | Global supply chain restructuring; Free trade agreements impact; Cross-border regulatory harmonization |
Table 10: Topic taxonomy for automated query generation: 12 topics, each with 3 subtopics.
| # | Label | Scope |
| 1 | finance | Financial markets, investment analysis, banking, macroeconomics, corporate earnings, and economic data. |
| 2 | policy | Government policy, regulation, governance, and institutional rule-making at local, national, and international levels. |
| 3 | tech | Technology, software engineering, AI/ML, hardware, and internet products. Also serves as the default fallback label. |
| 4 | cybersecurity | Digital security threats, defenses, vulnerability research, threat intelligence, and security operations. |
| 5 | health | Healthcare, medicine, pharmaceuticals, clinical research, medical devices, and public health. |
| 6 | science | Natural sciences, academic research, engineering, mathematics, and research infrastructure. |
Table 11: The 11 canonical domain labels.
| Model | Synthesis | Factuality | Process | Overall | Avg | Std | ||||||||
| R1 | R2 | R3 | R1 | R2 | R3 | R1 | R2 | R3 | R1 | R2 | R3 | |||
| OpenAI Deep Research | 66.7 | 66.5 | 66.7 | 77.0 | 75.0 | 76.9 | 66.8 | 65.0 | 66.2 | 70.2 | 68.8 | 69.9 | 69.6 | 0.6 |
| Gemini-3.1-Pro | 66.4 | 66.3 | 66.9 | 73.7 | 70.0 | 72.7 | 64.1 | 63.0 | 62.8 | 68.1 | 66.4 | 67.5 | 67.3 | 0.6 |
| MiroThinker-1.7 | 69.0 | 68.7 | 68.6 | 78.5 | 80.9 | 79.6 | 67.4 | 67.7 | 67.4 | 71.6 | 72.4 | 71.9 | 72.0 | 0.3 |
| MiroThinker-H1 | 71.5 | 72.0 | 70.8 | 78.4 | 76.2 | 77.4 | 73.5 | 73.6 | 73.3 | 74.5 | 73.9 | 73.8 | 74.1 | 0.3 |
Table 12: Intra-judge stability on MultiModal (30 tasks). Three independent runs with the same GPT judge configuration. Each dimension reports scores from Run 1 / Run 2 / Run 3, followed by the mean and standard deviation.
| Model | Synthesis | Factuality | Process | Overall | Rank |
| Cross-Judge: Text-Only (70 Tasks) — GPT Series (orig.) / Gemini Series (alt.) / | |||||
| OpenAI Deep Research | 73.8 / 90.2 / +16.4 | 83.3 / 89.3 / +6.0 | 73.1 / 90.2 / +17.1 | 76.7 / 89.9 / +13.2 | 0 |
| Gemini-3.1-Pro | 71.2 / 89.5 / +18.3 | 71.3 / 81.8 / +10.5 | 67.1 / 87.9 / +20.9 | 69.9 / 86.4 / +16.5 | 0 |
| ChatGLM Agent | 63.2 / 82.2 / +19.0 | 68.6 / 80.3 / +11.7 | 65.6 / 85.7 / +20.1 | 65.8 / 82.7 / +16.9 | 0 |
| MiroThinker-1.7-mini | 74.0 / 90.3 / +16.3 | 76.2 / 86.2 / +10.0 | 68.5 / 89.8 / +21.3 | 72.9 / 88.8 / +15.9 | 0 |
| MiroThinker-1.7 | 74.3 / 90.8 / +16.5 | 79.4 / 87.6 / +8.2 | 72.7 / 91.0 / +18.3 | 75.5 / 89.8 / +14.3 | 0 |
Table 13: Robustness to judge model choice and prompt variation. Each cell reports original / alternative / . Rank: rank change based on Overall score. Upper : Cross-judge consistency on Text-Only (70 tasks), GPT vs. Gemini. Lower : Prompt sensitivity on MultiModal (30 tasks), original vs. modified prompt with the same GPT judge.
| System | Human | MiroEval | Rank |
| MiroThinker-H1 | 1.8 | 1 | |
| OpenAI Deep Research | 2.5 | 2 | |
| MiroThinker-1.7 | 2.8 | 3 | |
| Claude-Opus-4.6 Research | 5.2 | 5 | |
| Gemini-3.1-Pro Deep Research | 5.3 | 4 | |
| MiniMax-M2.5 Research | 6.0 | 6 |
Table 14: Comparison between human rankings and MiroEval rankings. Human rankings are averaged across 5 annotators. Rank: positive values indicate higher human ranking than MiroEval.