cs.CLJan 19, 2026
SaveBeyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision
Organizations: New York University · NYU Shanghai · University of Alberta · University of Waterloo ·
Abstract
Existing benchmarks for Deep Research Agents (DRAs) treat report generation as a single-shot writing task, which fundamentally diverges from how human researchers iteratively draft and revise reports via self-reflection or peer feedback. Whether DRAs can reliably revise reports with user feedback remains unexplored. We introduce Mr Dre, an evaluation suite that establishes multi-turn report revision as a new evaluation axis for DRAs. Mr Dre consists of (1) a unified long-form report evaluation protocol spanning comprehensiveness, factuality, and presentation, and (2) a human-verified feedback simulation pipeline for multi-turn revision. Our analysis of five diverse DRAs reveals a critical limitation: while agents can address most user feedback, they also regress on 16-28% of previously covered content and citation quality. Over multiple revision turns, even the best-performing agents leave significant headroom, as they continue to disrupt content outside the feedback's scope and fail to preserve earlier edits. We further show that these issues are not easily resolvable through inference-time fixes such as prompt engineering and a dedicated subagent for report revision.
Figures & tables
Figure 1: Illustrative example of multi-turn revision failure in Deep Research Agents. The revised report incorporates the user feedback but removes previously covered content that is outside the feedback’s scope.
Figure 2: Mr Dre Evaluation Suite. In multi-turn report revision, a DRA iteratively drafts and revises a report for question given user feedback (top left). Mr Dre provides a unified Deep Research report evaluation protocol (bottom left) along three dimensions: Comprehensiveness, Factuality, and Presentation. To evaluate multi-turn revision performance, Mr Dre provides a pipeline to simulate content, format, and self-reflection feedback (right).
| Rubric-based Comprehensiveness | Claim-level Factuality | Report Presentation | Reference-free Evaluation | Multi-turn Revision | |
| ResearchRubrics ( Sharma et al., 2025 ) | ✓ | ✗ | ✗ | ✓ | ✗ |
| RACE&FACT ( Du et al., 2025 ) | ✓ | ✓ | ✓ | ✗ | ✗ |
| Dr. Bench ( Yao et al., 2025 ) | ✓ | ✗ | ✓ | ✓ | ✗ |
| ResearcherBench ( Xu et al., 2025 ) | ✓ | ✗ | ✗ | ✓ | ✗ |
| DR-ReportEval ( Fan et al., 2025 ) | ✗ | ✓ | ✓ | ✗ | ✗ |
| DeepEval ( Wang et al., 2025 ) | ✓ | ✓ | ✓ | ✓ | ✗ |
Table 1: Comparison of Mr Dre and previous Deep Research report evaluation protocols. Mr Dre checks all five aspects: rubric-based comprehensiveness evaluation, claim-level factuality assessment, report presentation scoring, requiring no reference answer, and support for multi-turn report revision (§ 4 ).
Figure 3: Results for extending to 4 turns of revision under Content 1 setting . We report the (top) coverage score (actual vs. oracle), (middle) incorporation rate, and (bottom) break rate.
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Size | # Criteria/ Question | Domains |
| ResearchRubrics | 101 | 25.67 | General |
| Dr. Bench | 214 | 14.32 | General |
| ResearcherBench | 65 | 13.33 | AI&ML |
Table 4: Dataset statistics . # Criteria / Question denotes the average number of question-specific rubric criteria per question.
Figure 5: Negative-weight Reminder Text.
| Question | Source/Rationale | |
| Does the report follow a clear, logically ordered structure that is easy to navigate (e.g., problem approach results), with sections that match the report’s stated purpose and directly address the research question? | Q1 in LiveResearchBench’s Table 3; GRR 1 and 2 in Dr. Bench; Definition of “Clear and Logical Structure” in DeepResearch-ReportEval | |
| Do different sections logically follow or build on one another with minimal redundant restatement, and is any repetition clearly purposeful (e.g., brief recap before a new stage)? | Definition of “Redundancy” in DeepResearch-ReportEval; Q2 in LiveResearchBench’s Table 3; Refined so that recap/summary is not counted as redundancy | |
| Where content is naturally parallel (steps, criteria, comparisons, key takeaways), does the report use lists and/or tables to present it in a scannable form rather than dense prose? | Newly written. This is not present in any previous evaluations, but is essential for penalizing dense paragraphs without proper formatting. | |
| Are headings/subheadings consistent in level and hierarchy (H1/H2/H3), and are comparable sections named with parallel phrasing (e.g., “Method,” “Results” rather than inconsistent mixes like “How they did it,” “Findings”)? | Further specified GRR8, 42 in Dr. Bench. Added that the heading names should be parallel and comparable. | |
| Does the report use concise transition sentences/phrases to signal why the subsequent content follows and to reduce abrupt jumps and make the report easier to follow? | Further specified GRR7 in Dr. Bench and Definition of “Clear and Logical Structure” in DeepResearch-ReportEval. | |
| If there are cross-references, are they consistent and unambiguous (figure/table numbers, section references, in-text citation), with no missing/duplicate numbering and no “see above/below” without an anchor? If no cross-references are present, the score should be 1. | Extended Q3, 4, 6, 10 in LiveResearchBench’s Table 3 and GRR24 in Dr. Bench to all types of cross-references. |
Table 5: Presentation Evaluation Questions
| LLM-Human Agreement Rate | |
| Turn 1 | 95.0% |
| Turn 2 | 96.7% |
| All | 95.8% |
Table 6: LLM judge–human agreement rate across turns.
| ID | Feedback |
| 1 | Please rewrite this so the language is clearer and more straightforward, suitable for a reader with no prior knowledge. |
| 2 | Whenever you introduce a technical concept, add a simple and real-world analogy to illustrate it. |
| 3 | Standardize heading levels and naming so similar sections use parallel phrasing (e.g., ‘Approach’, ‘Results’, ‘Limitations’). |
| 4 | Make sure that each section ends with a short summary sentence that emphasizes the main takeaway. |
| 5 | Add a concise TL;DR at the beginning of the report that states the main question and key takeaways from the report. |
| 6 | It would help if the report indicated which parts are essential reading and which parts are optional background. |
Table 7: 21 Seed Format Feedback Pieces in Mr Dre .
| Content | Format | |
| Naturalness | 100% | 100% |
| Draft-specificness | 92% | 90% |
| Actionability | 98% | 98% |
| Content-preserving | – | 100% |
| Agreement Rate | 98% | 96% |
Table 8: Human verification results of simulated feedback quality.
Figure 6: Full multi-turn results for the Reflect setting. Tongyi DR’s citation faithfulness and claim groundedness are omitted since it is not trained to generate citations. Error bars indicate standard errors.
Figure 7: Full multi-turn results for the Content 1 setting. Tongyi DR’s citation faithfulness and claim groundedness are omitted since it is not trained to generate citations. Error bars indicate standard errors.
Figure 8: Full results for content feedback with multiple feedback targets ( ).
| Feedback type | Feedback example |
| Format | Including a glossary of key terms at the end of the report would greatly benefit beginners by providing a quick reference to important concepts like forward propagation, backpropagation, and optimization methods, helping to reinforce understanding as they read through the material. |
| Format | Consider applying bold formatting to key technical terms, main product features, and critical advantages when they first appear, while using italics for secondary details like specific APIs or model names; this will help readers quickly identify the most important information and improve overall readability. |
| Content 1 | Thanks for covering the MYC pathway—it’s a great start! To make the overview stronger, could you also include how NELF-E affects other important genes or pathways like BRCA1, RAD51, or its role in promoter-proximal pausing in HCC? That would really round out the explanation. |
| Content 1 | Hey, could you add a part explaining how S1 shows that smaller, high-quality datasets can match the performance of much larger ones? Right now, it mostly talks about large-scale data but misses that important insight about data quality over quantity. |
| Content 3 | Hey, the report would be way stronger if it included a clear marketing strategy covering at least four channels like social media influencers, local events, digital ads, and food delivery promos, with a quick note on how each helps build the brand. Also, it’s important to add staffing details for each concept—like specific roles needed for the food truck, fine dining, and fast casual spots—so we get a better sense of the team structure. Lastly, while naming a couple of design firms was helpful, including their contact info and some rough cost estimates would make it easier to move forward and compare options. |
Table 14: Format and Content k feedback examples. We show representative format feedback and content feedback with one or three targets.
Figure 9: Citation Degradation Example. After revision, the model reduces the number of citations and omits in-context citations from the original response.
Figure 10: Example of missing content. The revised response preserves the overall narrative but omits the final paragraph from the original.
Figure 11: Content 1 feedback Simulation System Prompt.
Figure 12: Content k feedback Simulation System Prompt.
Figure 13: Format feedback Simulation System Prompt.
Figure 14: Coverage Evaluation System Prompt.
Figure 15: Coverage Evaluation User Prompt.
Figure 16: Presentation Evaluation System Prompt.
Figure 17: Presentation Evaluation User Prompt.
Figure 18: Claim Extraction System Prompt.
Figure 19: Claim Extraction User Prompt.
Figure 20: Claim Supportedness Judge System Prompt
Figure 21: Claim Supportedness Judge User Prompt.
Figure 22: URL Content Summarization System Prompt
Figure 23: URL Content Summarization User Prompt.
Figure 24: Pairwise Formal Feedback Judge System Prompt.
Figure 25: Pairwise Formal Feedback Judge User Prompt.
Figure 26: Prompt Engineering System Prompt.
Figure 27: Prompt Engineering Hard-coded Constraint Suffix.
Figure 28: Reviser Subagent System Prompt.
Figure 29: Reviser Subagent User Prompt.
More in this topic
Deep Research Agents
Deep research agents plan long-horizon searches, synthesize cited evidence, and produce reports for complex open-ended questions.
See all 119 papers