DeepResearch RewardBench: Evaluating Reward Models for Deep Research Report
Organizations: NLP2CT Lab, University of Macau · Monash University · Nanjing University
Abstract
Empowered by Large Language Models (LLMs), deep research systems integrate online information into research reports through multi-step web searching and information synthesis. Evaluating these reports presents a significant challenge due to their long context, high information density, and domain specificity, necessitating the use of AI evaluation as a substitute for prohibitively expensive human assessment. However, few studies have examined the effectiveness of existing reward models or LLMs in understanding and evaluating such research reports. To bridge this gap, we introduce DeepResearch RewardBench, the first benchmark dedicated to evaluating reward modeling for research reports. We collect diverse report candidates from multiple agents or tool-augmented LLMs. These candidates are scored using query-specific golden rubrics, and we ensure data quality and difficulty through a combination of multi-round filtering and human verification. Our benchmark comprises preference datasets formatted as pairs, triplets, and quadruplets to test ranking capabilities across varying levels of difficulty. Extensive experiments demonstrate that while current reward models handle general coherence effectively, their performance significantly degrades when evaluating research reports. Generative LLMs acting as judges demonstrate comprehensive superiority in this domain, with comparative evaluation proving more effective than direct scoring methods. We anticipate that this benchmark will serve as a vital testbed for the evaluation and advancement of research agents.
Figures & tables
| Subset | # Instances (Tasks) | # Reports |
| Pairs | 160 | 320 |
| Triplets | 138 | 414 |
| Quadruplets | 33 | 132 |
| Total | 331 | 866 |
| Models | Pair | Triplet | Quadruplet | Average | Types | ||
| BoN | Rank | BoN | Rank | ||||
| google/gemini-2.5-flash | 81.87 | 91.30 | 70.28 | 81.25 | 31.25 | 78.87 | Generative |
| Qwen/Qwen3-235B-A22B-Instruct-2507 | 81.25 | 84.78 | 66.66 | 78.79 | 60.61 | 77.79 | Generative |
| Qwen/Qwen3-235B-A22B-Thinking-2507 | 80.62 | 83.33 | 70.29 | 87.88 | 48.48 | 77.79 | Generative |
| moonshotai/Kimi-K2-Instruct-0905 | 87.50 | 79.71 | 61.59 | 66.66 | 39.39 | 77.04 | Generative |
| openai/gpt-4.1 | 91.87 | 60.14 | 47.10 | 57.57 | 33.33 | 71.29 | Generative |
| Abbr. | Ratio | Full Name | General Explanation |
| RMI | 63.5% | Rubric Misalignment Insensitivity | Judge evaluates based on features orthogonal to the core rubric. |
| EKE | 7.70% | External Knowledge Error | Judge has a personal knowledge gap or factual misunderstanding, mistakenly penalizing true facts as fabricated |
| CEB | 7.70% | Citation / Sourcing Error Blindness | Judge overlooks or explicitly tolerates missing or fictional references, often misled by numerical precision |
| SLB | 7.70% | Structural / Length Bias | Judge equates surface organization, length, and visual formatting with deep content quality |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | Description |
| Comprehensiveness (COMP) | Report covers key areas of the topic, ensures overall understanding, and does not omit important parts. |
| Insight/Depth (DEPTH) | Report deeply analyzes causes, impacts, and trends, providing valuable insights. |
| Instruction-Following (INST) | Report closely follows the research topic and directly answers questions. |
| Readability (READ) | Report has a clear structure, fluent language, and is easy to understand. |
| # | Rubric item | Report A: gpt-4o-search-preview | Report B: Perplexity | |||
| o3-mini justification (abbrev.) | o3-mini justification (abbrev.) | |||||
| 1 | 2 | Direct application of RL to small ( 7B) models produces substantial gains, with experimental evidence. | Mainly theoretical discussion on RL-vs.-distillation challenges; no experimental evidence or measured improvements on 7B-scale models. | ✓ | Includes detailed experimental evidence of substantial performance improvements from applying RL directly to smaller models. | |
| 2 | 3 | Why distillation from larger models is more effective than RL for enhancing small models’ capabilities. | ✓ | Explains RL is less practical for small models (compute constraints, sample inefficiency, overfitting); contrasts with distillation as more efficient transfer. | Provides balanced RL-vs.-distillation comparison but does not explicitly explain why distillation is superior. | |
| 3 | 2 | A multi-stage training process (base prep RL in verifiable domains iterative SFT+RL) for small models. | Discusses RL and distillation generally; no mention of base RL-in-math/code iterative SFT+RL pipeline. | Covers various aspects of applying RL to small models but does not describe the specified multi-stage pipeline. | ||
| 4 | 2 | Self-improvement RL enables small models to extend reasoning beyond initial token-length limits. | Discusses RL / distillation in general but does not address self-improvement RL or reasoning-chain extension. | Discusses RL for small models vs. distillation; does not address self-improvement RL or token-length extension. | ||
| 5 | 2 | Parameter-capacity limits of small models hinder RL for complex reasoning. | ✓ | Explicitly explains that limited capacity (overfitting risks, sample inefficiency) hinders RL for complex reasoning. | ✓ | Mentions “squashing” of rare behaviors in small models and contrasts with RL benefits, implicitly capturing capacity limits. |
| Category & case | Manual | Automated | Query (gist) | Report A (chosen) vs B (rejected) | Wrong-judge’s key claim | Why this category |
| RMI id=120 (RB) | 63.5% | 61.5% | RL techniques for small LLMs vs distillation | A : gpt-4o-search-preview 55.6; 21 refs cycling only DistilBERT 1 blog. B : Perplexity 33.3; 32 unique refs incl. Mar. 2025 arXiv. | All 3 judges: “B has 2025 arXiv / GitHub citations; A repeats DistilBERT excessively. B’s quantifiable results (AMC23 63 80%, $42 cost) support its claims.” | Judge’s axis: B’s RL freshness + concrete RL benchmarks (2025 arXiv, AMC23 63 80%, $42 cost) — only the benchmark maps to a rubric point ( direct RL works on 7B , w=2/18). Rubric’s axis: “ why distillation beats RL ” (3 sub-points: superiority-rationale, pre-optimized transfer, complexity-circumvention; w=6/18 ) — all covered by A, all missed by B. Judge and rubric are scoring different things. |
| EKE id=40 (DRB) | 7.7% | 12.1% | Top-10 Chinese box-office films future high-grossing types | A : doubao-deepresearch 46.4. B : kimi-researcher 38.9. Both reports cite “154.46 yi yuan” verbatim. | Qwen3: “154.46 billion RMB \times$ Avatar’s global box office. Clearly fabricated.” Gemini: “Astronomically high… far exceed any real-world record globally.” | Judge doesn’t know about Nezha 2 (Jan. 2025 real record, 10^{8}10^{9}21.5B. A real record becomes “fabricated” from a judge knowledge gap — Report B cites the same number but is not flagged . |
| CEB id=139 (RB) | 7.7% | 8.8% | Architectural innovations for lifelong learning in neural networks | A : Perplexity 57.1; 40 real references. B : Claude 50.0; 0 references. | Gemini: “B highlights breakthroughs with specificity, e.g., ‘CH-HNN in Nature Communications 2025’ with ‘60.82% power reduction’ … even if the specific citations are not real .” | Report B has 0 citations yet produces specific-looking artifacts (“60.82% power reduction”, “Nature Communications 2025”). Judge treats numerical precision as evidence, explicitly flagging the citations as possibly fictional yet still picking B. |
| SLB id=19 (DRB) | 7.7% | 5.5% | Anthropic’s Streamable HTTP engineering implementation | A : grok-deeper-search 51.4; 6.2 KB; cites the canonical MCP spec. B : kimi-researcher 43.8; 27.0 KB (4 longer); benchmark from a single CNode blog. | DeepSeek: “B provides architecture diagrams, protocol layer breakdowns, performance benchmarks (175% throughput, 74% memory reduction), concrete code snippets, clearer section headings.” | Every deciding factor is structural — diagrams, code blocks, headings, B is 4 longer. The lone numeric benchmark traces to a non-authoritative blog; A cites the canonical MCP spec . Judge equates surface organization with depth. |
| Other | 13.5% | 12.1% | (empty CoT, 7/7 from Gemini-2.5-Flash) . No representative case shown. | |||
Reward Model Evaluation
Reward model evaluation measures how reliably models rank, score, and guide the selection or improvement of AI-generated responses and actions.
See all 54 papers