Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks
Authors: Jena D. Hwang, Varsha Kishore, Amanpreet Singh, Dany Haddad, Aakanksha Naik, Malachi Hamada, Jonathan Bragg, Mike D'Arcy, Daniel S. Weld, Lucy Lu Wang, Doug Downey, Sergey Feldman
Recent advances have made long-form report-generating systems widely available. This has prompted evaluation frameworks that use LLM-as-judge protocols and claim verification, along with meta-evaluation frameworks that seek to validate these methods. Many of the meta-evaluations estimate an evaluation quality's by comparing its assessments against human pairwise preferences. Prior work, however, suggests that human pairwise preference may be overly simplistic and can fail to capture nuances of expert expectations. We conduct a case study in meta-evaluation for long-form QA benchmarks using ScholarQA-CS2, a benchmark designed for assessing retrieval-augmented deep-research QA in the scientific domain. We comprehensively validate the benchmark through human pairwise preference judgments, then critically examine the strengths, weaknesses, and confounders of this approach. We show that pairwise preference rankings are best suited for system-level evaluation, while explicit metric-wise annotations and expert annotators are critical for reliable metric-level assessment, with subjectivity remaining a key challenge. Based on our findings, we offer practical guidelines for designing future meta-evaluations that better align evaluation methods, annotator expertise, and reporting practices. By surfacing these methodological challenges, we aim to advance evaluation standards for deep-research systems.
Figures & tables
Human Meta-Evaluation
(Expert Agreement w/ LLM-judge)
Evaluation
Dom
Input Source
Ret Src
Metrics Used
Expert Eval
PPR
Metric -Wise
Reported Comp.
Qs Assign
# Sys Comp.
IAA
OA
ScholarQA-CS2 ( Bragg et al., 2025 ) + present study
Sci:CS
real user queries
PI
AR,RCC, CP,CR
✓
✓
✓✓
SC,IC, OA,MA, MC
R,C,A
6
55.0%
51.6%
OpenScholar ( Asai et al., 2024 )
Sci
written queries
PI
RCC,CP, CR,AQ
✓
✓
✓ *
OA
R
3
68%
50%
DeepResearchGym ( Coelho et al., 2025 )
Gen
real user queries
WI
FC,AQ, CP,CR
✓
✓
✓ *
SC,OA
R
7
0.87 Cohen κ
0.72 Cohen κ
DeepResearch Bench ( Du et al., 2025 )
Gen
curated queries
WS
RC,CP, CR
✓
✓
-
SC,OA
R
4
68.4%
71.3%
Table 1: We summarize benchmarks that evaluate scientific or research-oriented systems and represent the subset that are most closely related to ScholarQA-CS2 . Ordered alphabetically by author.
Figure 1: Our experimental settings for meta-evaluation of ScholarQA-CS2 . We investigate meta-evaluation that assesses agreement between evaluation score and expert preference ranking (Setting 1). We also investigate settings (Settings 2 & 3) that compare evaluation scores against metric-wise judgment annotation and control for annotator expertise level.
All Data
Expert-
(IAA: 55.0%)
Agreed
System-level Corr ( τ -b)
0.40
0.60
↪ w/o Elicit
0.70
1.00
Instance-level Corr ( τ -b)
0.25
0.36
Table 2: System and sample-level correlations (Kendall τ -b) and Human IAA. Our system-level τ s range from moderate-to-strong. The strong correlations are not reflected at instance-level comparison.
Human Score
Model Score
All Data
Expert-Agreed
Overall preference ranking
Overall Score
51.6%
62.4%
Answer Relev.
34.5%
35.2%
Answer Recall
47.3%
54.3%
Citation Prec.
42.3%
59.0%
Citation Recall
41.9%
58.4%
Table 3: Pairwise agreements between expert overall preference ranking vs. model scores. Our overall score agreement approaches IAA levels, however, individual metric comparison to human preference show lower levels of agreement.
Human-Model Score comparison
Near-Expert
Deep-Expert
Human Score
Model Score
agree.
τ -b
agree.
τ -b
Overall preference ranking
Overall score
65.0%
0.50
57.3%
0.25
Answer relev.
31.7%
0.00
22.7%
-0.02
Answer recall
25.0%
0.00
25.3%
0.09
Citation prec.
53.3%
0.38
49.3%
0.22
Citation recall
43.3%
0.31
49.3%
0.29
Table 4: Results from per metric human evaluation with expertise control. We compare instance-level agreement between model scores and two human signals: (TOP) human preference, and (BOTTOM) metric-wise human ratings collected in a setup parallel to ScholarQA-CS2 metrics. The results include thresholded ties (see § 4.1 ). Overall score (for both human and model) is the computed average of the metric scores at instance-level.
Figure 2: When evaluating answer relevance disagreements, near- and deep-experts choose to keep their own judgments at similar rates; however, when they do not, near-experts tend to defer to the LLM’s judgment more often than deep-experts.
Figure 3: Expert preference ranking vs. model scores. Model score alignments with expert preferences vary substantially across annotators.
Gemini-flash
Gemini-pro
Claude-sonnet
Claude-sonnet-r
GPT-5.1
GPT-5 (test only)
agree.
τ -b
agree.
τ -b
agree.
τ -b
agree.
τ -b
agree.
τ -b
agree.
τ -bs
overall score
51.8%
0.22
52.0%
0.24
57.1%
0.31
53.3%
0.25
55.2%
0.28
51.5%
0.24
answer relev.
29.6%
-0.07
30.4%
-0.12
24.3%
0.00
25.8%
-0.14
40.5%
0.01
39.2%
-0.03
answer recall
45.9%
0.15
48.1%
0.16
47.3%
0.14
48.4%
0.15
48.7%
0.17
44.6%
0.10
citation prec.
40.5%
0.20
41.0%
0.19
42.5%
0.25
40.8%
0.19
38.9%
0.20
43.3%
0.23
citation recall
40.2%
0.15
42.9%
0.21
44.4%
0.25
43.4%
0.18
43.8%
0.26
42.3%
0.22
Table 5: Evaluation results using six alternative LLM judges: Gemini-2.5-flash (Gemini-flash, fresh run), Gemini-2.5-pro (Gemini-pro), Claude-sonnet-4.5 with (Claude-sonnet-r) and without reasoning (Claude-sonnet), GPT-5.1 without reasoning, and reasoning GPT-5. We limit reasoning GPT-5.1 run to test set due to cost constraints. We present the agreement (TOP) and the system correlations (BOTTOM) for both all data and expert-agreed subset.
Figure 4: Expert preference ranking vs. expert’s metric-wise scores obtained in annotations settings 2 & 3. The expert preferences alignments with their own scores vary substantially across annotators.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Human annotation is conducted as a 3-way comparison. The results are converted into pairwise judgments, and is compared to the model scores for score (dis)agreements.
Figure 6: Annotation Procedure and Screenshots
Figure 7: rubric comparison
Figure 8: The annotation UI used to collect validation data from CS researchers regarding the quality of the citation evaluation quality.
IAA
Agree.
2 systems
55.3%
43.4%
3 systems
57.0%
44.6%
all 6 systems
55.0%
51.6%
↪
w/o Elicit
57.0%
58.9%
Appendix
Table 6: Comparing most competitive or "best" systems. 3 systems : ScholarQA-CS2 , ScholarQA-CS2 -SFT, OpenAI-DR; 2 systems : ScholarQA-CS2 , ScholarQA-CS2 -SFT.
Table 8: Final rankings based on human overall preference ratings for the full dataset.
Figure 9: Analysis of qualitative assessments shows that each annotator calibrates what constitutes a good answer differently.
Figure 10: Cross-correlation heatmap for Pearson ρ coefficient. Cross-correlation analysis indicates that all models are highly correlated with one another, and that our findings are robust across evaluator choice.
Human Meta-Evaluation
(Expert Agreement w/ LLM-judge)
Evaluation
Dom
Input Source
Ext. Src
Metrics Used
Expert Eval
PPR
Metric -Wise
Reported Comp.
Qs Assign
# Sys Comp.
IAA
OA
ScholarQA-CS2 ( Bragg et al., 2025 ) + present study
Sci:CS
real user queries
PI
AR,RCC, CP,CR
✓
✓
✓✓
SC,IC, OA,MA, MC
R,C,A
6
55.0%
51.6%
OpenScholar ( Asai et al., 2024 )
Sci
written queries
PI
RCC,CP, CR,AQ
✓
✓
✓ *
OA
R
3
68%
50%
Deep Research Bench ( Bosse et al., 2025 )
Gen
synthetic queries
WS
RC,CP
-
-
-
-
-
DeepResearchGym ( Coelho et al., 2025 )
Gen
real user queries
WI
FC,AQ, CP,CR
✓
✓
✓ *
SC,OA
R
7
0.87 Cohen κ
0.72 Cohen κ
Appendix
Table 9: Related Deep Research Evaluation Literature. Ordered alphabetically by author.