We consider the problem of evaluating AI agents undertaking investment-research tasks, where agents interact with external information sources and computational tools to produce financial analyses and research reports. An important challenge is that the quality of an agent's final output does not necessarily reflect the quality of its execution process, while the reliability of the resulting assessment also depends on the validity of the evaluation procedure. Furthermore, investment-research tasks differ considerably in their requirements for numerical accuracy, information retrieval, financial analysis and supporting evidence, making it difficult to assess agent performance using a uniform set of criteria. We propose a domain-oriented evaluation framework that distinguishes outcome quality from execution quality and accommodates these differences through task-specific criteria, deterministic validators and independent large language model judges. We also introduce an evaluation-validity mechanism that examines the integrity of evaluation inputs, reference information, supporting evidence and grading outputs, with expert review used to resolve identified deficiencies and significant grading disagreements. Numerical results show that the proposed framework can distinguish differences in agent performance across evaluation dimensions, while identifying potential reliability issues in the evaluation process. The associated implementation and evaluation materials will be made available on GitHub.
Figures & tables
Figure 1: Illustrative evaluation of an investment-research task. Two execution traces produce the same final report through different processes. Although the report contains financial comparisons, its investment recommendation lacks sufficient risk analysis and supporting evidence. The example illustrates the distinction between outcome quality and execution quality, and the need to verify the validity of their assessments.
Figure 2: The proposed evaluation framework for investment-research agents. Each evaluation record is first examined for input, reference, and evidence integrity. Following these preliminary checks, outcome and execution quality are assessed using the applicable grading mechanisms. The resulting performance scores are then checked for grading integrity before the evaluation result is accepted. Scoring is deferred if a preliminary integrity requirement is not satisfied.
Planning, tool usage, and the appropriateness of the execution path.
Execution trace and tool interactions.
Table 1: Assessment focus and supporting evidence for outcome and execution quality.
Check
Verification requirement
Failure condition
Input integrity
Correct association of the task, agent execution, assessment criteria, and reference information.
Missing or mismatched evaluation inputs.
Reference integrity
Provenance, correctness, and temporal consistency of reference information.
Invalid sources, reporting periods, or reference information.
Evidence integrity
Availability and consistency of the final output and execution trace required for assessment.
Missing or inconsistent assessment evidence.
Grading integrity
Compliance with scoring rules, completeness of grading outputs, and consistency between scores, reasons, and supporting evidence.
Invalid grading outputs or unresolved discrepancies.
Table 2: Integrity requirements used to verify evaluation validity. The first three checks precede scoring; grading integrity is verified after the grading outputs have been produced.
Agent
Scored tasks
Outcome
Execution
Raw overall
Final overall
Codex
62
82.07
86.05
83.04
82.94
DSH
62
78.41
81.90
79.29
77.75
OpenClaw
62
76.00
81.87
77.46
76.99
Table 3: Overall performance of the three agents across 62 tasks (0–100 scale). Final overall scores incorporate expert-reviewed adjustments to raw scores.
Perspective
Agent
Mean
10th
25th
50th
75th
90th
P1
Codex
7.20
0.00
11.14
9.42
1.91
3.59
DSH
2.36
-4.03
-1.14
9.03
0.34
0.44
P2
Codex
8.19
3.94
11.29
11.07
3.26
3.59
DSH
3.19
-1.73
0.58
10.29
0.90
0.11
Table 4: Percentage improvements relative to OpenClaw under common-scope comparisons P1 and P2 (62 tasks). P2 excludes execution efficiency and renormalises the remaining weights.
Component
Dimension
Tasks
Codex
DSH
OpenClaw
Outcome
Correctness
50
4.02
3.82
3.85
Coverage
62
4.08
4.10
3.73
Evidence quality
61
3.99
3.71
3.61
Research analysis quality
47
4.26
4.16
4.00
Deliverable quality
62
4.58
4.04
4.25
Execution
Functional path
62
4.44
4.20
4.13
Table 5: Mean dimension scores (0–5 scale) on commonly assessable tasks.
Task type
Component
Tasks
Codex
DSH
OpenClaw
T1: Factual retrieval
Outcome
14
82.81
79.64
72.50
Execution
86.33
84.92
82.85
T2: Multi-step financial analysis
Outcome
15
73.25
67.59
68.21
Execution
83.31
79.14
74.51
T3: Time-sensitive research
Outcome
27
86.86
85.51
81.58
Execution
86.73
82.53
84.19
Table 6: Mean outcome-quality and execution-quality scores by task type (0–100 scale) on commonly assessable tasks.
Requirement
Failed runs
Failure rate (%)
Input integrity
0
0.0
Reference integrity
35
18.8
Evidence integrity
28
15.1
Grading integrity
90
48.4
Table 7: Integrity failures across 186 agent–task records.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Principal task requirement
Tasks
Expected records
T1
Factual retrieval and numerical accuracy
14
42
T2
Multi-step financial analysis and comparison
15
45
T3
Time-sensitive information and supporting evidence
27
81
T4
Open-ended research and interpretation
6
18
Total
62
186
Appendix
Table 8: Classification of the 62 evaluated investment-research tasks and corresponding agent–task records.
Agent
Expected
Evidence
Pre-scoring pass
Scored
Availability (%)
Codex
62
62
62
62
100.0
DSH
62
62
62
62
100.0
OpenClaw
62
62
62
62
100.0
Total
186
186
186
186
100.0
Appendix
Table 9: Assessment availability and pre-scoring clearance for the 62-task evaluation.
Measure
Denominator
Count
Rate (%)
Assessments with two eligible judgements
All eligible paired assessments
484
100.0
Not referred
484 paired assessments
248
51.2
Referred for further examination
484 paired assessments
236
48.8
Review required
484 paired assessments
201
41.5
Correction proposed
484 paired assessments
35
7.2
Referrals resolved and accepted
236 assessments
236
100.0
Appendix
Table 10: Independent-judge disagreement referrals and review outcomes.
We introduce Deep FinResearch Bench, a practical and comprehensive evaluation framework for deep research (DR) agents in financial investment research. The benchmark assesses three dimensions of report quality: qualitative rigor, quantitative forecasting and valuation accuracy, and claim credibility and verifiability. Particularly, we define corresponding qualitative and quantitative evaluation metrics and implement an automated scoring procedure to enable scalable assessment. Applying the benchmark to financial reports from frontier DR agents and comparing them with reports authored by financial professionals, we find that AI-generated reports still fall short across these dimensions. These findings underscore the need for domain-specialized DR agents tailored to finance, and we hope the work establishes a foundation for standardized benchmarking of DR agents in financial research.
Mirazul Haque, Antony Papadimitriou, Samuel Mensah +6
Financial agents must form forecasts and investment judgments from decision-time evidence, with predictions evaluated against subsequent outcomes. Existing benchmarks assess financial knowledge, research workflows, or forecasting, but evaluating the full research process requires a strictly frozen information set and professional evaluation criteria. We introduce Investor's Last Exam (ILE), an equity-research benchmark led by experienced buy-side investors and fund managers. Its three components are a versioned financial database spanning 3.43 million document versions and nine source categories, supplemented by structured market data, with validated timestamps governing retrieval and computation; a task taxonomy organized around professional analyst deliverables; and an expert-led framework linking scoring criteria to sources, calculations, and assumptions through financial dependency graphs. Evaluation spans 86 assignments and jointly assesses research quality, forecast accuracy, and simulated investment performance. GPT-6 Astra and Claude Opus 5.5 lead overall at 68.1 and 66.9 out of 100, yet even Astra scores only 52.5 on Forecast & Model. Our results show that even state-of-the-art models fail to translate overall research capability into predictive performance. ILE shifts the evaluation unit from isolated capabilities to the evidence--assumption--judgment chain, allowing researchers to locate where analytical advantages fail to translate into forecasts and investment judgments, and providing a traceable, diagnostic foundation for training and evaluating financial agents.
Huangfei Jiang, Anning Hu, Kai Sheng +7
Prophetlab · RethinkLab · Antai College of Economics and Management, SJTU +3
Deep Research Agents generate analyst-grade reports, yet evaluating them remains challenging due to the absence of a single ground truth and the multidimensional nature of research quality. Recent benchmarks propose distinct methodologies, yet they suffer from the Mirage of Synthesis, where strong surface-level fluency and citation alignment can obscure underlying factual and reasoning defects. We characterize this gap by introducing a taxonomy across four verticals that exposes a critical capability mismatch: static evaluators inherently lack the tool-use capabilities required to assess temporal validity and factual correctness. To address this, we propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that instantiates the principle of capability parity by making evaluation itself agentic. DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent, enabling temporally aware coverage, grounded verification, and systematic reasoning probes. Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks, offering a scalable, reference-free evaluation paradigm.