We consider the problem of evaluating AI agents undertaking investment-research tasks, where agents interact with external information sources and computational tools to produce financial analyses and research reports. An important challenge is that the quality of an agent's final output does not necessarily reflect the quality of its execution process, while the reliability of the resulting assessment also depends on the validity of the evaluation procedure. Furthermore, investment-research tasks differ considerably in their requirements for numerical accuracy, information retrieval, financial analysis and supporting evidence, making it difficult to assess agent performance using a uniform set of criteria. We propose a domain-oriented evaluation framework that distinguishes outcome quality from execution quality and accommodates these differences through task-specific criteria, deterministic validators and independent large language model judges. We also introduce an evaluation-validity mechanism that examines the integrity of evaluation inputs, reference information, supporting evidence and grading outputs, with expert review used to resolve identified deficiencies and significant grading disagreements. Numerical results show that the proposed framework can distinguish differences in agent performance across evaluation dimensions, while identifying potential reliability issues in the evaluation process. The associated implementation and evaluation materials will be made available on GitHub.
Figures & tables
Figure 1: Illustrative evaluation of an investment-research task. Two execution traces produce the same final report through different processes. Although the report contains financial comparisons, its investment recommendation lacks sufficient risk analysis and supporting evidence. The example illustrates the distinction between outcome quality and execution quality, and the need to verify the validity of their assessments.
Figure 2: The proposed evaluation framework for investment-research agents. Each evaluation record is first examined for input, reference, and evidence integrity. Following these preliminary checks, outcome and execution quality are assessed using the applicable grading mechanisms. The resulting performance scores are then checked for grading integrity before the evaluation result is accepted. Scoring is deferred if a preliminary integrity requirement is not satisfied.
Planning, tool usage, and the appropriateness of the execution path.
Execution trace and tool interactions.
Table 1: Assessment focus and supporting evidence for outcome and execution quality.
Check
Verification requirement
Failure condition
Input integrity
Correct association of the task, agent execution, assessment criteria, and reference information.
Missing or mismatched evaluation inputs.
Reference integrity
Provenance, correctness, and temporal consistency of reference information.
Invalid sources, reporting periods, or reference information.
Evidence integrity
Availability and consistency of the final output and execution trace required for assessment.
Missing or inconsistent assessment evidence.
Grading integrity
Compliance with scoring rules, completeness of grading outputs, and consistency between scores, reasons, and supporting evidence.
Invalid grading outputs or unresolved discrepancies.
Table 2: Integrity requirements used to verify evaluation validity. The first three checks precede scoring; grading integrity is verified after the grading outputs have been produced.
Agent
Scored tasks
Outcome
Execution
Raw overall
Final overall
Codex
62
82.07
86.05
83.04
82.94
DSH
62
78.41
81.90
79.29
77.75
OpenClaw
62
76.00
81.87
77.46
76.99
Table 3: Overall performance of the three agents across 62 tasks (0–100 scale). Final overall scores incorporate expert-reviewed adjustments to raw scores.
Perspective
Agent
Mean
10th
25th
50th
75th
90th
P1
Codex
7.20
0.00
11.14
9.42
1.91
3.59
DSH
2.36
-4.03
-1.14
9.03
0.34
0.44
P2
Codex
8.19
3.94
11.29
11.07
3.26
3.59
DSH
3.19
-1.73
0.58
10.29
0.90
0.11
Table 4: Percentage improvements relative to OpenClaw under common-scope comparisons P1 and P2 (62 tasks). P2 excludes execution efficiency and renormalises the remaining weights.
Component
Dimension
Tasks
Codex
DSH
OpenClaw
Outcome
Correctness
50
4.02
3.82
3.85
Coverage
62
4.08
4.10
3.73
Evidence quality
61
3.99
3.71
3.61
Research analysis quality
47
4.26
4.16
4.00
Deliverable quality
62
4.58
4.04
4.25
Execution
Functional path
62
4.44
4.20
4.13
Table 5: Mean dimension scores (0–5 scale) on commonly assessable tasks.
Task type
Component
Tasks
Codex
DSH
OpenClaw
T1: Factual retrieval
Outcome
14
82.81
79.64
72.50
Execution
86.33
84.92
82.85
T2: Multi-step financial analysis
Outcome
15
73.25
67.59
68.21
Execution
83.31
79.14
74.51
T3: Time-sensitive research
Outcome
27
86.86
85.51
81.58
Execution
86.73
82.53
84.19
Table 6: Mean outcome-quality and execution-quality scores by task type (0–100 scale) on commonly assessable tasks.
Requirement
Failed runs
Failure rate (%)
Input integrity
0
0.0
Reference integrity
35
18.8
Evidence integrity
28
15.1
Grading integrity
90
48.4
Table 7: Integrity failures across 186 agent–task records.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Principal task requirement
Tasks
Expected records
T1
Factual retrieval and numerical accuracy
14
42
T2
Multi-step financial analysis and comparison
15
45
T3
Time-sensitive information and supporting evidence
27
81
T4
Open-ended research and interpretation
6
18
Total
62
186
Appendix
Table 8: Classification of the 62 evaluated investment-research tasks and corresponding agent–task records.
Agent
Expected
Evidence
Pre-scoring pass
Scored
Availability (%)
Codex
62
62
62
62
100.0
DSH
62
62
62
62
100.0
OpenClaw
62
62
62
62
100.0
Total
186
186
186
186
100.0
Appendix
Table 9: Assessment availability and pre-scoring clearance for the 62-task evaluation.
Measure
Denominator
Count
Rate (%)
Assessments with two eligible judgements
All eligible paired assessments
484
100.0
Not referred
484 paired assessments
248
51.2
Referred for further examination
484 paired assessments
236
48.8
Review required
484 paired assessments
201
41.5
Correction proposed
484 paired assessments
35
7.2
Referrals resolved and accepted
236 assessments
236
100.0
Appendix
Table 10: Independent-judge disagreement referrals and review outcomes.