cs.AINov 17, 2026

Reliable Evaluation of AI Agents for Investment Research: A Framework for Performance and Validity Assessment

Authors: Dongnuan Tian, Yan Yu, Yangfan Liu, Wenjun Tang, Junting Kan, Xiang Wang, Sicong Liu, Yangyang Zhang, 

Organizations: Gildata GAOS Team, Shanghai Hundsun JuYuan Data Service Co., Ltd.

Abstract

We consider the problem of evaluating AI agents undertaking investment-research tasks, where agents interact with external information sources and computational tools to produce financial analyses and research reports. An important challenge is that the quality of an agent's final output does not necessarily reflect the quality of its execution process, while the reliability of the resulting assessment also depends on the validity of the evaluation procedure. Furthermore, investment-research tasks differ considerably in their requirements for numerical accuracy, information retrieval, financial analysis and supporting evidence, making it difficult to assess agent performance using a uniform set of criteria. We propose a domain-oriented evaluation framework that distinguishes outcome quality from execution quality and accommodates these differences through task-specific criteria, deterministic validators and independent large language model judges. We also introduce an evaluation-validity mechanism that examines the integrity of evaluation inputs, reference information, supporting evidence and grading outputs, with expert review used to resolve identified deficiencies and significant grading disagreements. Numerical results show that the proposed framework can distinguish differences in agent performance across evaluation dimensions, while identifying potential reliability issues in the evaluation process. The associated implementation and evaluation materials will be made available on GitHub.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future?

    Oct 28, 2026Huangfei Jiang, Anning Hu, Kai Sheng +7Financial Reasoning BenchmarksAI Agent Evaluation

  2. DREAM: Deep Research Evaluation with Agentic Metrics

    Feb 21, 2026Elad Ben Avraham, Changhao Li, Ron Dorfman +8AI Agent EvaluationDeep Research Agents