FinHopper: A Progressive Multi-Hop Benchmark for Complex Financial Agents with Step-wise Evaluation

Yuxuan Wu; Miaopei Lin; Qiufang Ying; Lin Dan; Hao Zhao; Qirui Liu; Yuteng Shen; Shaohui Wu

cs.LGOct 19, 2026

FinHopper: A Progressive Multi-Hop Benchmark for Complex Financial Agents with Step-wise Evaluation

Authors: Yuxuan Wu, Miaopei Lin, Qiufang Ying, Lin Dan, Hao Zhao, Qirui Liu, Yuteng Shen, Shaohui Wu,

Organizations: South China University of Technology · FiT, Tencent · Shenzhen Technology University

Abstract

Financial agents are expected to answer factual investment queries through open search, but financial multi-hop retrieval differs from general multi-hop QA in a crucial way: each intermediate answer is an exact typed constraint, such as an entity, date, event, or numerical indicator, that determines the next query. This strict answer-conditioned dependency makes small intermediate errors cascade into plausible but wrong search trajectories. We introduce FinHopper, a benchmark for complex financial agents with controlled reasoning depth from 1 to 6 hops, covering stocks, funds, and macroeconomic facts, with gold intermediate answers for process-level evaluation. To construct FinHopper at scale, we design a state-transition-based synthesis framework that composes verified atomic financial QA patterns into dependency-preserving chains, followed by targeted in-synthesis verification, model-ensemble checking, and public-search validation by financial experts. Experiments with 20 LLM-agent configurations show that public search tools are necessary but insufficient: performance drops sharply as hop depth increases, the best model remains far below human experts, and process-aware scoring reveals partial reasoning progress hidden by final-answer-only evaluation. The benchmark is available at https://github.com/justajustin/FinHopper/.

Figures & tables

Figure 1: FinHopper abstracts financial multi-hop search as state-conditioned answer transitions, where each verified answer constrains the next hop through dependency-preserving answer substitution.

Figure 2: Fine-grained construction pipeline of FinHopper. The framework first samples reusable financial patterns according to valid answer-state transitions, then generates and validates single-hop QA pairs, and finally merges verified hops into multi-hop questions while preserving intermediate answers for process-level evaluation.

Panel of Figure 3 — Figure 3: FinHopper provides a controlled difficulty gradient: hop depth increases from L1 to L3, question components grow denser with difficulty, and items cover diverse financial domains across stocks, funds, and macroeconomics.

Hop	1	2	3	4	5	6
Level	L1	L1	L2	L2	L3	L3
#Items	50	50	39	21	19	21
#Step answers	50	100	117	84	95	126

Table 1: Hop-wise composition of FinHopper. Each

K

-hop item contains

K

step-level gold answers, yielding 572 step-level checking points in total. These step-level annotations are diagnostic signals rather than independent test samples.

	without tools		use tools
model	Mic Avg.	Time	1hop	2hop	3hop	4hop	5hop	6hop	Mic Avg.	Mac Avg.	Time
Human Performance	0	5	94.0	90.0	84.6	73.4	65.9	64.4	81.7	78.7	324
Claude-4.5-Sonnet-thinking	2.5	18	81.3	72.5	70.2	63.5	54.1	31.1	67.3	62.1	31
Claude-4.5-Sonnet-no-thinking	1.0	10	79.6	70.7	63.2	60.0	49.6	25.6	63.7	58.1	24
Gemini-3-Pro-thinking	3.2	18	80.0	69.1	69.5	60.1	52.2	26.5	65.0	59.6	34
Gemini-3-Pro-no-thinking	2.8	14	81.2	70.3	65.5	53.0	44.2	20.2	62.7	55.7	25

Table 2: Performance of various models and human on FinHopper. All scores are reported in %.

	without tools		use tools
model	Mic Avg.	Time	1hop	2hop	3hop	4hop	5hop	6hop	Mic Avg.	Mac Avg.	Time
Gemini-3-Pro-thinking	3.5	17	81.5	68.3	70.8	60.8	51.5	26.0	65.2	59.8	33
GPT-5.2-thinking	4.8	29	84.8	69.0	57.2	54.0	45.2	19.8	61.7	55.0	118
GLM-4.7-thinking	7.5	74	75.0	48.2	51.0	42.8	22.2	14.8	48.8	42.3	95
Kimi-k2-thinking	3.0	25	66.0	59.2	60.5	52.5	48.8	38.2	57.3	54.2	208

Table 3: Performance of representative models on FinHopper (English). All scores are reported in %.

Figure 4: Model performance across 1–6 hop tasks, grouped by thinking mode. The curves expose model-specific capability boundaries: most agents remain competitive on shallow tasks but degrade sharply once the reasoning chain exceeds their effective search and state-tracking depth.

Figure 5: Breakdown of model performance into Final Answer Score(

s_{final}

) and Process Score (

s_{process}

). The left and right panels show results for 5-hop and 6-hop questions, respectively.

Figure 6: Score gain of Thinking Mode across tasks with depth from 1 to 6 hops. The grey lines represent the score trajectories of individual models, while the bold red line denotes the average gain of all models.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: In-synthesis verification independently re-derives single-hop answers and critiques chain-merged questions before they are accepted into longer FinHopper instances.

Model	Mic Avg.	Mac Avg.
Human Performance	$81.7\pm 1.6$	$78.7\pm 1.4$
Claude-4.5-Sonnet-thinking	$67.3\pm 0.8$	$62.1\pm 1.1$
Claude-4.5-Sonnet-no-thinking	$63.7\pm 1.0$	$58.1\pm 0.9$
Gemini-3-Pro-thinking	$65.0\pm 1.2$	$59.6\pm 0.9$
Gemini-3-Pro-no-thinking	$62.7\pm 1.1$	$55.7\pm 1.3$
GPT-5.2-thinking	$61.3\pm 0.7$	$54.6\pm 1.3$

Appendix

Table 4: Stability analysis on FinHopper. Scores are reported as mean

\pm

standard deviation in %.

Input State	Data Pool	Examples
Entity	Stocks and funds	Tencent, CATL, fund/ETF tickers
Time	Trading days, a period of time	2025-11-19; FY2025_Q3, 2024-10-05–2024-11-05
Sector	Industry classifications, sector indices	New Energy Vehicles; Semiconductor; CSI 300 sector index
Event	Earnings releases, dividends, corporate announcements, …	Q3 earnings release; dividend distribution; share buyback announcement

Appendix

Table 5: Data pool selection based on input state.

From State	To State	Hop $k$ Question	Hop $k$ +1 Question
Entity	Entity	Question: Which fund holds the highest value of Moutai? (Answer Type: Entity )	Who is the fund manager of that fund ? (Answer Type: Entity )
	Sector	Among 4 companies, which had the highest net profit margin in Q3 2025? (Answer Type: Entity )	What sector does that company belong to? (Answer Type: Sector)
	Time	Among 4 companies, which had the highest net profit margin in Q3 2025? (Answer Type: Entity )	When was that company listed? (Answer Type: Time)
	Indicator	What was CATL’s closing price on Dec 30, 2025? (Answer Type: Indicator )	369.20 CNY (Terminal, Answer Type: Indicator)
Time	Entity	When did Jingjin Equipment Inc release its 2nd report in 2024? (Answer Type: Time )	On that day , which fund had weekly return $>$ -0.5%? (Answer Type: Entity)
	Indicator	Which day did Moutai’s price peak in Nov 2025? (Answer Type: Time )	What was CATL’s closing price on that day ? (Answer Type: Indicator)

Appendix

Table 6: State transition rules with examples. Each row shows a valid transition illustrated with consecutive hop questions. The “Indicator” state terminates the reasoning chain.

#	Question	Condition	Answer	Example
Entity-Time Comparison Patterns
1	ME + ST	Rank by indicator, select top- $n$	SE	Which of Tencent, Xiaomi, Alibaba, CATL had highest volume on 2025-11-19?
2	ME + ST	Multi-indicator filtering (unique match)	SE	Which of Tencent, Xiaomi, Alibaba, CATL had volume $>$ 100M and turnover $<$ 1% on 2025-11-19?
3	ME + ST	Compare quantitative indicator values	IND	What are net capital inflows for Tencent, Xiaomi, Alibaba on 2025-11-19? Which is largest?
4	SE + MT	Rank by indicator, select top- $n$	ST	On which day did Tencent have highest volume among 2025-11-17/18/19?
5	SE + MT	Compare quantitative indicator values	IND	What was Tencent’s volume on 2025-11-17/18/19? Which day was highest?

Appendix

Table 7: Single-Hop QA Generation Patterns: Question Input

\rightarrow

Answer Type.

Question
Process Concealment	Description : Intermediate results and final answers must not be revealed via direct mentions or contextual cues.
	Non-compliant Q : "What was Apple’s stock price on Dec 1, 2025, given it reported higher net income than NVIDIA in FY2026_Q3?"
	Compliant Q : "What was the stock price on Dec 1, 2025, of the company (Apple or NVIDIA) with higher net income in FY2026_Q3?"
Intent Preservation	Description : The semantic intent and information of both source questions must be preserved in the merged question.
	Source Q : · "Which company had higher net income in FY26_Q3: Apple or Nvidia?" · "What was Apple’s stock price on Dec 1, 2025?"
	Non-compliant Q : "Between Apple and Nvidia, what was the Dec 1, 2025 stock price of the company with the better performance in FY26_Q3?"

Appendix

Table 8: Guide For Chain-Merge Verification.

Financial QA

Financial question answering develops systems that answer questions about reports, markets, regulations, and other financial data with reliable evidence and reasoning.

See all 51 papers

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability
Sep 21, 2026Wenqing Wang, Haitao Xiang, Xinyi Zhao +8Financial QA AI Agent Benchmarks
Ling Team, Inclusion AI · Peking University · China International Capital Corporation Limited
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
Mar 26, 2026Jie Zhu, Yimin Tian, Boyang Li +8LLM Agent Evaluation LLM Tool Use
Qwen DianJin Team, Alibaba Cloud Computing · YINGMI Wealth Management · Soochow University
BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents
Jun 2, 2026Alex Wang, Georg Meinhardt, Jacob Katz +4Benchmark Design Financial QA
1Rogo · 2OpenAI · †Work done while at Rogo.

Sep 21, 2026J/K move · Enter open · S save

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability

Wenqing Wang, Haitao Xiang, Xinyi Zhao, Mingming Yin+7

Ling Team, Inclusion AI · Peking University · China International Capital Corporation Limited

Financial QA AI Agent Benchmarks Information Retrieval IR Evaluation Evidence Retrieval Financial Reasoning Benchmarks

Open paper AskPDF arXiv

Financial search is a highly demanding task for LLM agents, requiring not only a correct final answer but also temporally valid information retrieval, authoritative source selection, entity and period alignment, unit and definition consistency, and verifiable evidence for all conclusions. Existing benchmarks predominantly evaluate only the final answer, making it difficult to localize errors or assess whether an answer is well-founded. To address this gap, we introduce FinFIRST (Financial Information Retrieval, Sourcing and Traceability), the first financial benchmark to jointly evaluate answers and supporting evidence through atomic rubrics. FinFIRST comprises 123 expert-authored tasks spanning a graduated difficulty spectrum, constructed from aggregate patterns of real-world financial scenarios through an 18-field taxonomy, a six-axis coverage blueprint, a registry of 138 financial sources, contributions from over 50 finance experts, and a six-stage quality-control pipeline. Each task is accompanied by an evidence-grounded reference package decomposed into atomic criteria across three dimensions: raw-information acquisition, source verification, and computation and answer formation. We evaluate 15 model configurations under a unified tool setting. Claude-Opus-5 achieves the highest atomic score of 87.59%, while GPT-5.6-Sol attains the highest strict pass rate of 71.54%. Computation and answer formation consistently lag behind raw-information acquisition across systems. FinFIRST retains final-answer correctness as the primary objective while making the supporting research process measurable, verifiable, and diagnosable.

Abstract

Financial QA

Explore similar work

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents