FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

Jie Zhu; Yimin Tian; Boyang Li; Kehao Wu; Zhongzhi Liang; Junhui Li; Xianyin Zhang; Lifan Guo

cs.AIMar 26, 2026

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

Authors: Jie Zhu, Yimin Tian, Boyang Li, Kehao Wu, Zhongzhi Liang, Junhui Li, Xianyin Zhang, Lifan Guo,

Organizations: Qwen DianJin Team, Alibaba Cloud Computing · YINGMI Wealth Management · Soochow University

Abstract

This paper introduces FinMCP-Bench, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.

Figures & tables

Figure 1: The 10 main scenarios and 33 sub-scenarios in FinMCP-Bench.

Figure 2: Overview of the chain-based method for synthesizing multi-tool samples.

Figure 3: Overview of the role-playing-based method for synthesizing multi-turn samples.

FinMCP-Bench dataset
Type	Number
All	613
- Easy/Medium/Hard	343 / 245 / 25
Single-Tool	145
Multi-Tool	249
- MCP calls per instance (Median / Avg)	8 / 7.32

Table 1: Statistics of FinMCP-Bench.

Model	Single-Tool				Multi-Tool				Multi-Turn				All
	TP	TR	TF1	EMR	TP	TR	TF1	EMR	TP	TR	TF1	EMR	TP	TR	TF1	EMR
DeepSeek-R1	59.87	65.28	62.46	49.31	65.95	43.42	52.36	4.85	46.15	5.15	9.27	0.00	64.79	40.55	49.88	18.08
GPT-OSS-20B	13.50	68.97	22.57	15.17	28.26	60.59	38.54	1.20	9.51	6.03	7.38	0.00	25.97	43.84	32.62	4.43
Seed-OSS-36B	35.59	72.41	47.73	35.17	58.21	28.13	37.93	3.61	39.26	45.49	42.15	2.17	50.36	32.28	39.34	13.86
Qwen3-4B-Thinking	63.01	75.17	68.55	65.52	64.44	41.16	50.23	4.82	49.23	46.16	47.65	2.26	58.02	44.06	50.08	18.82
Qwen3-30B-A3B-Thinking	40.96	70.34	51.78	55.17	67.92	54.92	60.73	7.47	53.17	41.17	46.40	4.10	61.75	50.53	55.58	18.24

Table 2: Results (%) on FinMCP-Bench.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
Mar 9, 2026Jiaxuan Lu, Kong Wang, Yemin Wang +9LLM Agent Evaluation Tool-Using Language Agents
Shanghai AI Laboratory · Hunan University · Xiamen University +3
BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation
Jan 10, 2026Xin Guo, Rongjunchen Zhang, Guilong Lu +4Financial Services Multilingual Language Model Evaluation
HiThink Research · Shanghai University of Finance and Economics
FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks
Apr 11, 2026Yupeng Cao, Haohang Li, Weijin Liu +11Agent Trajectory Analysis LLM Tool Use
Stevens Institute of Technology · Independent Researcher · The FinAI +2

Mar 9, 2026J/K move · Enter open · S save

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

Jiaxuan Lu, Kong Wang, Yemin Wang, Qingmei Tang+8

Shanghai AI Laboratory · Hunan University · Xiamen University+3

LLM Agent Evaluation Tool-Using Language Agents Tool Use AI Agent Benchmarks Finance

Open paper AskPDF arXiv

The integration of Large Language Models (LLMs) into the financial domain is driving a paradigm shift from passive information retrieval to dynamic, agentic interaction. While general-purpose tool learning has witnessed a surge in benchmarks, the financial sector, characterized by high stakes, strict compliance, and rapid data volatility, remains critically underserved. Existing financial evaluations predominantly focus on static textual analysis or document-based QA, ignoring the complex reality of tool execution. Conversely, general tool benchmarks lack the domain-specific rigor required for finance, often relying on toy environments or a negligible number of financial APIs. To bridge this gap, we introduce FinToolBench, the first real-world, runnable benchmark dedicated to evaluating financial tool learning agents. Unlike prior works limited to a handful of mock tools, FinToolBench establishes a realistic ecosystem coupling 760 executable financial tools with 295 rigorous, tool-required queries. We propose a novel evaluation framework that goes beyond binary execution success, assessing agents on finance-critical dimensions: timeliness, intent type, and regulatory domain alignment. Furthermore, we present FATR, a finance-aware tool retrieval and reasoning baseline that enhances stability and compliance. By providing the first testbed for auditable, agentic financial execution, FinToolBench sets a new standard for trustworthy AI in finance. The tool manifest, execution environment, and evaluation code will be open-sourced to facilitate future research.

Abstract

LLM Agent Evaluation

Explore similar work

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation

FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks