FinToolSyn: A forward synthesis Framework for Financial Tool-Use Dialogue Data with Dynamic Tool Retrieval

Caishuang Huang; Yang Qiao; Rongyu Zhang; Junjie Ye; Pu Lu; Wenxi Wu; Meng Zhou; Xiku Du

cs.CLMar 25, 2026

FinToolSyn: A forward synthesis Framework for Financial Tool-Use Dialogue Data with Dynamic Tool Retrieval

Authors: Caishuang Huang, Yang Qiao, Rongyu Zhang, Junjie Ye, Pu Lu, Wenxi Wu, Meng Zhou, Xiku Du,

Organizations: College of Computer Science and Artificial Intelligence, Fudan University · FiT, Tencent · Zhejiang University

Abstract

Tool-use capabilities are vital for Large Language Models (LLMs) in finance, a domain characterized by massive investment targets and data-intensive inquiries. However, existing data synthesis methods typically rely on a reverse synthesis paradigm, generating user queries from pre-sampled tools. This approach inevitably introduces artificial explicitness, yielding queries that fail to capture the implicit, event-driven nature of real-world needs. Moreover, its reliance on static tool sets overlooks the dynamic retrieval process required to navigate massive tool spaces. To address these challenges, we introduce FinToolSyn, a forward synthesis framework designed to generate high-quality financial dialogues. Progressing from persona instruction and atomic tool synthesis to dynamic retrieval dialogue generation, our pipeline constructs a repository of 43,066 tools and synthesizes over 148k dialogue instances, incorporating dynamic retrieval to emulate the noisy candidate sets typical of massive tool spaces. We also establish a dedicated benchmark to evaluate tool-calling capabilities in realistic financial scenarios. Extensive experiments demonstrate that models trained on FinToolSyn achieve a 21.06% improvement, providing a robust foundation for tool learning in financial scenarios.

Figures & tables

Figure 1: Comparison between reverse synthesis and authentic interaction dynamics.

Figure 2: The architecture of FinToolSyn. Our framework employs a forward synthesis paradigm that progresses from persona-based intent generation to dynamic tool retrieval, effectively capturing the implicit and event-driven nature of real-world financial inquiries.

Dataset	# Tools	# Dialogues	# Turns	Avg. Turns
xLAM Zhang et al. (2024a)	3,673	60,000	60,000	1.00
ToolAlpaca Tang et al. (2023)	426	3,938	6,537	1.66
ToolLLM Qin et al. (2023)	16,464	126,486	126,486	1.00
When2Call Ross et al. (2025)	511	27,952	27,952	1.00
BUTTONInstruct Chen et al. (2025)	13,553	8,000	8,000	1.00
APIGen-MT-5k Prabhakar et al. (2025)	248	5,000	24,229	4.85

Table 1: Comparison of FinToolSyn with existing data synthesis resources.

Class	Tool Setting	Response Type
Tool-Call	Single-tool	Single, Parallel
Tool-Call	Multi-tool	Single, Parallel, Serial
Non-Tool-Call	Multi-tool	UD, CI, DR

Table 2: Hierarchical classification of FinToolBench. UD: Unavailability Detection; CI: Clarification Inquiry; DR: Direct Response.

Panel A: Prompt Mode
Models	Avg	Tool-Call				Non-Tool-Call			Financial Indicators
Models	Avg	ST-SC	ST-MC	MT-SC	MT-MC	UD	CI	DR	ITA	KDA
Closed-Source LLMs
Gemini-2.5-Flash	40.85	1.92	4.60	22.22	18.71	100.00	38.46	100.00	25.93	10.25
Gemini-2.5-Pro	62.15	81.56	72.34	67.32	67.27	54.64	25.00	66.90	14.81	72.75
Claude-4.0-Sonnet	57.96	58.09	37.59	37.09	43.98	100.00	31.73	97.25	37.04	44.03

Table 3: Main results of FinToolBench. The table is split into Panel A (Prompt Mode) and Panel B (Function Calling Mode). The best and second-best results are highlighted separately for each panel.

Dimension	BFCL	$\tau$ -bench	FinToolBench
Tool Space	Small & Oracle	$<$ 20 APIs/domain	43,066 domain tools
Data Style	Explicit	Task-driven	Underspecified, event-driven
Candidate Env.	Static & given	Fixed domain	Dynamic + hard negatives
Task Focus	Syntax & exec.	State tracking	Intent elicitation & navigation
Finance-Specific	×	×	✓
Key Metrics	AST Match	Task Success	CB-HWS, KDA, ITA

Table 4: Comparison of FinToolBench with existing benchmarks.

Non-Tool Scenarios
Metric	FC Mode			Prompt Mode
Metric	GPT-4o	Qwen3-14B	FinTool-14B	GPT-4o	Qwen3-14B	FinTool-14B
Total Error Rate	38.2%	51.4%	58.7%	43.8%	68.9%	59.7%
Invalid Invocation	34.7%	41.9%	56.8%	35.2%	64.8%	59.0%
Intent/CI Failure	3.5%	9.5%	1.9%	8.6%	4.1%	0.6%
Correct Rate	61.8%	48.6%	41.3%	56.2%	31.1%	40.3%

Table 5: Fine-grained error distribution comparing FinTool-Qwen3-14B against baselines. Bold indicates best performance per column group.

Figure 3: Performance comparison of various general capabilities before and after training different models.

Model Setting	Methodology		Performance
Model Setting	Synthesis	Retrieval	Tool-Call	Non-Tool	Avg.
Base (Qwen3-8B)	-	-	48.05	52.75	50.40
Reverse Syn.	Reverse	Static	64.01	15.01	39.51
FinToolSyn (Static)	Forward	Static	65.84	44.03	54.93
FinToolSyn (Full)	Forward	Dynamic	68.31	43.40	55.86

Table 6: Ablation study. Reverse Synthesis improves tool calling but degrades general logic (Non-Tool score drops). FinToolSyn restores logic and enhances robustness via dynamic retrieval.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Round Type	Reply Type	Execution Pattern	Tool Context	Count	Pct. (%)
Single-turn	Tool Call	Single	Single-tool	63	7.47
Single-turn	Normal Reply	Null	Null	56	6.64
Single-turn	Tool Call	Single	Multi-tool	55	6.52
Multi-turn	Tool Call	Single	Single-tool	54	6.41
Single-turn	No Tool Reply	Null	Null	53	6.29
Multi-turn	Normal Reply	Null	Null	52	6.17

Appendix

Table 7: Detailed distribution of the 16 combined scenarios. The dataset maintains a balanced distribution across these combinations (50–63 samples each) to ensure comprehensive evaluation coverage without bias toward specific query types.

Category & Functions
Payment & Settlement
Payment transaction processing, Payment status query and monitoring, Settlement, clearing and reconciliation, Cross-border payments, Payment channel management
Insurance Services
Underwriting and quoting, Policy lifecycle management, Claims services and processing, Reinsurance and risk transfer, Insurance product catalog and configuration management, Insurance operations analysis and monitoring
Account & User Management
User registration and account opening, KYC and compliance management, User profile management, Identity authentication and authorization, Permissions and role management, Account lifecycle management, Account information query, Account security management

Appendix

Table 8: Financial API System Overview

Hyperparameter	Value
Optimizer	AdamW (Fused)
Learning Rate	$2\times 10^{-4}$
LR Scheduler	Cosine
Warmup Ratio	0.1
Global Batch Size	128
Gradient Accumulation Steps	16

Appendix

Table 9: Hyperparameters for Supervised Fine-Tuning (Applied to all 8B/14B and Prompt/FC models).

Panel of Figure 5 — Figure 5: System Prompt for Prompt Mode

Panel A: Prompt Mode
Model	Avg	Single-Turn							Multi-Turn
		Single-Tool			Multi-Tool				Single-Tool			Multi-Tool
		Avg	single	parallel	Avg	single	parallel	serial	Avg	single	parallel	Avg	single	parallel	serial
Closed-Source Models
Claude-4.0-Sonnet	44.19	58.09	73.93	42.25	37.59	45.81	25.98	40.99	37.09	37.18	36.99	43.98	52.20	39.46	40.28
Gemini-2.5-Flash	11.86	1.92	0.00	3.84	4.60	1.76	1.96	10.08	22.22	12.89	31.56	18.71	13.72	23.74	18.67

Appendix

Table 10: Detailed Tool-Call Performance Breakdown. Overall Acc is calculated as the average of OA across four scenarios.

Panel A: Prompt Mode
Model	Overall Acc	No-Tool-Reply			Follow-up			Normal-Reply
Model	Overall Acc	Overall	Single-Turn	Multi-Turn	Overall	Single-Turn	Multi-Turn	Overall	Single-Turn	Multi-Turn
Closed-Source Models
Claude-4.0-Sonnet	57.96	100.00	31.73	30.77	38.46	26.83	50.00	97.25	27.59	78.57
Gemini-2.5-Flash	40.85	100.00	38.46	36.54	25.93	10.25	100.00	100.00	25.93	10.25
Gemini-2.5-Pro	62.15	54.64	25.00	21.15	66.90	60.71	73.08	14.81	72.75	54.64

Appendix

Table 11: Non-Tool-Call Performance Comparison

Tool-Call Performance (Ablation Study)
Model	Avg	Single-Turn							Multi-Turn
		Single-Tool			Multi-Tool				Single-Tool			Multi-Tool
		Avg	single	parallel	Avg	single	parallel	serial	Avg	single	parallel	Avg	single	parallel	serial
Base (Qwen3-8B)	48.03	66.55	69.66	63.45	43.37	51.47	45.59	33.06	39.22	38.12	40.32	42.97	57.20	47.54	24.16
Ablation-Reverse	64.01	73.85	73.59	74.12	53.52	58.33	42.06	60.19	68.43	77.53	59.33	60.24	67.87	58.13	54.71
Ablation-Static	65.84	72.12	69.83	74.41	57.32	59.67	55.95	56.35	70.62	74.54	66.70	63.30	72.66	63.23	54.02

Appendix

Table 12: Detailed Tool-Call Performance for Ablation Models (FC Mode). The Full Method achieves the best balance, particularly in complex serial tasks.

Non-Tool-Call Performance (Ablation Study)
Model	Overall Acc	No-Tool-Reply			Follow-up			Normal-Reply
Model	Overall Acc	Overall	Single-Turn	Multi-Turn	Overall	Single-Turn	Multi-Turn	Overall	Single-Turn	Multi-Turn
Base (Qwen3-8B)	50.05	73.42	52.83	94.00	3.85	1.92	5.77	80.98	69.64	92.31
Ablation-Reverse	43.01	18.72	9.43	28.00	0.00	0.00	0.00	26.30	16.07	36.54
Ablation-Static	56.49	39.15	28.30	50.00	48.08	42.31	53.85	44.85	33.93	55.77
FinTool-Qwen3-8B (Full)	57.64	42.98	33.96	52.00	44.23	38.46	50.00	42.99	32.14	53.85

Appendix

Table 13: Non-Tool-Call Performance for Ablation Models. Reverse Synthesis causes a collapse in rejection and clarification capabilities, while our Full Method restores them.

Panel of Figure 6 — Figure 6: Prompt for Clarification Intent Detection.

Panel of Figure 7 — Figure 7: Prompt for Tool Selection Evaluation.

Panel of Figure 8 — Figure 8: Prompt for Parameter Evaluation.

Qwen3-8B-FC (Baseline)	Ours-8B-FC (Finetuned)
Case 1: Entity Recognition & Stock Code Matching
User Question: The financial report of "Xiamen Gibit" has a standard unqualified audit opinion. How reliable is this data?
Called Tool: evaluate_financial_report_reliability
Filled Parameters: {	Filled Parameters: {
"stock_code": "300430.SZ" ,	"stock_code": "603444.SH" ,
"comparison_scope": "Industry"	// Optional parameter omitted (valid)

Appendix

Table 14: Performance comparison between Qwen3-8B-FC and Ours-8B-FC in financial tool calling scenarios.

Model	GSM8K	MMLU	HumanEval	BFCL	$\tau$ -bench
Qwen3-8B (base)	90.98	85.16	89.63	0.430	0.328
FinTool-Qwen3-8B	90.90	84.34	89.02	0.469	0.328
$\Delta$	$-$ 0.08	$-$ 0.82	$-$ 0.61	+0.039	0.000
Qwen3-14B (base)	92.04	87.56	92.68	0.468	0.333
FinTool-Qwen3-14B	90.30	87.02	94.51	0.443	0.383
$\Delta$	$-$ 1.74	$-$ 0.54	+1.83	$-$ 0.025	+0.050

Appendix

Table 15: Generalization performance before and after fine-tuning.

\Delta

= FinTool score

-

Base score. Bold highlights notable improvements.

LLM Tool Use

Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.

See all 295 papers

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
Mar 9, 2026Jiaxuan Lu, Kong Wang, Yemin Wang +9LLM Agent Evaluation Tool-Using Language Agents
Shanghai AI Laboratory · Hunan University · Xiamen University +3
FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks
Apr 11, 2026Yupeng Cao, Haohang Li, Weijin Liu +11Agent Trajectory Analysis LLM Tool Use
Stevens Institute of Technology · Independent Researcher · The FinAI +2
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
Mar 26, 2026Jie Zhu, Yimin Tian, Boyang Li +8LLM Agent Evaluation LLM Tool Use
Qwen DianJin Team, Alibaba Cloud Computing · YINGMI Wealth Management · Soochow University

Mar 9, 2026J/K move · Enter open · S save

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

Jiaxuan Lu, Kong Wang, Yemin Wang, Qingmei Tang+8

Shanghai AI Laboratory · Hunan University · Xiamen University+3

LLM Agent Evaluation Tool-Using Language Agents Tool Use AI Agent Benchmarks Finance

Open paper AskPDF arXiv

The integration of Large Language Models (LLMs) into the financial domain is driving a paradigm shift from passive information retrieval to dynamic, agentic interaction. While general-purpose tool learning has witnessed a surge in benchmarks, the financial sector, characterized by high stakes, strict compliance, and rapid data volatility, remains critically underserved. Existing financial evaluations predominantly focus on static textual analysis or document-based QA, ignoring the complex reality of tool execution. Conversely, general tool benchmarks lack the domain-specific rigor required for finance, often relying on toy environments or a negligible number of financial APIs. To bridge this gap, we introduce FinToolBench, the first real-world, runnable benchmark dedicated to evaluating financial tool learning agents. Unlike prior works limited to a handful of mock tools, FinToolBench establishes a realistic ecosystem coupling 760 executable financial tools with 295 rigorous, tool-required queries. We propose a novel evaluation framework that goes beyond binary execution success, assessing agents on finance-critical dimensions: timeliness, intent type, and regulatory domain alignment. Furthermore, we present FATR, a finance-aware tool retrieval and reasoning baseline that enhances stability and compliance. By providing the first testbed for auditable, agentic financial execution, FinToolBench sets a new standard for trustworthy AI in finance. The tool manifest, execution environment, and evaluation code will be open-sourced to facilitate future research.

Abstract

LLM Tool Use

Explore similar work

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol