The integration of Large Language Models (LLMs) into the financial domain is driving a paradigm shift from passive information retrieval to dynamic, agentic interaction. While general-purpose tool learning has witnessed a surge in benchmarks, the financial sector, characterized by high stakes, strict compliance, and rapid data volatility, remains critically underserved. Existing financial evaluations predominantly focus on static textual analysis or document-based QA, ignoring the complex reality of tool execution. Conversely, general tool benchmarks lack the domain-specific rigor required for finance, often relying on toy environments or a negligible number of financial APIs. To bridge this gap, we introduce FinToolBench, the first real-world, runnable benchmark dedicated to evaluating financial tool learning agents. Unlike prior works limited to a handful of mock tools, FinToolBench establishes a realistic ecosystem coupling 760 executable financial tools with 295 rigorous, tool-required queries. We propose a novel evaluation framework that goes beyond binary execution success, assessing agents on finance-critical dimensions: timeliness, intent type, and regulatory domain alignment. Furthermore, we present FATR, a finance-aware tool retrieval and reasoning baseline that enhances stability and compliance. By providing the first testbed for auditable, agentic financial execution, FinToolBench sets a new standard for trustworthy AI in finance. The tool manifest, execution environment, and evaluation code will be open-sourced to facilitate future research.
Figures & tables
Figure 1: FinToolBench overview. Left: the scope of our benchmark across representative categories. Right: an example of the standardized execution pipeline, where an LLM agent selects a tool, observes the environment output, and produces a final answer with an auditable tool trace.
Figure 2: FinToolBench dataset construction pipeline. Stage 1 collects raw tool sources. Stage 2 performs tool curation and executability filtering to obtain a validated tool inventory. Stage 3 normalizes tools into a unified manifest with standardized signatures, canonical arguments, and aligned output schemas. Stage 4 annotates each tool with finance attributes (timeliness, intent type, regulatory domain). Stage 5 sources and selects tool-required questions. Stage 6 aligns questions with tools via semantic retrieval, multi-sample verification, and execution checks. Stage 7 adds human-in-the-loop quality assurance. Stage 8 outputs the benchmark tool library and benchmark question set as a runnable benchmark.
Attribute
Values
Evaluation role
timeliness
realtime , daily , as_filed , periodic , static
Penalize stale calls when timeliness is required.
intent_type
informational , advisory , transactional
Penalize escalation beyond user intent.
regulatory_domain
set-valued
Penalize domain-mismatched tool usage.
Table 1: Finance attribute schema used in FinToolBench.
Figure 3: Overview of Finance-Aware Tool Routing (FATR). FATR takes a Question Inventory and a Tool Inventory & Retrieval module that performs Top- K retrieval and formats retrieved tools as Tool Cards. An LLM Planner runs Finance-Aware Tool Routing by (A) Infer Constraints to derive (T(q),I(q),D(q)) over timeliness, intent, and domain, (B) Constraint-Aware Planning, and (C) a ReAct loop. An Executor & Trace Recorder dispatches tool calls and trace logs, which are then scored by Evaluation & Metrics for capability ( TIR , TESR , CER , Soft Score , CSS ) and compliance ( TMR , IMR , DMR ).
Figure 4: Tool cards for attribute injection and constraint checking.
Model
TIR
TESR
CER ↑
Soft Score ↑
CSS ↑
TMR ↓
IMR ↓
DMR ↓
Doubao-Seed-1.6
0.6508
0.3254
0.5000
0.4627
0.3958
0.3438
0.6563
0.1719
Qwen3-8B
0.8712
0.2949
0.3385
0.4040
0.4234
0.3307
0.6887
0.1673
GLM-4.7-Flash
0.4407
0.2102
0.4769
0.3791
0.2769
0.4615
0.7231
0.1769
Claude-Sonnet-4.5
0.4814
0.2407
0.5000
0.7119
0.5775
0.2746
0.6479
0.0423
GPT-5.4
0.2407
0.1322
0.5493
0.6254
0.4103
0.2535
0.5775
0.0141
Grok-3-beta
0.2576
0.1932
0.7500
0.6847
0.4298
0.1316
0.4868
0.0132
Table 2: Main results on FinToolBench. Higher is better for TIR , TESR , CER , Soft Score , and CSS ; lower is better for TMR , IMR , and DMR .
Figure 5: Attribute injection ablation in FATR. We compare full tool cards with finance tags against a variant without attribute injection. Attribute injection improves execution success conditioned on tool use ( CER ) and reduces mismatch rates ( TMR , IMR , DMR ).
Figure 6: Last-round tool usage on FinToolBench.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Criterion
Rule
Description
Missing or empty → excluded.
Deduplication
Duplicate names: keep first only.
Authentication
Exclude if Authorization or multi-step token (beyond single API key) required.
Existence
Remove if endpoint does not resolve or returns persistent errors.
Bank-card
Exclude if free-tier requires card binding.
Rate limits
Require ≥ 10/h, ≥ 100/d, ≥ 300/m.
Appendix
Table 3: First-stage RapidAPI filter criteria.
Field
Description
step
The sequential order of the call within the multi-turn process.
tool_name
The identifier of the specific tool invoked ( e.g. , symbols_sec_filings ).
parameters
The JSON-formatted arguments generated by the model.
output
The tool response, including data or structured error messages.
error
Overall execution status, which captures null or specific system-level failures.
Appendix
Table 4: Normalized tool-trace fields.
Validation target
Agreement
Secondary metric
Question timeliness
85.0%
Jaccard 0.82
Question intent
93.3%
Jaccard 0.91
Question domain
80.0%
Jaccard 0.76
Trace TMR
86.0%
F1 0.83
Trace IMR
94.0%
F1 0.91
Trace DMR
82.0%
F1 0.78
Appendix
Table 5: Human validation of question-side attributes and trace-level compliance.
K
1
5
10
20
30
Tool Hit@ K
61.2%
75.1%
83.0%
88.1%
88.3%
Appendix
Table 6: Retrieval-depth ablation. Tool Hit@ K measures whether the gold required tool set is covered by the retrieved candidate pool.
Figure 7: Metrics by category for Doubao-Seed-1.6 on FinToolBench. Rows are question categories and columns are evaluation metrics.
Figure 8: Single-box, end-to-end comparison for financebench_9 . The baseline fails due to an interface mismatch. FATR recovers a valid tool trace, but the final answer remains incorrect because it does not follow the dataset’s intended reasoning criterion.
Figure 9: Single-box comparison for openfindata_release_155 . Injected finance attributes reduce redundant and incompatible tool calls while preserving correctness.
Figure 10: Single-box comparison for financebench_33 . Injected finance attributes change the tool strategy, but the resulting proxy-based estimate remains incorrect.