FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks

Yupeng Cao; Haohang Li; Weijin Liu; Wenbo Cao; Anke Xu; Lingfei Qian; Xueqing Peng; Minxue Tang

cs.AIApr 11, 2026

FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks

Authors: Yupeng Cao, Haohang Li, Weijin Liu, Wenbo Cao, Anke Xu, Lingfei Qian, Xueqing Peng, Minxue Tang,

Organizations: Stevens Institute of Technology · Independent Researcher · The FinAI · Duke University ·

Abstract

Recent studies demonstrate that tool-calling capability enables large language models (LLMs) to interact with external environments for long-horizon financial tasks. While existing benchmarks have begun evaluating financial tool calling, they focus on limited scenarios and rely on call-level metrics that fail to capture trajectory-level reasoning quality. To address this gap, we introduce FinTrace, a benchmark comprising 800 expert-annotated trajectories spanning 34 real-world financial task categories across multiple difficulty levels. FinTrace employs a rubric-based evaluation protocol with nine metrics organized along four axes -- action correctness, execution efficiency, process quality, and output quality -- enabling fine-grained assessment of LLM tool-calling behavior. Our evaluation of 13 LLMs reveals that while frontier models achieve strong tool selection, all models struggle with information utilization and final answer quality, exposing a critical gap between invoking the right tools and reasoning effectively over their outputs. To move beyond diagnosis, we construct FinTrace-Training, the first trajectory-level preference dataset for financial tool-calling, containing 8,196 curated trajectories with tool-augmented contexts and preference pairs. We fine-tune Qwen-3-8B/32B using supervised fine-tuning followed by direct preference optimization (DPO) and show that training on FinTrace-Training consistently improves intermediate reasoning metrics, with DPO more effectively suppressing failure modes. However, end-to-end answer quality remains a bottleneck, indicating that trajectory-level improvements do not yet fully propagate to final output quality.

Figures & tables

Figure 1: FinTrace benchmark construction pipeline.

Axis	Metric	Type	What it captures
Action Correctness	Tool-Calling F1	Algorithmic	Set- and bag-level tool selection accuracy
Action Correctness	Task Relevance	LLM-judged	Relevance of each tool call to the query
Execution Efficiency	Step Efficiency	Algorithmic	Conciseness relative to the golden trajectory
Execution Efficiency	Redundancy Score	Algorithmic	Absence of duplicate tool calls
Process Quality	Logical Progression	LLM-judged	Coherence and ordering of the tool-call sequence
	Information Utilization	LLM-judged	Effective use of retrieved data in reasoning

Table 1: Overview of the nine-metric rubric evaluation protocol. Algorithmic metrics are scored on a continuous

[0,1]

scale; LLM-judged metrics use a five-point Likert scale normalized to

[0,1]

for aggregation.

Rank	Model	Tool Call	Step	Redund.	Pass	Task	Logical	Info	Progress	Answer	Overall
		F1	Effic.	Score	Rate	Relev.	Prog.	Util.	Score	Quality	Score
1	Claude-Opus-4.6	0.896	0.926	0.997	2.65	4.14	4.51	3.23	3.49	3.34	0.788
2	Claude-Sonnet-4.6	0.799	0.879	0.996	2.65	4.06	4.04	3.13	3.42	3.11	0.750
3	GPT-5.4	0.804	0.966	0.994	3.00	3.79	3.71	2.89	2.99	2.99	0.737
4	Gemini-3.1-Pro	0.810	0.735	0.995	2.36	3.80	3.76	2.68	2.97	2.71	0.688
5	KiMi-2.5	0.729	0.773	0.987	2.04	3.63	3.35	2.22	2.73	2.52	0.643

Table 2: Performance of 13 LLMs on the FinTrace test set. Tool Calling F1, Step Efficiency, and Redundancy Score are reported on a [0, 1] scale; all other metrics are on a [1, 5] scale. The Overall score is the mean of all nine metrics after normalizing to [0, 1]. Bold indicates the best result per column; underline indicates the second best.

Figure 2: Overall performance of 13 LLMs on the FinTrace benchmark.

Figure 3: Mean scores on all nine rubric metrics for the 8B (blue) and 32B (orange) models, evaluated on the same query test set. Within each model family, shades encode the training stage: Base (light), SFT (dark), and DPO (darkest, hatched). Left: algorithmic metrics (0–1). Right: LLM-judged metrics (1–5).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Distribution of 15,095 source queries across 12 task buckets encompassing 34 financial tasks.

Robustness check	Statistic	Value
Out-of-family judge (DeepSeek-V4-Pro)	Spearman’s $\rho$ (overall)	0.989
	Kendall’s $\tau$ (overall)	0.949
	Pearson $r$ (overall)	0.994
Human–human agreement (Krippendorff’s $\alpha$ )	Task Relevance	0.709
	Logical Progression	0.979
	Final Answer Quality	0.567

Appendix

Table 3: Summary of the three robustness studies. Rank correlations are computed over all 13 evaluated models; agreement coefficients are computed over 90 human-annotated trajectories.

Rank	Model	Pass	Task	Logical	Info	Progress	Answer	Overall
		Rate	Relev.	Prog.	Util.	Score	Quality
1	Claude-Opus-4.6	1.93	4.29	4.28	2.84	4.11	2.93	0.767
2	GPT-5.4	2.60	4.04	3.72	2.70	3.44	2.79	0.734
3	Claude-Sonnet-4.6	1.94	4.28	3.72	2.77	4.04	2.76	0.731
4	Gemini-3.1-Pro	1.92	3.95	3.32	2.19	3.26	2.43	0.666
5	KiMi-2.5	1.71	3.82	3.18	1.95	3.27	2.39	0.643

Appendix

Table 4: Evaluation under the out-of-family judge DeepSeek-V4-Pro. All LLM-judged metrics are on a

[1,5]

scale. Tool-Calling F1, Step Efficiency, and Redundancy Score are computed deterministically and are identical to Table 2 ; the Overall score is the mean of all nine metrics after normalization to

[0,1]

Model	Annotator	Final Answer Quality	Task Relevance	Logical Progression	Average
GPT-5.4	A	3.20	4.20	3.70	3.70
	B	2.40	4.30	3.90	3.53
	Consensus	2.80	4.25	3.80	3.62
Qwen3.5-397B	C	2.90	3.80	3.20	3.30
	D	2.10	3.70	3.00	2.93
	Consensus	2.50	3.75	3.10	3.12

Appendix

Table 5: Human annotation scores on 90 sampled trajectories. Consensus is the mean of the two annotators assigned to each model. The model ordering is identical for every annotator and for the consensus across all three metrics.

Metric	$\alpha$ (Human–Human)	$\kappa_{w}$ (LLM–Human Consensus)
Task Relevance	0.709	0.631
Logical Progression	0.979	0.857
Final Answer Quality	0.567	0.720

Appendix

Table 6: Inter-annotator agreement (Krippendorff’s

\alpha

) and agreement between the LLM judge and the human consensus (quadratic weighted Cohen’s

\kappa

Dropped metric	Kendall’s $\tau$	Top-3 overlap	Top-5 overlap
Tool-Calling F1	0.974	1.00	1.00
Step Efficiency	0.974	1.00	0.80
Redundancy Score	1.000	1.00	1.00
Pass Rate	1.000	1.00	1.00
Task Relevance	1.000	1.00	1.00
Logical Progression	1.000	1.00	1.00

Appendix

Table 7: Leave-one-metric-out sensitivity. Rank correlations are computed against the published ranking over all 13 models.

Aggregation variant	Kendall’s $\tau$	Top-3 overlap	Top-5 overlap
2-axis (algorithmic / LLM-judged)	0.974	1.00	1.00
3-axis (tool-use / process / outcome)	0.974	1.00	1.00
Median aggregation	0.974	1.00	1.00

Appendix

Table 8: Alternative aggregation schemes compared against the published ranking. All three preserve the top-7 ordering and differ only by one mid-table swap.

Parameter	8B
Learning rate	$1\times 10^{-5}$
Epochs	15.9
Effective batch size	24
Max sequence length	16,384
LoRA rank ( $r$ )	128
LoRA alpha ( $\alpha$ )	128

Appendix

Table 9: Training configuration for Qwen 3 8B.

Parameter	8B
Learning rate	$1\times 10^{-6}$
Epochs	5.7
Effective batch size	32
Max sequence length	16,384
LoRA rank ( $r$ )	128
LoRA alpha ( $\alpha$ )	128

Appendix

Table 10: DPO training configuration for Qwen 3 8B.

Parameter	Value
Max sequence length	28,000
Max turns per rollout	7
Sampling (top- $p$ )	0.95

Appendix

Table 11: Rollout configuration for rejected trajectory generation for Qwen 3 8B.

Parameter	Value
Max sequence length	28,000
Max turns per rollout	7
Sampling (top- $p$ )	0.95

Appendix

Table 12: Rollout configuration for trajectory evaluation for Qwen 3 8B.

Parameter	32B
Learning rate	$5\times 10^{-5}$
Epochs	7.2
Effective batch size	64
Max sequence length	16,384
LoRA rank ( $r$ )	128
LoRA alpha ( $\alpha$ )	128

Appendix

Table 13: Training configuration for Qwen 3 32B.

Parameter	8B
Learning rate	$1\times 10^{-6}$
Epochs	6
Effective batch size	32
Max sequence length	16,384
LoRA rank ( $r$ )	128
LoRA alpha ( $\alpha$ )	128

Appendix

Table 14: DPO training configuration for Qwen 3 32B.

Parameter	Value
Max sequence length	28,000
Max turns per rollout	7
Sampling (top- $p$ )	0.95

Appendix

Table 15: Rollout configuration for rejected trajectory generation for Qwen 3 32B.

Parameter	Value
Max sequence length	28,000
Max turns per rollout	7
Sampling (top- $p$ )	0.95

Appendix

Table 16: Rollout configuration for trajectory evaluation for Qwen 3 32B.

Agent Trajectory Analysis

Agent trajectory analysis studies how agents move or act over time to predict behavior, evaluate execution, and diagnose failures.

See all 47 papers

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
Mar 9, 2026Jiaxuan Lu, Kong Wang, Yemin Wang +9LLM Agent Evaluation Tool-Using Language Agents
Shanghai AI Laboratory · Hunan University · Xiamen University +3
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
Mar 26, 2026Jie Zhu, Yimin Tian, Boyang Li +8LLM Agent Evaluation LLM Tool Use
Qwen DianJin Team, Alibaba Cloud Computing · YINGMI Wealth Management · Soochow University
Time Series Augmented Generation for Financial Applications
Apr 21, 2026Anton Kolonin, Alexey Glushchenko, Evgeny Bochkov +1LLM Agent Evaluation AI Agent Benchmarks
SingularityNET Foundation

Mar 9, 2026J/K move · Enter open · S save

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

Jiaxuan Lu, Kong Wang, Yemin Wang, Qingmei Tang+8

Shanghai AI Laboratory · Hunan University · Xiamen University+3

LLM Agent Evaluation Tool-Using Language Agents Tool Use AI Agent Benchmarks Finance

Open paper AskPDF arXiv

The integration of Large Language Models (LLMs) into the financial domain is driving a paradigm shift from passive information retrieval to dynamic, agentic interaction. While general-purpose tool learning has witnessed a surge in benchmarks, the financial sector, characterized by high stakes, strict compliance, and rapid data volatility, remains critically underserved. Existing financial evaluations predominantly focus on static textual analysis or document-based QA, ignoring the complex reality of tool execution. Conversely, general tool benchmarks lack the domain-specific rigor required for finance, often relying on toy environments or a negligible number of financial APIs. To bridge this gap, we introduce FinToolBench, the first real-world, runnable benchmark dedicated to evaluating financial tool learning agents. Unlike prior works limited to a handful of mock tools, FinToolBench establishes a realistic ecosystem coupling 760 executable financial tools with 295 rigorous, tool-required queries. We propose a novel evaluation framework that goes beyond binary execution success, assessing agents on finance-critical dimensions: timeliness, intent type, and regulatory domain alignment. Furthermore, we present FATR, a finance-aware tool retrieval and reasoning baseline that enhances stability and compliance. By providing the first testbed for auditable, agentic financial execution, FinToolBench sets a new standard for trustworthy AI in finance. The tool manifest, execution environment, and evaluation code will be open-sourced to facilitate future research.

Abstract

Agent Trajectory Analysis

Explore similar work

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

Time Series Augmented Generation for Financial Applications