Organizations: Shenzhen Technology University Shenzhen, China · FiT, Tencent Shenzhen, China · Huazhong University of Science and Technology Wuhan, China · McGill University Montreal, Canada · The Hong Kong University of Science and Technology Hong Kong SAR, China · Shenzhen University Shenzhen, China
Large language models (LLMs) have been incorporated into numerous industrial applications. Meanwhile, a vast array of API assets is scattered across various functions in the financial domain. An online financial question-answering system can leverage both LLMs and private APIs to provide timely financial analysis and information. The key is equipping the LLM model with function calling capability tailored to a financial scenario. However, a generic LLM requires customized financial APIs to call and struggles to adapt to the financial domain. Additionally, online user queries are diverse and contain out-of-distribution parameters compared with the required function input parameters, which makes it more difficult for a generic LLM to serve online users. In this paper, we propose a data-driven pipeline to enhance function calling in LLM for our online, deployed financial QA, comprising dataset construction, data augmentation, and model training. Specifically, we construct a dataset based on a previous study and update it periodically, incorporating queries and an augmentation method named AugFC. The addition of user query-related samples will exploit our financial toolset in a data-driven manner, and AugFC explores the possible parameter values to enhance the diversity of our updated dataset. Then, we train an LLM with a two-step method, which enables the use of our financial functions. Extensive experiments on existing offline datasets, as well as the deployment of an online scenario, illustrate the superiority of our pipeline. The related pipeline has been adopted in the financial QA of YuanBao\footnote{https://yuanbao.tencent.com/chat/}, one of the largest chat platforms in China.
Figures & tables
Figure 1. The overview of the key points and corresponding challenges in the online financial QA system powered by LLM.
Figure 2. The data-driven pipeline consisting of data collection, data construction, data augmentation, and model training.
Figure 3. The prompt template for prompt isolation.
Model Size
Training dataset
API-Bank L1
API-Bank L2
Tool-Alpaca
Seal-Tools
Nexus Raven
xLAM-small
Avg
Qwen2.5-1.5B
vanilla
0.6525
0.4293
0.4290
0.7491
0.4988
0.5577
0.5527
src-only
0.7826
0.6533
0.6084
0.8865
0.5348
0.5934
0.6765
aug-only
0.6973
0.4992
0.4242
0.8551
0.7500
0.6857
0.6519
src+aug
0.7226
0.6648
0.6233
0.8898
0.7840
0.7288
0.7256
Qwen2.5-7B
vanilla
0.6985
0.6036
0.6158
0.8083
0.7283
0.5914
0.6743
src-only
0.8295
0.6529
0.6745
0.9426
0.7892
0.7428
0.7719
Table 1. The overall performance to validate our AugFC. The vanilla denotes the base model without any training, the src-only denotes we only use the seed data, aug-only means the augmented data is used, and src+aug represents the combined dataset. All results are measured using the F1 score. The best results are bold, and the second-best results are underlined.
Figure 4. The number of blind spots to repair with different LLMs.
Figure 5. The heatmap illustrating how τg and τb affects the performance resepectively.
Model
AugFC
w/o blind spots
w/o designed prompt
w/o multi-round
Qwen2.5-1.5B
0.726
0.689
0.672
0.695
Qwen2.5-7B
0.786
0.742
0.738
0.752
Qwen2.5-32B
0.806
0.765
0.772
0.780
Table 2. The ablation study of AugFC. The results are the averages of the six benchmark datasets.
Model Size
Model Type
API-Bank L1
API-Bank L2
Tool-Alpaca
Seal-Tools
Nexus Raven
xLAM-small
Avg
1.5B
Qwen2.5-1.5B
0.6525
0.4293
0.4290
0.7491
0.4988
0.5577
0.5527
xLAM-1.3B-fc
0.8370
0.6432
0.5058
0.8043
0.5480
0.5368
0.6459
Hammer-1.5B
0.7230
0.5971
0.5348
0.8865
0.5688
0.5192
0.6382
ours
0.7226
0.6048
0.6233
0.8898
0.7840
0.7288
0.7256
7B
Qwen2.5-7B
0.6985
0.6036
0.6158
0.8083
0.7283
0.5914
0.6743
xLAM-7B-fc
0.8069
0.6424
0.5896
0.7687
0.5409
0.6378
0.6644
Table 3. The comparison of our two-step training model with existing models. All results are measured using the F1 score. The best results are bold, and the second-best results are underlined.
Figure 6. The overview illustration of the online financial QA system. Our pipeline is marked as red in the whole system.
Figure 7. The online results measured by four metrics.
Large language models are becoming increasingly significant in financial applications. Nevertheless, prevailing benchmarks are largely dependent on simulated or generic data, which leads to a significant gap between reported performance and actual efficacy in real-world scenarios. To tackle this challenge, we present BizFinBench.v2, the first integrated offline and online benchmark built upon authentic user query-response data from both Chinese and U.S. equity markets. It comprises 28,860 questions across eight offline and two online tasks. Experimental results show that GPT-5 achieves a mere 61.5% accuracy, still failing to meet the practical business requirement (84.8%). Among the evaluated commercial models, DeepSeek-R1 exhibits superior investment efficacy. Error analysis grounded in real financial practice reveals persistent limitations in existing models. By overcoming the constraints of prior benchmarks, BizFinBench.v2 provides a substantiated foundation for advancing LLM deployment in the financial sector. Our data and code are available at https://github.com/HiThink-Research/BizFinBench.v2.
Xin Guo, Rongjunchen Zhang, Guilong Lu +4
HiThink Research · Shanghai University of Finance and Economics
Large Language Models (LLMs) have demonstrated strong general capabilities, yet their deployment in finance remains challenging due to dense domain-specific terminology, stringent numerical reasoning requirements, and low tolerance for factual errors. We conduct a controlled empirical study showing that in specialized vertical domains, performance is largely determined by the quality and difficulty/verifiability profile of post-training data. We introduce ODA-Fin-SFT-318k, constructed via multi-stage distillation and verification to produce high-quality Chain-of-Thought supervision, and ODA-Fin-RL-12k, curated for hard-but-verifiable tasks that balance reward precision and task diversity. Using standard SFT and RL pipelines, we show that high-quality CoT distillation establishes a robust foundation during SFT, while difficulty- and verifiability-aware sampling improves RL generalization. Evaluated on nine benchmarks spanning general financial tasks, sentiment analysis, and numerical reasoning, our ODA-Fin-RL-8B consistently surpasses open-source state-of-the-art (SOTA) financial LLMs of comparable size. We release our ODA-Fin-SFT-318k and ODA-Fin-RL-12k datasets, along with trained models to advance data-centric financial AI research.
Chuxue Cao, Honglin Lin, Zhanping Zhong +5
Shanghai Artificial Intelligence Laboratory, OpenDataLab, OpenDataArena · Hong Kong University of Science and Technology
Large Language Models (LLMs) have significantly advanced online data services, particularly in the domain of financial question answering (FinQA). However, such systems remain susceptible to numerical reasoning hallucinations, which critically undermine reliability in high-stakes financial applications. Although retrieval-augmented generation (RAG) has been widely adopted to ground responses in external knowledge, it introduces three persistent challenges: noise sensitivity, calculation fragility, and an auditability crisis. Existing model-centric approaches, which primarily focus on optimizing either the retriever or generator in isolation, still struggle to address these issues in an integrated manner. In this work, we pioneer a data-centric paradigm and propose a novel framework, the Data-centric Reasoning Compiler (DCRC). The framework operates through three cohesive phases: (1) adversarial data construction, which synthesizes training examples with controlled noise to teach robustness; (2) multi-stage training that cultivates a Data-centric Structuring Agent (DSA) capable of explicit evidence auditing and program synthesis; and (3) a compile-and-execute inference process, where the DSA transforms user queries and retrieved documents into verifiable, executable reasoning programs. This data-driven framework ensures faithful numerical reasoning by design. We conduct extensive experiments on established offline benchmarks and further validate our framework through deployment in a real-world online financial QA system.
Hao Chen, Xing Tang, Qirui Liu +6
Shenzhen Technology University FiT, Tencent Shenzhen, China · South China University of Technology Guangzhou, China · The Hong Kong University of Science and Technology Hong Kong SAR, China +2