BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios

Yunseung Lee; Subin Kim; Youngjun Kwak; Jaegul Choo

cs.CLFeb 19, 2026

BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios

Authors: Yunseung Lee, Subin Kim, Youngjun Kwak, Jaegul Choo

Organizations: KakaoBank Corp., South Korea · Korea Advanced Institute of Science and Technology, South Korea

Abstract

Large language models (LLMs)-based chatbots are increasingly being adopted in the financial domain, particularly in digital banking, to handle customer inquiries about products such as deposits, savings, and loans. However, these models still exhibit low accuracy in core banking computations-including total payout estimation, comparison of products with varying interest rates, and interest calculation under early repayment conditions. Such tasks require multi-step numerical reasoning and contextual understanding of banking products, yet existing LLMs often make systematic errors-misinterpreting product types, applying conditions incorrectly, or failing basic calculations involving exponents and geometric progressions. However, such errors have rarely been captured by existing benchmarks. Mathematical datasets focus on fundamental math problems, whereas financial benchmarks primarily target financial documents, leaving everyday banking scenarios underexplored. To address this limitation, we propose BankMathBench, a domain-specific dataset that reflects realistic banking tasks. BankMathBench is organized in three levels of difficulty-basic, intermediate, and advanced-corresponding to single-product reasoning, multi-product comparison, and multi-condition scenarios, respectively. When trained on BankMathBench, open-source LLMs exhibited notable improvements in both formula generation and numerical reasoning accuracy, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning. With tool-augmented fine-tuning, the models achieved average accuracy increases of 57.6%p (basic), 75.1%p (intermediate), and 62.9%p (advanced), representing significant gains over zero-shot baselines. These findings highlight BankMathBench as a reliable benchmark for evaluating and advancing LLMs' numerical reasoning in real-world banking scenarios.

Figures & tables


Model	Basic	Interm.	Adv.

Financial LLMs
AdaptLLM ( Cheng et al., 2024 )	0.4	0.0	0.0
Fin-o1 ( Qian et al., 2025 )	3.2	3.3	0.3

Table 1: Zero-shot accuracy (%) of various models across the Basic, Intermediate (Interm.), and Advanced (Adv.) difficulty levels defined in the BankMathBench dataset. Inst. denotes instruction-tuned models.

Figure 1: Examples of frequently asked customer queries in real banking branches.

Figure 2: Overview of the BankMathBench data generation pipeline, which comprises three stages: (a) question generation, (b) solution generation and automatic verification, and (c) reasoning generation.


Level	Language	Train Set	Test Set

Basic	Korean	2,236	559
	English	2,236	559
Intermediate	Korean	1,708	426
	English	1,702	423

Table 2: Dataset statistics by difficulty level, showing the number of training and test samples for both Korean and English.


Level	Korean Question	English Question

Basic	매달 50만원씩 2년 동안 자유적금에 넣으면, 연 이자율이 복리 5.2%일 때 만기 수령 금액은 얼마인가요?	If you deposit 500,000 KRW every month into a free savings account for 2 years, what will be the maturity amount if the annual compound interest rate is 5.2%?

Intermediate	3년 만기 정기예금에 연 5.2% 단리로 700만원을 넣는 것과, 매월 20만원씩 36개월 적금에 연 6.5% 복리로 적금하는 것 중 만기 수령액 차이가 얼마나 되나요?	What is the difference in the maturity amount between depositing 7 million won in a 3-year fixed deposit with an annual simple interest rate of 5.2%, and saving 200,000 won every month for 36 months with an annual compound interest rate of 6.5%?

Table 3: Representative questions from BankMathBench dataset demonstrating variations in difficulty levels.

Figure 3: Example of the question augmentation process.

Figure 4: Instruction format used across all models for zero-shot evaluation and supervised fine-tuning (SFT).


Model Info		Zero-shot		SFT		Tool Aug. SFT
Model	Size	Kor	Eng	Kor	Eng	Kor	Eng
Claude Sonnet-3.5	-	69.2	66.7	-	-	-	-
Gemini-2.5-Flash	-	64.9	71.2	-	-	-	-
GPT-4o	-	67.8	67.8	-	-	-	-

Table 4: Results on the BankMathBench Basic dataset. Model accuracy (%) is reported for Zero-shot, SFT, and Tool-Augmented SFT settings across Korean (Kor) and English (Eng).


Model Info		Zero-shot		SFT		Tool Aug. SFT
Model	Size	Kor	Eng	Kor	Eng	Kor	Eng

Claude Sonnet-3.5	-	10.3	9.9	-	-	-	-
Gemini-2.5-Flash	-	10.7	11.8	-	-	-	-
GPT-4o	-	7.9	14.4	-	-	-	-

Table 5: Results on the BankMathBench Intermediate dataset. Model accuracy (%) is reported for Zero-shot, SFT, and Tool-Augmented SFT settings across Korean (Kor) and English (Eng).


Model Info		Zero-shot		SFT		Tool Aug. SFT
Model	Size	Kor	Eng	Kor	Eng	Kor	Eng

Claude Sonnet-3.5	-	15.8	10.0	-	-	-	-
Gemini-2.5-Flash	-	10.3	10.3	-	-	-	-
GPT-4o	-	4.3	6.3	-	-	-	-

Table 6: Results on the BankMathBench Advanced dataset. Model accuracy (%) is reported for Zero-shot, SFT, and Tool-Augmented SFT settings across Korean (Kor) and English (Eng).


	Absolute Error $\downarrow$		Error Ratio $\downarrow$		Accuracy $\uparrow$

	Kor	Eng	Kor	Eng	Kor	Eng

Zero-shot	2.65	2.65	100.00	100.00	2.75	0.50
SFT	2.05	2.12	1.28	1.89	1.75	1.75

Table 7: Comparison of absolute error, error ratio, and accuracy across Zero-shot, SFT, and Tool-Augmented SFT settings on the BankMathBench Advanced dataset.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

(Kor)

[Role]

당신은 현실에 기반한 상상력을 지닌 작가입니다. 대한민국 K은행을 방문하는 고객들의 실제 상담 장면을 바탕으로 소설을 쓰고 있습니다. 은행 상품의 기초 계산에 관심 있는 고객들의 페르소나와 질문을 생성하세요. 과장된 설정은 지양하고, 실제 상담 현장에서 있을 법한 자연스럽고 현실적인 상황을 기반으로 작성하세요.

[요구사항]

- 고객의 상황이나 목표(은퇴 준비, 결혼 준비, 주택 구입, 자녀 교육자금 마련 등)를 함께 언급하세요.

Appendix

Table 8: Prompt for generating basic-level questions in BankMathBench.

(Kor)

[역할] 당신은 금융 전문가로서, 사용자가 제공한 조건을 바탕으로 예금, 적금, 대출 상품 중 2가지를 비교하는 금융 계산 질문을 생성하는 역할을 합니다. 아래 조건을 참고하여 2개의 금융 상품을 비교하는 질문을 한 개만 생성하세요.

[요구사항]

- 질문은 상품별 금액, 이자율, 납입 방식, 기간 등을 모두 포함해야 하며, 최종적으로 “어느 쪽이 이자가 더 높은가요?”, “더 유리한 상품은 총 수령액이 얼마나 큰가요?” 등 비교 목적이 명확히 드러나야 합니다.

- 비교 상품 유형: 예금, 적금, 대출 중 2가지 (예: 예금 vs 예금, 예금 vs 적금, 적금 vs 적금, 대출 vs 대출 등)

Appendix

Table 9: Prompt for generating intermediate-level questions in BankMathBench.

(Kor)

[역할] 다음 금융상품 조건을 바탕으로 금융 계산 문제를 한 개만 생성하시오. 일상생활에서 실제로 발생할 수 있는 현실적인 상황을 묘사하시오.

[조건]

- 상품 유형: (예금 | 적금 | 대출)

- 납입 방식: (일시납 | 매월 | 원리금균등 | 만기일시상환 등)

Appendix

Table 10: Prompt for generating advanced-level questions in BankMathBench.

(Kor)

[역할]

당신은 금융 계산을 정확히 수행하는 계산기입니다.

주어진 문제에 대해 다음 두 가지만 출력하세요:

[요구사항]

Appendix

Table 11: Prompt for solution generation.

(Kor)

[역할] 당신은 주어진 질문, 수식/코드, 정답을 바탕으로, 한국어 줄글 형식의 리즈닝 과정을 작성하고 최종 정답을 출력하는 봇입니다.

[요구사항]

1. 리즈닝 과정은 <think> 와 </think> 태그 안에 작성합니다.

2. 리즈닝은 문제 풀이 과정을 서술형 한국어로 작성하며, 필요한 경우 간단한 수식을 포함합니다.

Appendix

Table 12: Prompt for generating reasoning and final answers.


Type 1: Formula Error
(Kor) Question: 2년 만기 정기예금에 연 3.5% 복리로 500만원을 예금하는 것과, 매달 20만원씩 2년간 자유적금에 연 4.0% 단리로 적금하는 것 중 만기 수령액이 얼마나 차이가 나나요?	(Eng) Question: What is the difference in the maturity amount between depositing 5,000,000 KRW in a 2-year fixed-term deposit at an annual compound interest rate of 3.5% and saving 200,000 KRW per month for 2 years in an installment savings account at an annual simple interest rate of 4.0%?

Zero-shot Output: <think> …다음으로, 자유적금의 만기 수령액을 계산해보겠습니다. 자유적금은 매달 20만원씩 2년간 예금하며, 연 4.0% 단리로 이자를 받습니다. [Error] 만기 수령액 = 월납입액12(1+연이율/100)^(납입연수) = 20000012(1+0.04)^2 <calc> 20000012(1+0.04)^2 </calc> <result> 4993920 </result> [Gold] 총 이자 = 월납입액납입개월수(납입개월수+1)/2(연이율/12/100) = 20000024(24+1)/2(4/12/100) = 200000 총 원금 = 200000 * 24 만기 수령액 = 20000024+200000 = 5000000 <calc> 20000024+20000024(24+1)/2*(4/12/100) </calc> <result> 5000000 </result> …</think> <answer> …</answer>	Zero-shot Output: <think> …Next, we calculate the maturity amount of the installment savings account. The installment savings account involves depositing 200,000 KRW per month for 2 years and earns interest at an annual simple rate of 4.0%. [Error] Maturity Amount = Monthly Payment12(1+Annual Interest Rate/100)^(Number of Years) = 20000012(1+0.04)^2 <calc> 20000012(1+0.04)^2 </calc> <result> 4993920 </result> [Gold] Total Interest = Monthly PaymentNumber of Months(Number of Months+1)/2(Annual Interest Rate/12/100) = 20000024(24+1)/2(4/12/100) = 200000 Total Principal = 200000 * 24 Maturity Amount = 20000024+200000 = 5000000 <calc> 20000024+20000024(24+1)/2*(4/12/100) </calc> <result> 5000000 </result> …</think> <answer> …</answer>

Type 2: Parameter Conversion Error

Appendix

Table 13: Illustrative Examples of Error Types

Numerical Reasoning in Language Models

Numerical reasoning in language models studies how systems represent, calculate with, and explain numbers across mathematical and real-world tasks.

See all 150 papers

Jun 2, 2026

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

Despite the pivotal role of numerical reasoning as the cornerstone of mathematical capabilities in large language models (LLMs) across applications, few benchmarks evaluate LLMs by integrating numerical processing and mathematical reasoning, hindering the interpretability of failures in math tasks. We introduce PyraMathBench, a comprehensive hierarchical benchmark with 32,505 questions derived from 7,404 math word problems, spanning 4 key cognitive aspects, 14 subcategories, and 2 modalities. Experiments reveal that LLMs' performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions. To address this, we propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO), which enhance LLMs' numerical-mathematical synergy via efficient tool calls (fuzzy matching and low-quality call rejection). Comparative experiments show Qwen-2.5 achieves a 5.0 score improvement with SOLVE and IRPO training.

Zetian Ouyang, Linlin Wang, Gerard de Melo +1

East China Normal University · Hasso Plattner Institute, University of Potsdam

Numerical Reasoning in Language Models Mathematical Reasoning Benchmarks RL for Language Model Reasoning Mathematical Reasoning LLM Tool Use LLM Mathematical Reasoning

Ask PDF

Aug 11, 2026

V-FiLLM: Verified Financial LLM Reasoning Benchmark

While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.

Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhmasari +2

ETH Zürich, Zürich · Aisot Technologies Ltd

Numerical Reasoning in Language Models Synthetic Data Generation Financial QA Financial Reasoning Benchmarks

Ask PDF

Jul 22, 2026

Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introduce FinIndices, a large-scale benchmark evaluating data-processing fidelity over uncropped financial statements (up to 32K tokens). Utilizing an automated synthesis pipeline with adversarial traps, FinIndices encompasses Single-Index computation and Table-Index tabulation to test complex domain, temporal, and caliber reasoning. Our evaluation reveals two severe LLM vulnerabilities. First, a "Knowledge Bottleneck": despite memorizing formulas during pre-training, models demonstrate fragile pattern matching. Removing explicit formula hints causes performance to collapse (e.g., Gemini-3.1-Pro drops from 70.70% to 38.22% on table tasks), exposing fatal flaws in temporal de-cumulation and stock-flow caliber mismatch. Second, a "Structural Bottleneck": the intense cognitive load of generating multi-metric, multi-period tables actively drains reasoning capacity. Under structural pressure, LLMs that flawlessly execute isolated derivations regress to shallow heuristics, such as fetching incorrect adjacent columns or substituting deep accounting adjustments with lazy literal arithmetic. Finally, Supervised Fine-Tuning (SFT) yields substantial zero-hint gains (+8.54% Single, +3.82% Table), validating that structured logic can be partially restored via data-centric alignment.

Xinke Tong, Xuanming Zhang, Tianyi Tang +10

Qwen Team, Alibaba Group · Tsinghua University · Alibaba Group

Numerical Reasoning in Language Models LLM Evaluation Financial Reasoning in Language Models Finance Large Reasoning Models

Ask PDF

Abstract

Numerical Reasoning in Language Models

Explore similar work

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

V-FiLLM: Verified Financial LLM Reasoning Benchmark

Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements