JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams
Organizations: Sakana AI · Institute of Science Tokyo
Abstract
Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.
Figures & tables
| Benchmark | Exams | Question types | Latest ed. | ||
| MC | Free | VQA | |||
| Masuda et al. (2023) | 1 | ✓ | ✗ | ✗ | ✗ |
| Hirano (2024) | 4 | ✓ | ✗ | ✗ | ✗ |
| JFinBench (ours) | 6 | ✓ | ✓ | ✓ | ✓ |
| Exam | Exam date | Sections | Question type | Examples | w/ image | |||
| MCQ | Short | Form | Essay | |||||
| CPA short-answer exam | May 24, 2026 | 4 | 93 | – | – | – | 93 | – |
| Certified Public Tax Accountant | Aug 4–6, 2026 | 11 | – | – | 17 | 23 | 40 | 1 |
| FP Level 1, written | May 24, 2026 | 2 | 50 | 15 | – | – | 65 | 6 |
| Financial counter services Level 1 | May 2026 ∗ | 2 | 50 | 16 | – | – | 66 | 5 |
| Nissho Bookkeeping Level 1 | Jun 14, 2026 | 4 | – | 23 | 1 | – | 24 | – |
| Score by exam (%) | Overall | ||||||||
| Model | CPA | Tax | FP | Counter | Nissho | Zensho | Score (%) | Cost ($/Q) | Tok. (K/Q) |
| Proprietary models | |||||||||
| Claude Fable 5 (max) | 97.1 | 79.4 | 100.0 | 100.0 | 100.0 | 98.0 | 95.8 0.4 | 0.555 | 10.9 |
| GPT-6 Astra (max) | 96.8 | 73.7 | 99.5 | 99.0 | 96.9 | 96.0 | 93.6 0.6 | 0.248 | 4.8 |
| Gemini 3.8 Flash (high) | 92.5 | 60.7 | 96.9 | 98.5 | 98.2 | 85.9 | 88.8 1.3 | 0.025 | 6.6 |
| Open-weight models | |||||||||
| Model | w/o tools | w/ tools (harness) | |
| Claude Fable 5 (max) | 95.8 | 96.4 (Terminus-2) | 0.6 |
| GPT-6 Astra (max) | 93.6 | 97.3 (Terminus-2) | 3.7 |
| Gemini 3.8 Flash (high) | 88.8 | 94.2 (Terminus-2) | 5.4 |
| Kimi K3.1 Pro (max) | 83.4 | 92.9 (Kimi Code) | 9.5 |
| Kimi K3 (max) | 79.9 | 91.0 (Kimi Code) | 11.1 |
| DS V4.1 Flash 1017 (max) | 73.1 | 91.5 (mini-SWE-agent) | 18.4 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Exam | Questions | Answer keys | Distribution | ||
| Source | License | Source | License | ||
| CPA short-answer exam (公認会計士試験 短答式) | CPA and Auditing Oversight Board (FSA), question PDFs | Public Data License 1.0 (CC BY 4.0 compatible) | CPA and Auditing Oversight Board (FSA), official answer PDF | Public Data License 1.0 | Hugging Face |
| Certified Public Tax Accountant exam (税理士試験) | National Tax Agency, question and answer-form PDFs | Public Data License 1.0 (CC BY 4.0 compatible) | TAC Co., Ltd., answer bulletins | All rights reserved | Hugging Face (questions), recipe (answer keys) |
| FP skill test (FP技能検定1級 学科) | Kinzai, question PDFs | Kinzai’s exam-question usage terms (attribution and modification notice, no application required) | Kinzai, official answer PDF | Kinzai’s usage terms cover the questions; coverage of the model answers is being confirmed | Hugging Face (questions; the basic section’s answers are option numbers), recipe (model answers of the applied section) |
| Financial counter services skill test (金融窓口サービス技能検定1級) | Kinzai, question PDFs | Kinzai’s exam-question usage terms (as above) | Kinzai, question PDFs (the model answers follow each question) | Kinzai’s usage terms cover the questions; the hosted answers are option symbols and figures | Hugging Face |
| Nissho Bookkeeping (日商簿記検定1級) | Japan Chamber of Commerce and Industry, question PDFs | All rights reserved | TAC Co., Ltd., answer bulletins | All rights reserved | Recipe |
| Model | Identical | Overall (run 1) | ||
| GPT-5.1 | Opus 5.5 | |||
| Claude Fable 5 | 0.92 | 103/128 | 95.3 | 95.1 |
| GPT-6 Astra | 0.72 | 94/128 | 93.8 | 94.6 |
| Gemini 3.8 Flash | 0.89 | 95/126 | 88.1 | 88.2 |
| Kimi K3.1 Pro | 0.79 | 89/125 | 83.1 | 82.5 |
| Kimi K3 | 0.94 | 96/128 | 77.4 | 77.2 |
| Score by exam (%) | Overall | ||||||||
| Model | CPA | Tax | FP | Counter | Nissho | Zensho | Score (%) | Cost ($/Q) | Tok. (K/Q) |
| Proprietary models | |||||||||
| Claude Fable 5 (max) | 97.1 0.6 | 79.4 2.4 | 100.0 0.0 | 100.0 0.0 | 100.0 0.1 | 98.0 1.7 | 95.8 0.4 | 0.555 0.010 | 10.9 0.2 |
| Claude Fable 5 (low) | 95.3 1.6 | 71.4 3.0 | 97.9 2.4 | 98.0 0.9 | 96.9 2.4 | 96.0 1.7 | 92.6 1.4 | 0.060 0.001 | 0.9 0.0 |
| GPT-6 Astra (max) | 96.8 0.0 | 73.7 1.9 | 99.5 0.9 | 99.0 0.9 | 96.9 2.5 | 96.0 1.7 | 93.6 0.6 | 0.248 0.003 | 4.8 0.1 |
| GPT-6 Astra (low) | 96.8 0.0 | 72.0 1.0 | 96.9 0.0 | 100.0 0.0 | 92.8 2.4 | 93.9 3.0 | 92.1 0.1 | 0.045 0.000 | 0.7 0.0 |
| Model | Attempts scored 0 |
| Claude Fable 5 | 0 |
| GPT-6 Astra | 0 |
| Gemini 3.8 Flash | 0 |
| Kimi K3.1 Pro | 0 |
| Kimi K3 | 0 |
| DS V4.1 Flash 1017 | 4 |
| Model | Terminus-2 | mini-SWE-agent | Official harness |
| Harbor 0.24.0 | 2.4.6 | ||
| Claude Fable 5 | 96.4 0.0 | 96.3 0.3 | 91.9 0.5 (Claude Code 2.1.306) |
| GPT-6 Astra | 97.3 0.3 | 96.8 0.4 | 97.0 0.6 (Codex 0.163.0) |
| Gemini 3.8 Flash | 94.2 0.9 | 94.0 0.7 | 92.5 0.6 (Gemini CLI 0.65.0) |
| Kimi K3.1 Pro | 92.1 1.1 | 84.9 1.9 | 92.9 1.2 (Kimi Code 2.3.0) |
| Kimi K3 | 86.1 1.9 | 90.1 0.7 | 91.0 1.1 (Kimi Code 2.3.0) |