Staged Continual Adaptation of Multimodal Foundation Models for Japanese Financial Documents

Genshin Kakimoto; Atsushi Yanagisawa

cs.AIOct 20, 2026

Staged Continual Adaptation of Multimodal Foundation Models for Japanese Financial Documents

Authors: Genshin Kakimoto, Atsushi Yanagisawa

Organizations: Kyoto University, Kyoto, Japan.

Abstract

Staged post-training of multimodal foundation models chains vision-language alignment, reasoning distillation, and domain tuning, yet how each phase trades capabilities against the others remains under-characterised. Tracking an 8.4B-parameter model from an un-fine-tuned baseline through three training phases on Japanese financial documents (evaluated on five text-input benchmarks), we find the phases interact non-additively: math-only distillation overwrites the financial knowledge acquired earlier even as procedural math skill survives, and the domain phase repairs more than it installs. A skip-phase ablation is best for domain knowledge but forfeits numerical-reasoning gains, so per-phase contributions are not decomposable and the best checkpoint is task-dependent, with implications for checkpoint retention and scheduling rather than added stages.

Figures & tables

Phase	Updated	Primary Objective
P1-S1	Projector	Alignment of image and language embeddings (vision and LLM frozen)
P1-S2	Vision + Projector + LLM (LoRA)	Visual instruction following in Japanese
P2-S1	LLM (QLoRA)	Chain-of-thought distillation for math reasoning
P3-S1	LLM (LoRA)	Text-only financial-domain adaptation
P3-S2	Projector + LLM (LoRA)	Visual domain QA on government financial documents

Table 1: Training phases and modules updated at each stage (P = Phase; S = Stage). “Vision” = SigLIP-v2 encoder; “Projector” = 2-layer MLP; “LLM” = llm-jp-4-8B-Instruct. “LoRA” ( Hu et al., 2022 ) denotes LLM adaptation via low-rank adapters; “QLoRA” ( Dettmers et al., 2023 ) denotes LoRA on a 4-bit quantized base model, used at P2-S1.

Benchmark	P0	P1	P2	P3	P1 $\to$ P3
GSM8K	58.5	64.3	73.8	71.6	59.6
EDINET earnings †	0.524	0.457	0.500	0.488	0.491
EDINET fraud †	0.516	0.554	0.534	0.614	0.499
EDINET industry	4.7	8.7	7.9	8.2	10.4
JP Fin Harness	45.0	53.5	31.6	49.6	56.2

Table 2: Evaluation against the un-fine-tuned baseline (P0), after each training phase (P1–P3), and for the skip-P2 ablation (P1

\to

P3). † ROC-AUC; other metrics are Accuracy. Bold marks the best score per row. EDINET earnings is near-chance for all our checkpoints (ROC-AUC

\approx

0.46–0.52), so the bold entry in that row is the numerical maximum, not a meaningful capability difference. Scores are single greedy runs; see Table 9 for multi-seed means.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Architecture of the proposed model. The input image is connected to the language model through SigLIP-v2 and the MLP projector, and is processed jointly with the input text. By reading document pages directly as images, the model generates output text while preserving layout information such as tables, body text, and numerical arrangements.

Figure 2: Overview of the three-phase staged continual adaptation pipeline. Phase 1 (Vision-Language Alignment) : P1-S1 trains only the projector on STAIR Captions and government-PDF OCR (vision encoder and LLM frozen); P1-S2 fine-tunes all three components (vision, projector, LLM via LoRA) on Visual QA and JA-VG-VQA. Phase 2 (Mathematical Reasoning Distillation) : P2-S1 updates the LLM via QLoRA on approximately 150K math problems with XML-structured chain-of-thought supervision from Qwen3-30B-A3B-Thinking-2507 (vision encoder and projector frozen). Phase 3 (Financial-Domain Adaptation) : P3-S1 adapts the LLM to text-only financial benchmarks (FinQA, ConvFinQA, TAT-QA); P3-S2 then trains the projector together with LoRA adapters on the LLM using visual financial QA from government PDFs (vision encoder frozen throughout P3). Flame icons indicate trainable modules; snowflake icons indicate frozen modules.

Phase	Type	Content	Examples	Type total
P1-S1	OCR / Caption	STAIR Captions ( Yoshikawa et al., 2017 )	820,310	1,084,952
		OCR data from government PDFs (synthesised)	264,642	1,084,952
P1-S2	VQA	Extractive + reasoning QA from government PDFs (synthesised)	264,642	365,137
		JA-VG-VQA conversational ( LLM-jp, 2024 )	99,995
		JA-VG-VQA-500 ( SakanaAI, 2024 )	500
P2-S1	Reasoning SFT	sbintuitions/MGSM_ja ( SB Intuitions, 2024 )	250	$\sim$ 150,000

Appendix

Table 3: Datasets used for training at each phase, with per-dataset sample counts and per-type subtotals. “Examples” counts captions for STAIR, image–QA pairs for VQA-style data, and problem–solution pairs for math/financial QA. Public-dataset counts are taken from the train splits of the original releases; the math corpus is a random sample of 150K problems drawn across the six listed sources (each entry shows the source’s total available size). The datasets constructed in this work (synthesised from government PDFs) are publicly released on the Hugging Face Hub (Appendix B ).

Phase	LR	Batch	LoRA $(r,\alpha)$	Epochs
P1-S1	1e-3	2	None	2
P1-S2	2e-5	2	64, 128	1
P1-S2B (LLM)	1e-5	2	64, 128	1
P1-S2B (Vision)	2e-6	2	64, 128	1
P2-S1 (SFT, QLoRA)	2e-4	2	32, 64	1
P3-S1 / P3-S2	5e-6	2	16, 32	3

Appendix

Table 4: Main training hyperparameters by stage. “P1-S2” is the primary projector-and-vision-update sub-step; “P1-S2B (LLM/Vision)” denotes the subsequent step where the LLM is adapted with LoRA using per-module learning rates. “P2-S1 (SFT)” lists the hyperparameters of the chain-of-thought supervised distillation sub-phase; this stage uses QLoRA (4-bit NF4 quantized base with LoRA adapters), whereas all other stages use full-precision LoRA. P3-S2 shares the P3-S1 configuration unless otherwise noted.

JP Fin task	P0	P1	P2	P3-1	P3-2	P1 $\to$ P3
chabsa	0.8591	0.9428	0.5814	0.9357	0.9353	0.9479
cma_basics	0.3421	0.5526	0.2105	0.3947	0.3947	0.5789
cpa_audit	0.2764	0.2236	0.1683	0.2487	0.2513	0.2312
fp2	0.2989	0.3432	0.1789	0.2863	0.3011	0.3832
security_sales_1	0.4737	0.6140	0.4386	0.5614	0.5965	0.6667
Average	0.4500	0.5352	0.3155	0.4854	0.4958	0.5616

Appendix

Table 5: Breakdown on JP Fin Harness tasks (Accuracy). P0 is the un-fine-tuned baseline; P3-1 / P3-2 are the two sub-stages of Phase 3; P1

\to

P3 is the skip-P2 ablation. Bold indicates the best-performing model for each task.

Benchmark	Llama-3.3-70B	GPT-5	DeepSeek-R1	Ours (P3)
EDINET earnings †	0.41	0.62	0.46	0.488
EDINET fraud †	0.59	0.62	0.56	0.614
EDINET industry	14.0	31.0	16.0	8.2

Appendix

Table 6: EDINET comparison. All scores use the EDINET-Bench text-input setting in which the financial statements (Summary, balance sheet, profit-and-loss, and cash-flow tables) are supplied to the LLM as text rather than page images; the vision modality is not exercised by this benchmark. The Llama-3.3-70B ( Meta, 2024 ) , GPT-5, and DeepSeek-R1 numbers are the Summary+BS+CF+PL rows of EDINET-Bench ( Sugiura et al., 2026 ) Table 6 (means over three runs), i.e. the same input setup as ours, and are listed for context only, since cross-source comparisons are indicative rather than head-to-head. † ROC-AUC; industry is Accuracy. Bold marks the best score per row.

Model	Size	Avg.
Claude 3.5 Sonnet ( Anthropic, 2024 )	proprietary	77.0
Qwen2-72B-Instruct ( Qwen Team, 2024 )	72B	67.7
Qwen-14B-Chat ( Qwen Team, 2023 )	14B	49.1
Meta-Llama-3-8B-Instruct ( Grattafiori et al., 2024 )	8B	44.7
Ours (P0)	8B	45.0
Ours (P1)	8B	53.5

Appendix

Table 7: JP Fin Harness comparison. External scores are from the official 0-shot leaderboard ( Hirano, 2023 ) . Ours (P0) is the un-fine-tuned llm-jp-4-8B-Instruct baseline; we additionally show Ours (P1), Ours (P1

\to

P3), and Ours (P3) because JP Fin Harness peaks at P1

\to

P3. Bold: best score.

Benchmark	Qwen-14B Chat	Qwen3 8B-base	Llama 3 8B	Ours (P0)	Ours (P1)	Ours (P2)	Ours (P3)
GSM8K	59.3	89.84	84.5	58.5	64.3	73.8	71.6

Appendix

Table 8: GSM8K accuracy comparison with external models and our four checkpoints (P0–P3). Qwen-14B Chat ( Qwen Team, 2023 ) was evaluated in our setting; Qwen3 8B-base ( Qwen Team, 2025a ) and Llama 3 8B ( Grattafiori et al., 2024 ) results are taken from their respective technical reports (both base-model values). Ours (P0) is the un-fine-tuned llm-jp-4-8B-Instruct baseline. Our Phase 2/3 checkpoints are evaluated on the GSM8K test split after distillation on a 150K-problem mix randomly sampled from MathInstruct, OpenMathReasoning, OpenR1-Math-220k, orca-math-word-problems-200k, NuminaMath-1.5, and MGSM-ja; some of these aggregate corpora include GSM8K-style problems, so the comparison should be read with this in-distribution fine-tuning in mind. Where GSM8K-derived problems appear in the training mix, they originate solely from the GSM8K train split; the GSM8K test split used here for evaluation is held out and never enters P2 distillation, so the reported gains are not attributable to test-set leakage. Bold indicates the highest score.

Benchmark	P1	P2	P3
GSM8K	$62.6\pm 0.5$	$69.4\pm 1.4$	$68.8\pm 1.3$
EDINET earnings †	$0.475\pm 0.014$	$0.484\pm 0.025$	$0.473\pm 0.025$
EDINET fraud †	$0.562\pm 0.008$	$0.523\pm 0.008$	$0.547\pm 0.005$
EDINET industry	$6.7\pm 0.4$	$6.8\pm 0.3$	$6.0\pm 0.4$
JP Fin chabsa	$0.909\pm 0.001$	$0.587\pm 0.001$	$0.924\pm 0.001$
JP Fin cma_basics	$0.500\pm 0.021$	$0.219\pm 0.012$	$0.360\pm 0.106$

Appendix

Table 9: Multi-seed robustness evaluation (temperature 0.7, top_p 0.95, 3 seeds). Values are mean

\pm

standard deviation. GSM8K and JP Fin Harness subtasks are Accuracy; EDINET metrics are ROC-AUC ( † ) or Accuracy.

Figure 4: Loss curves during three-stage training. Phase 1 achieves vision–language alignment, Phase 2 develops reasoning, and Phase 3 performs financial adaptation in sequence.

Multimodal Foundation Models

Multimodal foundation models learn shared capabilities across text, images, audio, video, and other signals for understanding, generation, and action.

See all 351 papers

Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning
Sep 9, 2026Zhirayr Hayrapetyan, Andrei Kalmykov, Denis Kokosinskii +2Financial Reasoning in Language Models RL for Language Model Reasoning
ASDA: Automated Skill Distillation and Adaptation for Financial Reasoning
Mar 17, 2026Tik Yu Yim, Wenting Tan, Sum Yee Chan +2Domain Adaptation for LLMs Agent Skill Learning
The University of Hong Kong, Hong Kong SAR, China
Toward Structured Financial Foundation Models
Oct 19, 2026Natraj Raman, Saumitra Mishra, Vamsi Potluru +1Finance Financial Services
JPMorgan AI Research

Sep 9, 2026J/K move · Enter open · S save

Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning

Zhirayr Hayrapetyan, Andrei Kalmykov, Denis Kokosinskii, Dmitry Stanishevskii+1

Financial Reasoning in Language Models RL for Language Model Reasoning Language Model Distillation LLM Post-Training Language Model Post-Training

Open paper AskPDF arXiv

Financial text, textbooks, and question-answer pairs are abundant, but only a small fraction is directly usable for reasoning-focused post-training. Existing QA pairs often lack explicit reasoning, sufficient context, or reliably verifiable answers, while textbooks must first be transformed into synthetic training examples. We present a data-centric pipeline that constructs complementary corpora by mining open-source reasoning traces, distilling financial instruction data, and generating knowledge-graph-guided question-answer pairs from financial educational material. After semantic deduplication, three lightweight sequence classifiers select finance-relevant examples, reject under-specified questions, and identify tasks suitable for reinforcement learning with compact rule-based verifiers. For model adaptation, we study supervised fine-tuning and reinforcement learning, while self-distilled fine-tuning and post-training model merging are used to prevent the loss of financial capabilities already present in the starting model. We evaluate the adapted language models using FINESSE-Bench, reporting aggregate performance and changes relative to their starting checkpoints. Across the selected comparisons, ordinary SFT reduces FINESSE-Bench accuracy by 3.2-4.0 percentage points, whereas self-distilled SFT improves over the corresponding starting models by 1.0-2.8 points. Equal-weight merging recovers 3.0 points over its SFT parent and finishes 0.9 points above the original model; GRPO on hard tasks adds 0.4 points after self-distilled SFT or 3.0 points when applied directly to verifiable tasks. These results show that retention-aware adaptation can improve financial reasoning without the regressions observed after ordinary SFT.

Abstract

Multimodal Foundation Models

Explore similar work

Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning

ASDA: Automated Skill Distillation and Adaptation for Financial Reasoning

Toward Structured Financial Foundation Models