LLM-jp-4 Technical Report

LLM-jp; :; Chaoran Liu; Daisuke Kawahara; Hideyuki Tachibana; Hirokazu Kiyomaru; Jun Suzuki; Koshi Eguchi

cs.CLOct 14, 2026

LLM-jp-4 Technical Report

Authors: LLM-jp, :, Chaoran Liu, Daisuke Kawahara, Hideyuki Tachibana, Hirokazu Kiyomaru, Jun Suzuki, Koshi Eguchi,

Abstract

This paper presents LLM-jp-4, a family of Japanese sovereign large language models (LLMs) developed with an emphasis on transparency, reproducibility, and practical usability. LLM-jp-4 comprises an 8B dense model and a 32B-A3B mixture-of-experts model, both trained from scratch on approximately 12 trillion tokens. To support independent verification and reproduction, we establish explicit criteria for selecting training data, external models, and software, and make the development process publicly available. Evaluation results demonstrate substantial improvements over previous LLM-jp models and competitive performance against open-weight models of similar scale. By documenting the development principles, training procedures, and empirical findings of LLM-jp-4, we aim to provide a reference for future sovereign AI development.

Figures & tables

Figure 1: Overview of the training pipeline of LLM-jp-4. Training proceeds through three stages: pre-training, mid-training, and post-training. Pre-training uses LLM-jp Corpus v4 to acquire broad knowledge (Phase 1; 6.3T tokens), followed by LLM-jp Corpus v4.1 with increased proportions of math and code (Phase 2; 4.2T tokens). Mid-training mixes LLM-jp Corpus Midtraining v2 with the pre-training corpus to improve question-answering capabilities (Phase 1; 0.8T tokens) and extend the maximum context length from 4K to 64K tokens via 16K (Phase 2; 0.4T tokens). Post-training aligns the models as thinking models through SFT and DPO. The bottom panels show the learning-rate schedule and maximum context length throughout training.

Level	Internal-model training	Public-model training	Retrieval	Redistribution
L1	✓	✓	✓	✓
L2	✓	✓	✓	✗
L3	✓	✓	✗	✗
LX	✓	✗	✗	✗
LZ	✗	✗	✗	✗

Table 1: Release levels used to manage the permitted uses of datasets in LLM-jp-4. Internal-model training denotes training models used only for internal experiments, whereas public-model training denotes training models intended for public release. Retrieval denotes use in a search or retrieval system in which portions of the dataset may be presented to users as retrieval results. Redistribution denotes redistribution of the dataset itself by LLM-jp. A check mark indicates that the corresponding use is permitted.

Property	8B	32B-A3B
Type	Dense	MoE
Params	8.6B	32.1B
Active params	8.6B	3.8B
Layers	32	32
Hidden	4,096	2,560
MoE Hidden	–	960

Table 2: Architecture summary of LLM-jp-4 models. Heads denote Q/KV heads; experts denote total/activated experts.

Setting	Value
Model type	Unigram (SentencePiece)
Vocabulary size	196,608
Character coverage	1.0
Byte fallback	enabled
Normalization	identity (no NFKC)
Whitespace pre-tokenization	enabled (tokens do not cross whitespace)

Table 3: Main training settings for the LLM-jp-4 tokenizer.

Lang.	Corpus	Domain	RL	Tokens	P1	P2
Ja	Aozora Bunko	Book	L1	0.1B	8	8
	Common Crawl	Web	L1	217.4B	4	4
	Ceek.jp News	News	L3	12.5B	8	8
	e-Gov	Government	L1	0.1B	8	8
	Fineweb2 Penedo et al. (2025)	Web	L1	216.1B	4	4
	KAKEN	Academic	L1	0.8B	8	8

Table 4: List of subcorpora in the LLM-jp-4 pre-training corpus. Token counts are based on the LLM-jp-4 tokenizer. The subcorpora marked with

\dagger

were introduced in pre-training Phase 2 and were not used in Phase 1. RL denotes the release level (c.f., § 2.1 ), and P1 and P2 denote the sampling rates in pre-training Phases 1 and 2, respectively. For FineWeb Rescored,

s

denotes the educational-quality score, and the corpus is divided into three subsets according to this score.

Dataset	# Tokens	# Documents	Tokens / Document
MegaMath-Web-Pro-Max	79.4B	69M	$\sim$ 1,140
MegaMath-Web-Pro-Max-OSS	67.4B	83M	$\sim$ 810

Table 5: Comparison of the MegaMath-Web-Pro-Max and MegaMath-Web-Pro-Max-OSS datasets.

	swallow-evaluation			llm-jp-eval
Factor	$\alpha$	$p$	$f$	$\alpha$	$p$	$f$
Replace Stack	-0.0004	0.803	0.20	0.0041	0.193	1.37
Reduce FineWeb	0.0004	0.769	0.24	0.0040	0.199	1.34
Add MegaMath	0.0079	0.025	4.39	0.0030	0.289	1.01
Add Laboro	0.0014	0.398	0.75	0.0004	0.862	0.14
Add FinePDFs	0.0004	0.765	0.24	0.0020	0.452	0.66

Table 6: Estimated effects of corpus modifications.

\alpha

denotes the estimated coefficient. Bold values indicate a positive effect satisfying

p\leq 0.05

and

f\geq 0.4

Hyperparameter	Dense 8B	MoE 32B-A3B
Total/activated parameters	8.59B / 8.59B	32.14B / 3.83B
Transformer layers	32	32
Hidden dimension	4,096	2,560
FFN/expert FFN dimension	14,336	960
Attention heads/KV heads	32 / 8	40 / 4
Routed/activated experts	–	128 / 8

Table 7: Model and optimization hyperparameters. P1 and P2 denote pre-training Phases 1 and 2, respectively.

Panel of Figure 2 — Figure 2: Evaluation scores during Phase 1 pre-training. The short branches labeled decay1m, decay2m, and decay3m show runs continued from intermediate checkpoints with learning-rate decay applied.

Lang.	Candidate corpus	Candidate subsets	License
En	SmolLM-Corpus allal et al. (2025)	cosmopedia-v2	odc-by
	Dolma3 Dolmino Pool Team Olmo et al. (2026)	wiki_to_rcqa	odc-by
	DOLMino Mix 1124 Walsh et al. (2025)	Flan, DolminoSynthMath, TinyGSM-MIND	odc-by
	Llama-Nemotron-Post- Training Dataset	science	CC BY 4.0
	LLM-jp-Corpus- Midtraining-v1	-	Apache-2.0, CC BY 4.0
	Nemotron Instruction Following Chat v1	-	CC BY 4.0

Table 8: Candidate corpora for mid-training. A hyphen ( - ) in the candidate subsets column indicates that all subsets were included as candidates.

Benchmark	A	B	C	D	E	F	G	H
Before SFT
llm-jp-eval	-0.4693	0.8209	-0.6125	0.0433	-0.4428	-0.2284	0.3277	-0.4794
Swallow	0.1289	1.8957	-0.1581	0.3818	1.0079	-0.1863	0.4719	0.2208
MT-Bench (JA)	-0.1322	0.0610	-0.3763	0.6306	0.0407	-0.4170	-0.2034	0.4780
MT-Bench (EN)	0.3845	0.1892	0.0061	0.7507	-0.2136	0.2868	0.2502	-0.1404
After SFT

Table 9: Estimated effects of the eight candidate corpora on downstream performance before and after SFT. Values represent signed Cohen’s

f

, with the sign determined by the estimated main effect.

Lang.	Corpus	Subset	Tokens	Sampling weight
En	SmolLM-Corpus allal et al. (2025)	cosmopedia-v2	21.9B	4
	Dolma3 Dolmino Pool Team Olmo et al. (2026)	wiki_to_rcqa	4.5B	4
	DOLMino Mix 1124 Walsh et al. (2025)	Flan DolminoSynthMath TinyGSM-MIND	18.7B 0.04B 7.5B	8 8 4
	Llama-Nemotron-Post- Training Dataset	science	0.6B	4
	LLM-jp-Corpus- Midtraining-v1	coding general math reasoning	21.7B 0.945B 14.8B 15.0B	8
	GSM8K † Cobbe et al. (2021)	-	0.003B	8

Table 10: Corpora included in the final mid-training corpus and their sampling weights. Token counts denote the size of each source before sampling. When multiple subsets are listed in one row, the reported sampling weight applies to each subset unless otherwise indicated. A dagger (

\dagger

) indicates a corpus that is also included in the Phase 2 pre-training corpus.

Hyperparameter	Stage 1	Stage 2	Stage 3
Sequence length	4,096	16,384	65,536
Training steps	100,000	25,000	25,000
Global batch size	2,048	512	128
Tokens per update	8.39M	8.39M	8.39M
Training tokens	0.84T	0.21T	0.21T
Dense learning rate	$1.5\times 10^{-4}$	$1.5\times 10^{-4}$	$1.5\times 10^{-4}\rightarrow 0$

Table 11: Main mid-training schedule. Token counts are rounded to two decimal places.

Panel of Figure 4 — Figure 4: Evolution of the average score (AVG) and selected subtask scores from llm-jp-eval and the Swallow evaluation across pre-training Phases 1 and 2 and mid-training.

Lang.	Sources	License
En	Daring-Anteater	CC BY 4.0
	FLAN	ODC-BY
	Nemotron-Post-Training-Dataset-v2 (STEM)	CC BY 4.0
	Nemotron-IF-Chat-v1 (Chat)	ODC-BY
	Nemotron-IF-Chat-v1 (IF)	CC BY 4.0, ODC-BY
	Nemotron-Math-v2	CC BY-SA 4.0

Table 12: Prompt sources used for SFT.

Subset	Input	Synthetic targets				Filtered samples
	prompts	$R_{1}$	$A_{1}$	$R_{2}$	$A_{2}$	Filtered samples
Daring-Anteater	99,532	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	0 76,207/ 0 85,033/ 0 77,990
FLAN	105,120	$\checkmark$	$\checkmark$	–	–	0 74,612/ 0 83,170/ 0 95,585
Nemotron-Post-Training-Dataset-v2 (STEM)	100,000	$\checkmark$	$\checkmark$	–	–	0 63,701/ 0 81,059/ 0 89,741
Nemotron-IF-Chat-v1 (Chat)	30,000	$\checkmark$	$\checkmark$	–	–	0 22,257/ 0 24,891/ 0 25,552
Nemotron-IF-Chat-v1 (IF)	31,670	$\checkmark$	$\checkmark$	–	–	0 21,877/ 0 25,845/ 0 26,908

Table 13: Input prompts, synthetic targets, and number of SFT samples after filtering.

R_{1}

and

A_{1}

denote the generated reasoning process and final answer for the first assistant turn, respectively.

R_{2}

and

A_{2}

denote those for the additional second assistant turn. Filtered samples are reported for low/medium/high reasoning effort.

Source	Tokenized chunks	Training chunks	Upsampling	Training tokens
Daring-Anteater (low)	5,575	5,000	1	327,680,000
Daring-Anteater (medium)	8,268	5,000	1	327,680,000
Daring-Anteater (high)	9,919	5,000	1	327,680,000
FLAN (low)	542	500	1	32,768,000
FLAN (medium)	869	500	1	32,768,000
FLAN (high)	2,255	500	1	32,768,000

Table 14: Number of tokenized chunks, training chunks, upsampling factors, and training tokens for each SFT source. The number of training tokens is computed as the number of training chunks multiplied by 64K tokens.

Source	8B	32B-A3B
Daring-Anteater (low)	4,352	3,557
Daring-Anteater (medium)	3,827	3,092
Daring-Anteater (high)	3,833	3,297
FLAN (low)	4,729	4,254
FLAN (medium)	4,101	3,476
FLAN (high)	2,093	1,702

Table 15: Number of training preference pairs for each DPO source. Unlike the SFT counts in Table 14 , each DPO sample corresponds to one tuple of a question, a chosen answer, and a rejected answer.

Figure 5: Scores by task category on the Japanese subset of llm-jp-eval v2.1.3.

Figure 6: Scores by task category on the English subset of llm-jp-eval v2.1.3.

Figure 7: Scores on challenging benchmarks requiring knowledge and reasoning.

Model	MT-Bench JA	MT-Bench EN	AnswerCarefully	llm-jp-inst.
gpt-4o-2024-08-06	7.29	7.69	4.00	4.07
gpt-5.4-2026-03-05 (low)	8.87	8.76	4.38	4.79
gpt-5.4-2026-03-05 (medium)	8.87	8.89	4.43	4.82
gpt-5.4-2026-03-05 (high)	8.98	8.85	4.41	4.83
gpt-oss-20b (low)	7.21	7.95	3.39	3.08
gpt-oss-20b (medium)	7.33	7.85	3.55	3.16

Table 16: LLM-as-a-Judge conversational evaluation results. Scores are averaged over three inference and evaluation runs. MT-Bench scores range from 1 to 10, while AnswerCarefully and llm-jp-instructions scores range from 1 to 5, with higher scores indicating better performance. Low, medium, and high indicate the reasoning effort setting used during inference.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 522 papers

Cost of Reasoning in non-English Languages: A Case Study on Japanese
Jul 11, 2026Yuu JinnaiCross-Lingual Reasoning in Language Models Multilingual Language Model Evaluation
CyberAgent / Tokyo, Japan
Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs
Aug 14, 2026Florian BraunPragmatic Reasoning in Language Models Cross-Lingual Reasoning in Language Models
Independent Researcher
Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization
Mar 12, 2026Takato YasunoLanguage Modeling Small Language Models

Jul 11, 2026J/K move · Enter open · S save

Cost of Reasoning in non-English Languages: A Case Study on Japanese

Yuu Jinnai

CyberAgent / Tokyo, Japan

Cross-Lingual Reasoning in Language Models Multilingual Language Model Evaluation RL for Language Model Reasoning Large Reasoning Models Multilingual Language Models Structured Reasoning in Language Models+1

Open paper AskPDF arXiv

Reasoning Language Models (RLMs) achieve their strongest performance when they reason in English, the language for which reasoning-oriented training data is most abundant. However, reasoning trace is a clue for model interpretability and safety, and useful in practice for both the model users and for model developers. Thus, it is desirable to be able to develop a model that reasons in a language of the user's choice, while still maintaining strong reasoning performance. To this end, we study the feasibility of training a model that reasons in Japanese. We develop a Japanese-reasoning variant of Qwen-3-Swallow-8B, which is a Japanese LLM continually pretrained from Qwen-3-8B, with GRPO and evaluate it across coding, math, and science benchmarks. The study shows that reasoning-language control is feasible by training a Japanese continually pretrained model with GRPO. However, its performance is at best on par with strong English-reasoning baselines on several benchmarks. We also evaluate the trained model on Japanese cultural benchmarks and observe that the model's performance is worse than the baseline models, suggesting that the reasoning in Japanese does not immediately improve performance on culturally relevant tasks for free.

Abstract

Language Modeling

Explore similar work

Cost of Reasoning in non-English Languages: A Case Study on Japanese

Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs

Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization