Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization

Takato Yasuno

cs.LGMar 12, 2026

Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization

Authors: Takato Yasuno

Abstract

This paper presents a systematic methodology for building domain-specific Japanese small language models using QLoRA fine-tuning. We address three core questions: optimal training scale, base-model selection, and architecture-aware quantization. Stage 1 (Training scale): Scale-learning experiments (1k--5k samples) identify n=4,000 as optimal, where test-set NLL reaches minimum (1.127) before overfitting at 5k samples. Stage 2 (Compare finetuned SLMs): Comparing four Japanese LLMs shows that Llama-3 models with Japanese continual pre-training (Swallow-8B, ELYZA-JP-8B) outperform multilingual models (Qwen2.5-7B). Stage 3 (Quantization): Llama-3 architectures improve under Q4_K_M quantization, while GQA architectures degrade severely (Qwen2.5: -0.280 points). Production recommendation: Swallow-8B Q4_K_M achieves 2.830/3 score, 8.9 s/question, 4.9 GB size. The methodology generalizes to low-resource technical domains and provides actionable guidance for compact Japanese specialist LMs on consumer hardware.

Figures & tables

Panel of Figure 1 — Figure 1: Three-stage progressive optimization methodology. Stage 1 identifies optimal training scale (n=4,000) via test-set NLL analysis across five dataset sizes. Stage 2 compares four Japanese LLMs under identical QLoRA training, revealing Japanese continual pre-training advantage (Swallow-8B: 2.830/3). Stage 3 discovers that Llama-3 models improve under Q4_K_M quantization (+0.01 to +0.03 points). Dashed arrows indicate data flow between stages.

Figure 2: Relation-type distribution across five training scales. HAS_CHAPTER proportion stabilizes at 17.1% across all scales after undersampling, confirming uniform distribution scaling without introducing bias.

Figure 3: Answer length stability across five training scales. Average answer length remains consistent (370–380 characters) across all subset sizes, indicating that scale variation does not affect response complexity.

Figure 4: Quality metrics summary across five training scales. All quality metrics show stable distributions with coefficient of variation $<$ 3%, confirming that scale variation does not introduce confounding factors in dataset composition.

Figure 5: Training loss curves for five dataset scales (n=1,000 to 5,000). All models exhibit a characteristic three-phase pattern: rapid initial descent (epoch 0–0.4), plateau (1.0–2.0), and late-stage collapse (2.2–3.0). Larger datasets maintain consistently lower loss throughout training, with final loss decreasing from 0.869 (n=1,000) to 0.767 (n=5,000).

n	Final Loss	Avg Loss	Time (min)
1,000	0.8687	1.097	37.4
2,000	0.8413	1.038	73.5
3,000	0.7894	0.996	111.1
4,000	0.7800	0.980	148.2
5,000	0.7666	0.962	185.2

Table 1: Final training loss and wall-clock time by dataset scale.

Figure 6: Test-set generalization metrics across training scales. Top: Training loss decreases monotonically (left axis), but test-set NLL exhibits a U-curve with minimum at n=4,000 (right axis). Bottom: Test perplexity (PPL) mirrors NLL, confirming n=4,000 as optimal. The n=5,000 model overfits despite achieving lowest training loss.

n	NLL	PPL	$\Delta$ NLL	Status
1,000	1.3421	3.83	—	Underfit
2,000	1.2758	3.58	-0.066	Improving
3,000	1.2150	3.37	-0.061	Improving
4,000	1.1272	3.09	-0.088	Optimal
5,000	1.3190	3.74	+0.192	Overfitting

Table 2: Test-set generalization: NLL, perplexity, and overfitting diagnosis.

Figure 7: Multi-model performance comparison (F16 precision). Swallow-8B achieves the highest average score (2.820/3) and perfect-score rate (84%), followed by ELYZA-JP-8B (2.700, 73%). Qwen2.5-7B underperforms (2.420, 49%) despite longest output length (677 chars), suggesting that multilingual pre-training dilutes Japanese technical vocabulary coverage.

Model	Avg Score	Perfect-Score	Score 0	Avg Length	Time
Swallow-8B	2.820	84%	1	310	54.3
ELYZA-JP-8B	2.700	73%	0	331	27.4
Qwen2.5-7B	2.420	49%	1	677	30.4
Tanuki-8B (HF 4bit)	2.620	65%	0	370	16.6

Table 3: Multi-model comparison (F16 precision, 100-question evaluation).

Figure 8: Score distribution across four models (100 questions). Swallow-8B and ELYZA-JP-8B show strong concentration on score-3 (green), with zero score-0 failures for ELYZA. Qwen2.5-7B exhibits the highest proportion of score-2 (yellow, 45%) and score-0 (red, 1 case), indicating frequent under-supported or incorrect answers.

Model	Scr 3	Scr 2	Scr 1	Scr 0
Swallow-8B	84	15	0	1
ELYZA-JP-8B	73	24	3	0
Qwen2.5-7B	49	45	5	1
Tanuki-8B	65	32	3	0

Table 4: Score distribution (F16 precision, 100 questions).

Figure 9: Category-level performance heatmap. Swallow-8B maintains consistently high scores across all categories (darker green). Qwen2.5-7B struggles particularly on Design, Maintenance-Sabo, and Comparative/Cross-Cutting queries, which require cross-referencing multiple standards—a task where Japanese continual pre-training provides clear advantages.

Figure 10: Five-axis radar chart comparison. Swallow-8B excels in quality metrics (score, perfect-score rate, stability) but lags in inference speed. ELYZA-JP-8B offers balanced performance across all dimensions. Qwen2.5-7B’s speed advantage is offset by significantly lower quality metrics.

Figure 11: Per-question score scatter plot (100 questions). Swallow-8B maintains consistently high scores across question IDs (average line at 2.83), with minimal variance. Qwen2.5-7B shows high variance and frequent drops to score-1 or score-0, indicating brittle performance on specific question types.

Model	Format	Avg Score	Perfect-Score	$\Delta$ Score	Time	Speedup	Size (GB)
Swallow-8B	F16	2.820	84%	—	54.3	—	16.0
	Q4_K_M	2.830	86%	+0.010	8.9	6.1 $\times$	4.9
ELYZA-JP-8B	F16	2.700	73%	—	27.4	—	16.0
	Q4_K_M	2.730	78%	+0.030	8.2	3.3 $\times$	4.9
Qwen2.5-7B	F16	2.420	49%	—	30.4	—	16.0
	Q4_K_M	2.140	30%	-0.280	12.4	2.5 $\times$	4.7

Table 5: F16 vs. Q4_K_M quantization: quality and efficiency comparison.

Architecture	Models	Recommended	Rationale
Llama-3 (MHA)	Swallow, ELYZA	Q4_K_M	Quality improves
Qwen2.5 (GQA)	Qwen2.5-7B	F16 or Q8_0	Q4 causes degradation
Llama-3 (llm-jp)	Tanuki-8B	HF 4bit (NF4)	GGUF tokenizer issue

Table 6: Recommended quantization levels by architecture.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation
Aug 25, 2026Ismail Hossain, Nafi Ullah Shafin, Mohammad Abdullah Al MuminLLM Quantization Post-Training Quantization
Institute of Information and Communication Technology (IICT) Shahjalal University of Science and Technology (SUST), Sylhet, Bangladesh
Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval
Feb 18, 2026Subrit DikshitSmall Language Models Legal IR
Department of Computer Science and Engineering, Indian Institute of Information Technology, Pune, India · Director of AI & Data Science, Compunnel Inc
More Human, More Efficient: Aligning Annotations with Quantized SLMs
Apr 1, 2026Jiayu Wang, Junyoung LeeLLM-Assisted Annotation LLM Fine-Tuning
Home Team Science and Technology Agency

Aug 25, 2026J/K move · Enter open · S save

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

Ismail Hossain, Nafi Ullah Shafin, Mohammad Abdullah Al Mumin

Institute of Information and Communication Technology (IICT) Shahjalal University of Science and Technology (SUST), Sylhet, Bangladesh

LLM Quantization Post-Training Quantization LLM Evaluation Bengali NLP Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference, which is why it is now common for on-device deployment. Most of what we know about its effects, however, comes from English benchmarks. It is not clear whether the same holds for morphologically complex, low-resource languages such as Bangla, and this gap is what we address here. We evaluate three model families---Qwen-2.5-7B, LLaMA-3.1-8B, and GPT-OSS-20B---in full precision and in three quantized formats (GPTQ-Int8, GPTQ-Q8, GGUF-W8A16) across five Bangla natural language understanding benchmarks (Bangla MMLU, CommonsenseQA-BN, OpenBookQA-BN, PIQA-BN, and BoolQ-BN), using zero-shot evaluation through lm-evaluation-harness. To our knowledge this is the first controlled comparison of quantization formats on Bangla NLU. The three families do not respond the same way: GPT-OSS loses up to 57.35% accuracy on reasoning-heavy tasks under GGUF-W8A16, while Qwen and LLaMA hold steady under GPTQ, and in a few cases the quantized version edges out the full-precision one. BoolQ-BN, a comprehension task, stays stable across all three families regardless of format. Taken together, these results suggest quantization can work well for Bangla deployment, but the choice of architecture and quantization method matters more than the bit width alone. We discuss what this means for practitioners choosing a model to run on constrained hardware.

Abstract

Language Modeling

Explore similar work

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval

More Human, More Efficient: Aligning Annotations with Quantized SLMs