CantoneseLLM v2: Reasoning in a Low-Resource Language

Tsz Chung Cheng; Chung Shing Cheng; Chaak Ming Lau; Cheuk Hei Chong

cs.CLSep 7, 2026

CantoneseLLM v2: Reasoning in a Low-Resource Language

Authors: Tsz Chung Cheng, Chung Shing Cheng, Chaak Ming Lau, Cheuk Hei Chong

Organizations: Kyushu University · hon9kon9ize · The Education University of Hong Kong · Votee AI ·

Abstract

Cantonese is widely spoken but remains low-resource in written data, with no large corpus of native Cantonese reasoning traces available for model training. We develop and release CantoneseLLM v2, comprising models based on Qwen3 8B and 30B-A3B. The models are trained through CPT on 784 million Cantonese and Hong Kong-related tokens, chat-vector merging, SFT, DPO, and RLVR. Evaluation across the training stages shows that chat-vector merging transfers instruction following but preserves the donor model's reasoning language, while SFT with limited Cantonese reasoning data substantially shortens or removes reasoning traces and reduces benchmark performance. DPO restores the reasoning-block format, particularly for the 8B model, but recovers only part of the lost performance. The RLVR training with Cantonese language and Traditional Chinese scripts as multiplicative constraints introduced Cantonese language alignment and restored the lost performance. The 30B-A3B model reaches 73.16 on HKCanto-Eval, within 1.20 points of its merged checkpoint, while retaining the Cantonese reasoning behaviour absent from that checkpoint. We release the model checkpoints, the training environments, and a thirteen-year Traditional Chinese Common Crawl dataset. The models can be accessed at https://huggingface.co/collections/hon9kon9ize/cantonesellm-v20

Figures & tables

Dimension	Assumed by the established recipe	Available for Cantonese
Corpus scale	35.1B Taiwan-LLM, 200B Swallow, 200B SEA-LION	784M tokens, of which 5.4% is Cantonese Common Crawl
Teacher availability	A teacher model already fluent in the target language	None. No open-weight model produces colloquial Cantonese
Reasoning traces	Abundant native chain-of-thought data	Zero. Traces must be machine translated
Verifiable-constraint data	Available and reusable across languages	Untranslatable, as constraints might be defined against the orthography.
Evaluation	Generation benchmarks	Multiple-choice benchmarks only, and they may miss generation failures

Table 1: Comparison between established continuous pre-training and post-training recipe and the situation for Cantonese highlighted by the comparison of data volume in the first row.

Panel of Figure 1 — Figure 1: Training pipeline in this work. Each stage installs certain knowledge or alignment, but problems were identified after each.

Stage	Installs	Cannot supply	Diagnostic	Cost, 8B / 30B-A3B (device-hours)
Continuous pre-training (CPT)	Local knowledge and Cantonese lexis	Instruction following, and reasoning	Benchmark average against final training loss (Table 4 )	103 / 199 TPU
Chat-vector merging	Instruction following and alignment	The language of the reasoning trace	Reasoning-span script and language (Table 26 , Figure 7 )	0
Supervised fine-tuning (SFT)	Translation, curation and judging behaviour	Target-language reasoning, which it removes	Empty-reasoning rate and reasoning length (Tables 8 , 7 )	47 / 107 GPU
Direct preference optimisation (DPO)	The reasoning block, as a format	Problem-solving capability, and instruction compliance	Empty-block rate and bare-letter answer rate (Section 7.2.1 )	14 / 115 GPU
Verifiable reward (RLVR)	Target-language reasoning and recovered capability	Full recovery at 8B, and Cantonese reasoning on code generation	Per-environment reward and language multiplier (Figure 13 )	441 / 1,256 GPU

Table 2: The breakdown of the five-stage training pipeline showing what each installs, what it cannot supply, the diagnostic and the cost.

Category	Tokens	%
Common Crawl, written Chinese	215,407,694	27.5%
English replay (Nemotron)	163,657,706	20.9%
Web fiction and creative	133,788,878	17.1%
Encyclopaedic	98,147,161	12.5%
News and synth. commentary	65,854,845	8.4%
Common Crawl, Cantonese	42,390,861	5.4%

Table 3: The six largest categories by token count of the continuous pre-training corpus. The full breakdown by document count and source can be found in Table 15 in Appendix A .

Model	Avg.	Final Loss
Qwen3 30B-A3B Base	67.69	1.83
CPT, LR 1.0 $\times$ 10 -5	68.21	1.69
CPT, LR 1.5 $\times$ 10 -5	68.47	1.66
CPT, LR 5.0 $\times$ 10 -5	67.65	1.52

Table 4: The learning rate parameter sweep on the 30B-A3B MoE model with the evaluated benchmark average and the final training loss. The two values moved in opposite directions as the learning rate increased. The best benchmark score was obtained at the second-highest loss in the sweep, while the lowest loss returned a score lower than the selected runs. All eight learning rates and all five benchmark categories are in Table 20 , Appendix C .

Model	Avg.	$\Delta$
8B Chat Vector	69.51	$+$ 3.18%
30B-A3B Chat Vector	74.36	$-$ 2.55%

Table 5: HKCanto-Eval benchmark average for the two merged checkpoints against the officially released weights where the chat vectors were taken. The 8B chat vector merge improved against the donor model, but the 30B-A3B did not, with regression concentrated in the MMLU in English. Per-category figures for all four models are shown in Table 21 in Appendix D .

Class	Rows	Row %	Tokens	Token %	CoT tokens	CoT %
Cantonese SFT	36,745	49.1%	51,907,374	29.3%	16,354,573	31.5%
Distillation-like	22,120	29.5%	57,752,366	32.6%	27,092,090	46.9%
Replay	16,000	21.4%	67,705,664	38.2%	49,154,023	72.6%
Total	74,865	100%	177,365,404	100%	92,600,686	52.2%

Table 6: Composition of the CantoneseLLM v2 SFT mixture by functional class. The final column gives the proportion of each class that is chain-of-thought.

Checkpoint	Total	Mean	Med- ian	CoT to answer
8B official	3,927	491	502	2.94
8B chat vector	3,743	468	512	3.27
8B SFT	211	26	0	0.21
8B DPO	2,652	332	357	1.29
8B GRPO	1,855	232	237	1.09
30B-A3B Official	8,335	1,042	1,023	6.25

Table 7: Reasoning length across the pipeline, in Qwen3 8B tokens over the span between the reasoning tags, across the eight probes. The final column is the ratio of reasoning length to answer length.

Measure	8B Dense	30B-A3B MoE
Generations with no reasoning	64.5%	13.2%
Reasoning length, mean (tokens)	437	1,123
Reasoning length, median (tokens)	0	631
Balanced length ratio	21.1%	53.2%
Judge, complete_cot	2.36	4.33
Judge, cot_cantonese	1.93	3.19

Table 8: The reasoning defect in the two SFT checkpoints, measured over four generations on each of 60,850 prompts. The two judge dimensions are scored from 1 to 5 and are defined in Appendix G , where a generation carrying no reasoning block scores 1 on both.

Generation Dimension	Verbose	Plain
Overall, 30B-A3B	3.527	3.783
Overall, 8B	2.787	3.245
Only Complete CoT, 8B	1.705	3.017

Table 9: Mean judge scores for candidates drawn under the verbose system add-on against the plain system prompt.

	Step 1, arithmetic		Step 2, six environments
	8B	30B-A3B	8B	30B-A3B
Objective	Install output format and CoT language		Broaden the reward across tasks
Task term	Exact numeric verification		Six per-environment graders
Training rows	7473		8,291	7,086
Policy initialisation	DPO Final Step		Step 1 Final Step
Tuning	Full	LoRA $r$ 64	Full, FP8	LoRA $r$ 64

Table 10: Two-stage RLVR training. All four trainings were carried out with NeMo-RL (Megatron backend), and both scored every rollout with the reward of Section 8.1 . The released checkpoints are the final steps of the second run.

Factor	Condition	Value
Language	Dominant script Latin	0.00
	Han, Cantonese or mixed	1.00
	Han, neutral	0.55
	Han, Mandarin	0.10
Script	Fewer than two listed characters	1.00
	Share $\leq$ 0.05	1.00

Table 11: The language and script multiplier, which the product scales the combined score, for a Cantonese target prompt. The language multiplier first rejects Latin-dominant text and encourages the model to reason and answer in Cantonese. The script score is the share of allowed simplified characters over the generation length.

Figure 2: The stage 1 reinforcement run at both sizes, over 150 steps on the multilingual arithmetic blend. (a) Blended reward, with training rollouts in the background and validation means over 64 problems marked. (b) The proportion of groups whose sixteen rollouts did not all receive the same reward (i.e., the proportion that contributes to a gradient). (c) Mean generated tokens per rollout. Faint traces are per-step values, and bold traces are smoothed.

Environment	Source	Rows
Mathematics	Nemotron, adapted	3,917
Code Generation	Nemotron, adapted	2,505
STEM Multiple Choice	Nemotron, adapted	2,208
Workplace Assistant	Nemotron, adapted	796
Instruction Following	Built for this work	2,525
Structured Outputs	Built for this work	1,274

Table 12: The six environments of the second stage run. The Cantonese and traditional Chinese streams of the Instruction Following and Structured Outputs environments’ data were built for this work, using rule-based rather than machine-translated methods. The row count column is the number of samples drawn into the training blend before the difficulty profiling in Section 8.5 .

Environment	en	yue	Gap
Mathematics	0.631	0.384	$-$ 0.247
Workplace assistant	0.451	0.262	$-$ 0.189
Code generation	0.313	0.208	$-$ 0.105
STEM multiple choice	0.491	0.431	$-$ 0.060
Structured outputs	0.781	0.769	$-$ 0.012
Instruction following	0.461	0.458	$-$ 0.003

Table 13: Mean pass rate by prompt language, measured over 13,225 rows and approximately 97,000 rollouts before any reinforcement in the second step, with the same 30B-A3B MoE model from Stage 1. For the four machine-translated environments, the rows are translation pairs, so the prompt language is the only variable. The instruction-following and structured-output rows are generated independently per language rather than translated, so their gaps compare matched distributions rather than matched items.

Panel of Figure 3 — Figure 3: The translated GSM8K probe answered by the two released checkpoints. Every checkpoint in Figure 7 and both official releases used Written Chinese on this probe. Both models here reasoned and answered in Cantonese. Cantonese-specific forms are shown in bold, in particular the classifier zek3, where the official 8B model wrote the simplified-Chinese form (which can only be pronounced as zi2 in Hong Kong Cantonese), and the Cantonese words zing6 faan1 and zing6 dai1 replaced the incorrect sing6 haa6. English translations are by the authors.

Model	MMLU	Canto MMLU	Cultural	Linguistic	Academic & Prof.	Avg.	$\Delta$
Qwen3 8B	81.08	76.68	57.94	39.50	81.61	67.36	-
8B Chat Vector	80.04	76.96	66.67	41.50	82.36	69.51	3.18%
8B SFT	64.23	56.15	47.62	23.00	53.97	48.99	-27.27%
8B DPO	69.19	61.85	59.52	34.00	71.35	59.18	-12.14%
8B GRPO	73.86	69.73	58.33	36.00	76.08	62.80	-6.77%
Qwen3 30B-A3B Thinking 2507	86.65	82.52	68.65	57.00	86.70	76.30	-

Table 14: Benchmark evaluation of all the post-training applied in this work and compared against the official chat model at both sizes.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Documents	Tokens	%
Common Crawl, Written Chinese stream	141,230	215,407,694	27.5%
Encyclopaedic (cross-lingual Wikipedia, HK wiki)	27,980	98,147,161	12.5%
English replay (Nemotron)	177,621	163,657,706	20.9%
Web fiction and creative writing	6,819	133,788,878	17.1%
News, periodicals and synthetic commentary	69,268	65,854,845	8.4%
Common Crawl, Cantonese stream	31,505	42,390,861	5.4%

Appendix

Table 15: Composition of the CantoneseLLM v2 CPT corpus

Figure 4: Cantonese Common Crawl documents per year after the filtering pipeline (language filtering method in Step 1 highlighted by different colour bars) and a global deduplication.

Quantity	Value
Rows	28,478
Articles per cluster (median)	2
Articles per cluster (mean)	3.16
Articles per cluster (max)	10
Distinct persona occupations	1,643
Generated length (median)	528 chars

Appendix

Table 16: News Commentary dataset information

Panel of Figure 5 — Figure 5: The news commentary pipeline, shown through one row of the shipped dataset. (a) An example cluster of ten articles drawn from several outlets over thirteen months, on employers defaulting on Mandatory Provident Fund (MPF) contributions. (b) The conditioning persona and the opening of the generated commentary. English translations are the authors’ own.

Setting	Value
Sequence length	4,096
Multi-document packing	enabled
Per-device batch size	4
Devices	TPU v6e-64
Gradient accumulation steps	4
Optimizer	AdamW

Appendix

Table 17: Shared training configuration of both 8B and 30B-A3B model used in MaxText CPT.

	8B Dense	30B-A3B MoE
Step Time (s)	11.0	21.2
TFLOPS per Chip	292.7	71.5
Model FLOPs util. (%)	32%	8%
Agg. throughput (tokens/s)	382K	198K
Wall-clock (hour)	1.62	3.12
Chip-hours per run	103	199

Appendix

Table 18: Average model training throughput and cost for the two models on TPU v6e-64.

Model	MMLU	Canto MMLU	Cultural	Linguistic	Academic & Prof.	Avg.	$\Delta$
Qwen3 8B Base	76.78	71.04	61.11	30.00	76.44	63.07	-
CPT, 520 steps	76.18	70.76	64.68	36.00	77.73	65.07	3.17%
CPT, 530 steps	75.94	70.92	66.27	35.00	77.49	65.12	3.25%
CPT, 540 steps	76.18	70.93	67.46	32.00	77.66	64.85	2.81%

Appendix

Table 19: HKCanto-Eval Benchmark evaluation on the 8B dense model. Three step counts corresponding to 2.74, 2.79 and 2.85 epochs were used in three independent runs to examine the impact of the final stretch of the training on a small corpus.

Model	MMLU	Canto MMLU	Cultural	Lin- guistic	Academic & Prof.	Avg.	$\Delta$	Final Loss
Qwen3 30B-A3B Base	81.33	75.79	65.87	35.50	79.94	67.69	-	1.83
LR 1.0 $\times$ 10 -5	80.91	75.26	68.65	35.50	80.71	68.21	0.77%	1.69
LR 1.5 $\times$ 10 -5	80.78	75.37	68.25	37.00	80.95	68.47	1.16%	1.66
LR 2.0 $\times$ 10 -5	80.66	75.1	67.86	36.00	80.50	68.02	0.50%	1.63
LR 2.5 $\times$ 10 -5	80.61	75.14	68.65	34.00	80.01	67.68	0.00%	n.a.
LR 3.0 $\times$ 10 -5	80.43	74.98	68.65	33.00	79.67	67.35	-0.50%	1.59

Appendix

Table 20: HKCanto-Eval Benchmark evaluation on the 30B-A3B MoE model. Eight different LRs from 1.0

\times

10 -5 to 5.0

\times

10 -5 were tested to quantify the effect of LR on the CPT of MoE models.

Figure 6: Training telemetry (a) Training loss and (b) Unclipped gradient norm for the selected CPT runs: 8B Dense at 530 Steps and 30B-A3B MoE at 1.5 $\times$ 10 -5 (chosen), 3.0 $\times$ 10 -5 and 5.0 $\times$ 10 -5 learning rate.

Model	MMLU	Canto MMLU	Cultural	Linguistic	Academic & Prof.	Avg.	$\Delta$
Qwen3 8B	81.08	76.68	57.94	39.50	81.61	67.36	-
8B Chat Vector	80.04	76.96	66.67	41.50	82.36	69.51	3.18%
Qwen3 30B-A3B Thinking 2507	86.65	82.52	68.65	57.00	86.70	76.30	-
30B-A3B Chat Vector	80.71	80.26	70.24	55.00	85.59	74.36	-2.55%

Appendix

Table 21: HKCanto-Eval Benchmark evaluation on the Qwen3 official weights (Qwen3-8B and Qwen3-30B-A3B-Thinking-2507) and the chat vector merged model. The 8B hybrid dense model was evaluated with reasoning mode turned on.

Panel of Figure 7 — Figure 7: An example response across the two official chat models and the two chat vector merged models. (a) The system prompt explicitly requests Hong Kong Cantonese. (b) Machine-translated input questions from GSM8K (answer: 18 dollars). (c) Reasoning traces from the four models. The two 8B models used simplified Chinese in the reasoning process. The merged 30B-A3B model started its chain-of-thought in Cantonese but later switched to written Chinese. (d) Final response after the chain-of-thoughts. English translations are the authors’ own.

Task group	Rows	%
Language form and translation	20,274	55.2%
Corpus curation	6,575	17.9%
Reasoning and examination	5,590	15.2%
Hong Kong grounded content	1,878	5.1%
Long-form narrative	1,402	3.8%
Multi-turn dialogue	764	2.1%

Appendix

Table 22: Composition of the Cantonese SFT class by task group, counted as the rows enter the mixture and therefore after the repeats.

Panel of Figure 8 — Figure 8: Two examples from the expert-authored linguistic subset SFT data. (a) A fully expert-curated question, reasoning traces and the answer. (b) An expert-seed expansion example where the annotator drafts a question and a one-line response for Gemini 3.1 Pro Preview to expand. English translations are the authors’ own.

	HuggingFace Source	Rows	Tokens (M)	CoT Tokens (M)	CoT Share
Kimi 2.5	ianncity/KIMI-K2.5-1000000x	11,594	35.6	14.5	41.0%
Nemotron Math & Science	nvidia/Nemotron-Math-v2 nvidia/Nemotron-Science-v1	3,902	11.3	7.8	69.5%
Qwen3	-	6,624	10.8	4.7	43.4%
Total	-	22,120	57.8	27.0	46.7%

Appendix

Table 23: The distillation-like sources. The two Nemotron releases are sampled and translated as one source and are counted together. Token totals are rounded to 0.1M, while reasoning counts are exact. Every source is rendered into the same 25% English, 25% Hong Kong written Chinese and 50% Cantonese blend described below, so the Cantonese half of this class amounts to 28,876,183 tokens.

Portion	Uni- que	Re- peat	Pas- ses
Cantonese, duplicated	17,183	2	6
Cantonese, single	2,379	1	3
Distillation-like	22,120	1	3
Replay	16,000	1	3
Total	57,682	1.30

Appendix

Table 24: Unique row counts by portion, with the effective number of passes at three epochs.

Setting	8B dense	30B-A3B MoE
Framework	LLaMA-Factory ( Zheng et al., 2024 )	NeMo-RL (Megatron backend, Ray cluster)
Hardware	8 $\times$ H100 HBM3 80 GB, 1 node	16 $\times$ H100 HBM2e 94 GB, 4 nodes
Parallelism	8-way data parallel	EP 16, DP 1, TP 1, PP 1, CP 1
Dataset variant	Hybrid (154,637,972 Tokens)	Full (177,365,404 tokens)
Sequence length	8,192, neat packing	8,192, modified first-fit decreasing packing
Global batch	64 sequences (2 per device, 4 accumulation)	128 sequences (1 per device, 128 accumulation)

Appendix

Table 25: Training configuration for the two models. Both were fine-tuned in full from their respective chat vector checkpoints of Section 5 , with a seed of 42 and a gradient clipping threshold of 1.0. The dataset variants differ, and that difference is the subject of Section F.4 .

Figure 9: Training telemetry for the two SFT runs plotted against epoch. (a) Training loss, with validation loss overlaid at the evaluation points. (b) Unclipped gradient norm on a logarithmic scale, with the clipping threshold of 1.0 marked. The shaded region is the warmup phase, and the dotted vertical line marks epoch 1.9, beyond which validation loss is flat in both runs.

Panel of Figure 10 — Figure 10: The two SFT checkpoints on an arithmetic probe and a translation probe, under the system prompt of Figure 7 (a). (a) The 8B model closes an empty reasoning block and then reasons in Cantonese inside the answer, marked by the step connectives and the Cantonese function words shown in bold, while the 30B-A3B model reasons inside the block and answers concisely. Both returned the correct answer: 18 dollars. (b) On the translation probe, the 8B model again emits an empty block and returns only the translation, while the 30B-A3B model reasons within the block, states in that trace that no explanation is required, and then reproduces the Written Chinese source verbatim before its Cantonese translation. English translations are the authors’ own.

Reasoning span	SC per 1k Han	Probe with SC
8B Chat Vector	191.1	7 / 8
8B SFT	0.0	0 / 8
30B-A3B Chat Vector	0.0	0 / 8
30B-A3B SFT	0.0	0 / 8

Appendix

Table 26: Simplified-character density in the reasoning span across the eight probes.

Component	Source	Range	Weight
Overall	Judge	1–5	3.0
Complete response	Judge	1–5	3.0
Complete CoT	Judge	1–5	2.0
Reasoning language	Classifier	0/0.3/0.5/1	2.0
Response length	Generation	log chars	1.5
Reasoning length	Generation	log chars	0.8

Appendix

Table 27: Components of the composite score used for pair selection. The judge dimensions are normalised to the unit interval before weighting. The reasoning language term takes one of four band values, 0 for Mandarin, 0.3 for language-neutral, 0.5 for mixed and 1 for Cantonese, assigned by the cantofilter classifier over the reasoning span.

	Pairs	Side	CoT Mean	CoT Median	Resp. Mean
8B	12,204	chosen	1,114	632	885
		rejected	122	0	606
30B	14,679	chosen	1,124	733	1,861
		rejected	766	268	880

Appendix

Table 28: Preference-pair statistics after selection, including pair and token counts.

Setting	8B dense	30B-A3B MoE
Base checkpoint	SFT checkpoint at 801 steps	SFT checkpoint at 1,575 steps
Hardware	4 $\times$ H100 HBM2e 94 GB GPUs, 1 node	32 $\times$ H100 HBM2e 94 GB GPUs, 8 nodes
Parallelism	TP 4, DP 1, PP 1	EP 32, DP 32, TP 1, PP 1
Preference pairs	12,204 (11,594 train, 610 validation)	14,679 (13,946 train, 733 validation)
Global batch	16 pairs (1 per device, 16 accumulation)	32 pairs (1 per device, 1 accumulation)
Steps and warmup	724 steps, 72 warmup	435 steps, 44 warmup

Appendix

Table 29: Preference training configuration for the two models, both trained in full from their respective supervised checkpoints of Section F.3 with NeMo-RL on the Megatron backend in BF16. Shared settings are a reference policy KL penalty

\beta

of 0.1, the preference loss alone with the supervised term weighted at zero, a sequence length of 8,192, weight decay 0.1, gradient clipping at 1.0, a peak learning rate of 1.0

\times

10 -6 decaying by cosine to 1.0

\times

10 -8 after a 10% warmup, and one epoch.

Figure 11: Training telemetry for the two preference runs plotted against epoch, with validation points overlaid as circles for the 8B model and squares for the 30B-A3B model. (a) Preference accuracy, with the per-step training value in light shading and its smoothed trend in bold. (b) The implicit rewards assigned to the chosen and rejected responses, with the shaded gap between them giving the reward margin. (c) Unclipped gradient norm on a logarithmic scale, with the clipping threshold of 1.0 marked. The dotted vertical line marks epoch 0.14, corresponding to step 100 of the 8B run.

Panel of Figure 12 — Figure 12: The supervised and preference checkpoints on the translation probe of Figure 10 (b), under the same system prompt and decoding settings. (a) The 8B supervised checkpoint emits an empty reasoning block, while the preference checkpoint reasons in Cantonese about the lexical substitutions the task requires before answering. (b) The 30B-A3B supervised checkpoint states the task and lists two substitutions, while the preference checkpoint enumerates all four clauses of the source and gives a reason for each choice. English translations are the authors’ own.

Auxiliary	Curve
Reasoning steps	$\geq$ 3 indicators 1.0, 2 indicators 0.7,
	1 indicator 0.5, none 0
Response content	30–500 characters 1.0, 500–1,000
	cosine decay to 0.4, $>$ 1,000 at 0.4
Reasoning length	$<$ 100 characters 0.1, 100–1,300 at 1.0,
	1,300–3,584 cosine decay 1.0 to 0.5

Appendix

Table 30: The three auxiliary curves, averaged into

\overline{r_{a}}

. Reasoning steps counts indicator words from a 40-term set selected by the row’s language tag. Response content is measured over the span between the reasoning block and the answer span, and reasoning length over the reasoning block itself. Both length curves are counted in characters.

	Qwen3-8B	Qwen3-30B-A3B
Policy initialisation	DPO checkpoint, step 724	DPO checkpoint, step 435
Chat template	checkpoint’s native hybrid	checkpoint’s native hybrid
Parallelism	TP 1, PP 2, DP 2	TP 2, PP 1, EP 4, DP 4, sequence parallel
Generation engine	vLLM TP 1, memory utilisation 0.6	vLLM TP 4 decoupled, memory utilisation 0.5
Tuning	Full fine-tune	LoRA, $r$ 64, $\alpha$ 128, all linear
Learning rate	1.0 $\times$ 10 -6 to 1.0 $\times$ 10 -7	1.0 $\times$ 10 -5 to 1.0 $\times$ 10 -7

Appendix

Table 31: Configurations of the first reinforcement run. Shared settings, being the Megatron backend in BF16 with colocated vLLM generation and no reward server, Adam with

\beta

of 0.9 and 0.95 and weight decay 0.01, gradient clipping at 1.0, sequence packing disabled, full activation recomputation, validation at step 0, and a sampler at temperature 1.0 with top-

p

1.0 and no top-

k

, are omitted from the table. Each configuration is a leaf on a four-level inheritance chain and the values above are those in effect after resolution.

Setting	Step 1	Step 2
Task term	one verifier	six graders
Sequence length	4,096	16,384
Rollouts per step	128	192
Rollout precision	BF16	FP8 for 8B
Training precision	BF16	FP8 for 8B
Importance-sampling correction	off	on

Appendix

Table 32: Every setting that differs between the two reinforcement runs. All other settings, including the reward of Section 8.1 , the advantage estimator, the sampler and the seed, are shared.

#	Change	Steps	Outcome
1	correction disabled	115	entropy $-$ 82%,
			gradient norm 643
2	correction enabled,	79	stable, entropy
	truncation at 2.0		still $-$ 54%
3	memory utilisation 0.75	2	out of memory during
			generation wake

Appendix

Table 33: The four attempts at the 30B-A3B run of the second step. The first three consumed approximately 775 GPU-hours. Attempt 1 carried importance-sampling correction at its default of disabled. Attempt 4 is the run reported in Section 8.6 and released.

Figure 13: Telemetry for the second stage RLVR training, comparing the 8B training run against the three 30B-A3B attempts (Table 33 ). (a) Validation average reward. (b) Approximate policy entropy, which the first two attempts with the 30B-A3B collapsed. (c) Pre-clip gradient norm. (d) Solid lines trace the generation–training engine log-probability error between the generation and training engines, and the dotted lines trace the drift of the policy model from the rollout distribution. (e) Per-environment validation reward and (f) per-environment language multiplier for the 8B run, for which the multiplier increased in every environment.

	8B Dense		30B-A3B MoE
	h	%	h	%
Generation	0.99	37	0.94	23
Policy training	0.60	22	1.64	41
Log-probabilities	0.44	16	1.12	28
Preparation for generation	0.40	15	0.22	5
Remainder	0.28	10	0.10	3

Appendix

Table 34: Stage 1 training computation time over 150 steps. Reward computation is not shown as it totals 0.003 hours in both runs.

Disposition	Types	Basis
Transferred unchanged	22	Language-neutral, such as counts of paragraphs, sections or bullet points
Re-implemented	17	Depend on the writing system, so recounted over Jieba word segmentation, full-width punctuation and characters rather than words
Dropped	9	Specific to the Latin alphabet, such as capitalisation and letter-frequency rules
Retained from English	39
Added for the targets	13	New types defined over Cantonese and written Chinese orthography and phonology
Cantonese variant	52

Appendix

Table 35: Triage of the 48 English constraint types against the two target languages, and the 13 types added for them. The two variants differ only by the four types that have no written Chinese equivalent.

Low-Resource Language Processing

Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.

See all 565 papers

Jul 11, 2026

Cost of Reasoning in non-English Languages: A Case Study on Japanese

Reasoning Language Models (RLMs) achieve their strongest performance when they reason in English, the language for which reasoning-oriented training data is most abundant. However, reasoning trace is a clue for model interpretability and safety, and useful in practice for both the model users and for model developers. Thus, it is desirable to be able to develop a model that reasons in a language of the user's choice, while still maintaining strong reasoning performance. To this end, we study the feasibility of training a model that reasons in Japanese. We develop a Japanese-reasoning variant of Qwen-3-Swallow-8B, which is a Japanese LLM continually pretrained from Qwen-3-8B, with GRPO and evaluate it across coding, math, and science benchmarks. The study shows that reasoning-language control is feasible by training a Japanese continually pretrained model with GRPO. However, its performance is at best on par with strong English-reasoning baselines on several benchmarks. We also evaluate the trained model on Japanese cultural benchmarks and observe that the model's performance is worse than the baseline models, suggesting that the reasoning in Japanese does not immediately improve performance on culturally relevant tasks for free.

Yuu Jinnai

CyberAgent / Tokyo, Japan

Cross-Lingual Reasoning in Language Models Multilingual Language Model Evaluation RL for Language Model Reasoning Large Reasoning Models Multilingual Language Models Structured Reasoning in Language Models+1

Ask PDF

Aug 18, 2026

Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

Take three frontier mixture-of-experts models (Alibaba, OpenAI, NVIDIA; 3.6-4.0B active parameters each) and fine-tune them to reason in a low-resource language. On accuracy benchmarks almost nothing happens, and the benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured. That null is our first result. The real changes live where accuracy cannot see. Base models never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek, so the model answers correctly while reasoning in a form its user cannot read, audit, or correct. After supervised fine-tuning (SFT), every released checkpoint reasons in the language of the question on ~98% of items, one family at 3x fewer tokens, with judged grammaticality improving on all four models and general ability within a few points of each base: nothing was forgotten, and fluency was gained. We propose six behavioural dimensions that make such changes measurable, each gated to reject any metric that correlates with output length, and we report how our own instruments lied: six failures, each caught by a control. What SFT cannot do is fix its own defects: a quarter of answers skip the requested format, answers leak into the reasoning channel, and an explicit "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training, fixes the first two outright (fallback 24% to 2.5%, leak 3.5% to 0.0%, both against a flat random-reward control) and moves the third (+9.1pp), while the Greek reasoning habit survives an accuracy-only gradient untouched. We release five checkpoints. The instruments, the controls and the pre-registration travel to any low-resource language; Greek is the case that let us measure them.

Ayoub Kirouane, Christos Petrocheilos

Sophea AI, KIEFER SA, Athens, Greece

Fine-Tuning Language Models Structured Reasoning in Language Models Language Model Reasoning Evaluation Supervised Fine-Tuning Low-Resource Language Processing RL for Language Models

Ask PDF

Aug 1, 2026

Native Multilingual Chain-of-Thought Reasoning in Low-Resource Southeast Asian Languages

Large Language Models have achieved substantial progress in reasoning capabilities. Yet in low-resource native settings, many suffer from cross-lingual collapse, reverting to English during intermediate steps that require complex logical reasoning. This presents a cold-start bottleneck for policy optimization, whereas standard fine-tuning risks catastrophic forgetting due to cross-lingual representation drift. To address these challenges, we introduce the Onramp-Sequence Cross-Distillation (OSCD), a post-training algorithm that projects high-resource reasoning trajectories into low-resource vocabulary subspaces during generative training rollouts via an integrated translator agentic loop, ensuring the stable and efficient translation of dynamically generated reference samples for fine-tuning. This is coupled with joint-embedding semantic alignment of both reference and target-language reasoning traces, thereby bridging the pairwise cross-lingual representational gaps. Comprehensive evaluations using the AIME25 and HMMT25 benchmarks demonstrate that OSCD yields up to 3.2 times overall improvements in native Southeast Asian languages for mathematical reasoning, of which the joint-embedding semantic alignment component contributes up to 6.4% improvements in linguistic debiasing over translation-only baselines.

Sean Gip Lim, William Chandra Tjhi, Hai Leong Chieu

1Nanyang Technological University · 2AI Singapore · 3DSO National Laboratories

Numerical Reasoning in Language Models Multilingual Language Models Cross-Lingual Reasoning in Language Models Cross-Lingual Representation Alignment Language Model Distillation Low-Resource Language Processing+1

Ask PDF

Abstract

Low-Resource Language Processing

Explore similar work

Cost of Reasoning in non-English Languages: A Case Study on Japanese

Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

Native Multilingual Chain-of-Thought Reasoning in Low-Resource Southeast Asian Languages