When Does Continual Pre-Training Help? A Controlled Study of Dialectal Adaptation for Québec French
Authors: Eeham Khan, Richard Khoury, Leila Kosseim
Organizations: CLaC Lab, Department of Computer Science and Software Engineering, Concordia University · GRAAL, Department of Computer Science and Software Engineering, Université Laval
Continual pre-training (CPT) is widely used to adapt pretrained language models to dialectal and non-standard language varieties, yet it remains unclear when this additional training is necessary, what drives its benefits, and whether aggregate evaluation scores adequately capture its effects. In this paper, we present a controlled study of CPT for Québec French language understanding across nine LLMs spanning different families and scales. Using a budget-matched protocol of 30M tokens per experiment, our results show that the benefit of CPT depends strongly on the starting model's existing competence in the target variety. When CPT is beneficial, the improvements are driven more by corpus composition than by data volume, with literary text accounting for most of the gain. Paired Québec and standard French evaluations further show that CPT primarily improves expression comprehension rather than grammatical competence, and that corpus register determines which regional expressions are acquired. These gains also involve capability-specific trade-offs, including declines in reading comprehension that aggregate scores can conceal. Overall, these findings show that CPT for dialectal adaptation should not be treated as a uniformly beneficial default. Instead, adaptation strategies should begin with a careful assessment of the base model's existing competence, accompanied by source-level data analysis and task-level evaluation.
Figures & tables
Source
Genre
Register
Tokens
BEQ
Literary
Formal
15.4M
Québec Wikipedia
Encyclopedic
Formal
31.2M
Le Soleil
News
Formal
5.9M
Formal subtotal
52.5M
CRIFUQ
Speech
Informal
0.8M
Facebook
Social media
Informal
2.4M
Table 1: Composition of the Mixed-Source Québec French Adaptation Corpus. Token count was computed using the CroissantLLM tokenizer 1 1 1 available at https://huggingface.co/croissantllm . .
Figure 1: Register Sweep Across All Models: Shifts in grammatical performance (top row) and expression comprehension (bottom row) at the two register extremes for all nine models. Models are ordered left to right by baseline QFrCoRE. Results are difference in Macro-F1 compared to baseline result, averaged over three seeds.
Budget
Formal Sources (M)
Informal Sources (M)
Mixture Condition
(M)
BEQ
Wiki
Soleil
CRIFUQ
FB
Dép
MR
YT
Reddit
Original Corpus
86.7
15.4
31.2
5.9
0.8
2.4
22.8
4.2
2.2
1.8
Formal-only (100/0)
30.0
8.8
17.8
3.4
–
–
–
–
–
–
75% Formal (75/25)
30.0
6.6
13.4
2.5
0.2
0.5
5.0
0.9
0.5
0.4
Mixed (50/50)
30.0
4.4
8.9
1.7
0.4
1.1
10.0
1.8
1.0
0.8
75% Informal (25/75)
30.0
2.2
4.5
0.8
0.5
1.6
15.0
2.8
1.4
1.2
Table 2: Corpus Variants Used in the Register Sweep. Values represent the allocated token budget (in millions) for each source. Sources within each register are sampled proportionally to their raw availability to maintain exactly 30M tokens per condition.
Québec French
Standard French
PPL ↓
Condition
QFrBLiMP
QFrCoRE
AlloCiné
PAWS
Fr-BoolQ
MMS
QC
FR
Base (no CPT)
89.2
27.5
50.4
35.5
56.2
25.4
17.24
8.31
Formal-only
89.9
35.7
56.6
41.0
37.7
25.5
15.63
9.89
75/25
89.4
33.9
54.1
39.1
40.2
25.4
15.79
9.73
Mixed (50/50)
90.2
33.7
50.8
39.0
40.8
25.3
15.99
9.62
25/75
91.0
32.2
57.2
38.6
41.7
25.3
16.30
9.53
Table 3: Continuous Register Sweep on CroissantLLM 1.35B: Results are Macro-F1, averaged over three seeds. Perplexity (PPL) is evaluated on held-out Québec French and standard French sets.
Table 4: Leave-One-Genre-Out Ablations at fixed budget for CroissantLLM 1.35B. All rows train on the same 30M token budget. Results are the mean of three seeds.
Figure 2: Impact of Baseline Competence: CPT performance plotted against the unadapted baseline. The diagonal dashed line represents unchanged performance.
Figure 3: Standard French replay on the 50/50 Québec French mixture for CroissantLLM 1.35B.
Figure 4: Held-out Perplexity during standard French replay for CroissantLLM 1.35B.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Grammar
Expressions
QFrBLiMP
MultiBLiMP -Fr
QFrCoRE
MFrCoE
Model
Formal
Inf.
Formal
Inf.
Formal
Inf.
Formal
Inf.
Croissant 1.35B
+0.7
−0.8
−1.3
−1.3
+8.2
+0.7
+2.0
+4.8
Qwen2.5 0.5B
−0.7
−2.0
−4.8
+1.7
+1.5
−3.4
+0.7
+4.2
Qwen2.5 1.5B
+1.4
+0.4
+1.7
+1.3
+0.6
−4.0
+0.3
+5.3
Llama-3.2 1B
+0.8
−0.4
−0.9
−2.2
+1.4
−4.3
+0.4
+2.2
Appendix
Table 5: Grammatical performance and expression comprehension under the two register extremes (formal-only and informal-only CPT), for all nine models. Results are reported as the change in Macro-F1 relative to each model’s unadapted baseline.
Model
Family
Base
+ CPT
Δ
CroissantLLM 1.35B
Croissant
27.5
35.7
+8.2
Qwen2.5 0.5B
Qwen
30.2
31.7
+1.5
Qwen2.5 1.5B
Qwen
31.5
32.1
+0.6
Llama-3.2 1B
Llama
31.9
33.3
+1.4
Llama-3.2 3B
Llama
33.0
33.1
+0.1
Llama-3.1 8B
Llama
35.9
34.5
−1.4
Appendix
Table 6: Formal-only CPT versus each model’s own untrained baseline on QFrCoRE. Scores are reported as Macro-F1, and taken as the mean of three random seeds.
Québec French
Standard French
PPL ↓
Condition
BLiMP
CoRE
AlloCiné
PAWS
BoolQ
MMS
QC
FR
No replay (ref.)
90.2
33.3
50.8
39.0
40.8
25.3
15.99
9.62
10% replay
90.0
33.9
47.7
36.2
45.8
25.4
16.02
8.67
25% replay
90.3
33.7
45.4
36.0
48.1
25.4
16.08
8.55
50% replay
89.8
33.7
43.1
35.7
50.1
25.4
16.22
8.47
25% FR+EN
90.4
33.7
45.7
36.0
48.0
25.4
16.08
8.55
Appendix
Table 7: Standard French replay on a 50/50 Québec French mixture for CroissantLLM 1.35B. Deltas discussed in the text are relative to the unadapted base model in Table 3 . Results are the mean of three seeds.