When Does Continual Pre-Training Help? A Controlled Study of Dialectal Adaptation for Québec French
Organizations: CLaC Lab, Department of Computer Science and Software Engineering, Concordia University · GRAAL, Department of Computer Science and Software Engineering, Université Laval
Abstract
Continual pre-training (CPT) is widely used to adapt pretrained language models to dialectal and non-standard language varieties, yet it remains unclear when this additional training is necessary, what drives its benefits, and whether aggregate evaluation scores adequately capture its effects. In this paper, we present a controlled study of CPT for Québec French language understanding across nine LLMs spanning different families and scales. Using a budget-matched protocol of 30M tokens per experiment, our results show that the benefit of CPT depends strongly on the starting model's existing competence in the target variety. When CPT is beneficial, the improvements are driven more by corpus composition than by data volume, with literary text accounting for most of the gain. Paired Québec and standard French evaluations further show that CPT primarily improves expression comprehension rather than grammatical competence, and that corpus register determines which regional expressions are acquired. These gains also involve capability-specific trade-offs, including declines in reading comprehension that aggregate scores can conceal. Overall, these findings show that CPT for dialectal adaptation should not be treated as a uniformly beneficial default. Instead, adaptation strategies should begin with a careful assessment of the base model's existing competence, accompanied by source-level data analysis and task-level evaluation.
Figures & tables
| Source | Genre | Register | Tokens |
| BEQ | Literary | Formal | 15.4M |
| Québec Wikipedia | Encyclopedic | Formal | 31.2M |
| Le Soleil | News | Formal | 5.9M |
| Formal subtotal | 52.5M | ||
| CRIFUQ | Speech | Informal | 0.8M |
| Social media | Informal | 2.4M | |
| Budget | Formal Sources (M) | Informal Sources (M) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Mixture Condition | (M) | BEQ | Wiki | Soleil | CRIFUQ | FB | Dép | MR | YT | |
| Original Corpus | 86.7 | 15.4 | 31.2 | 5.9 | 0.8 | 2.4 | 22.8 | 4.2 | 2.2 | 1.8 |
| Formal-only (100/0) | 30.0 | 8.8 | 17.8 | 3.4 | – | – | – | – | – | – |
| 75% Formal (75/25) | 30.0 | 6.6 | 13.4 | 2.5 | 0.2 | 0.5 | 5.0 | 0.9 | 0.5 | 0.4 |
| Mixed (50/50) | 30.0 | 4.4 | 8.9 | 1.7 | 0.4 | 1.1 | 10.0 | 1.8 | 1.0 | 0.8 |
| 75% Informal (25/75) | 30.0 | 2.2 | 4.5 | 0.8 | 0.5 | 1.6 | 15.0 | 2.8 | 1.4 | 1.2 |
| Québec French | Standard French | PPL | ||||||
| Condition | QFrBLiMP | QFrCoRE | AlloCiné | PAWS | Fr-BoolQ | MMS | QC | FR |
| Base (no CPT) | 89.2 | 27.5 | 50.4 | 35.5 | 56.2 | 25.4 | 17.24 | 8.31 |
| Formal-only | 89.9 | 35.7 | 56.6 | 41.0 | 37.7 | 25.5 | 15.63 | 9.89 |
| 75/25 | 89.4 | 33.9 | 54.1 | 39.1 | 40.2 | 25.4 | 15.79 | 9.73 |
| Mixed (50/50) | 90.2 | 33.7 | 50.8 | 39.0 | 40.8 | 25.3 | 15.99 | 9.62 |
| 25/75 | 91.0 | 32.2 | 57.2 | 38.6 | 41.7 | 25.3 | 16.30 | 9.53 |
| Québec French | Standard French | PPL | ||||||
|---|---|---|---|---|---|---|---|---|
| Removed Genre | QFrBLiMP | QFrCoRE | AlloCiné | PAWS | Fr-BoolQ | MMS | QC | FR |
| None (full corpus) | 90.2 | 33.7 | 53.1 | 39.0 | 40.5 | 25.4 | 15.91 | 9.68 |
| - Encyclopedic (QC Wiki, 31.2M) | 90.1 | 33.8 | 57.3 | 39.7 | 40.3 | 25.4 | 16.32 | 9.51 |
| - Literary BEQ, (15.4M) | 90.3 | 28.3 | 50.8 | 40.3 | 35.2 | 25.3 | 10.46 | 11.73 |
| - News (Le Soleil, 5.9M) | 90.2 | 34.8 | 53.5 | 35.9 | 45.8 | 25.3 | 15.85 | 9.71 |
| - Forums/social (Dépotoir, MtlRacing, FB, Reddit, 31.2M) | 89.5 | 34.9 | 65.0 | 38.8 | 41.9 | 25.5 | 15.65 | 9.85 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Grammar | Expressions | |||||||
|---|---|---|---|---|---|---|---|---|
| QFrBLiMP | MultiBLiMP -Fr | QFrCoRE | MFrCoE | |||||
| Model | Formal | Inf. | Formal | Inf. | Formal | Inf. | Formal | Inf. |
| Croissant 1.35B | ||||||||
| Qwen2.5 0.5B | ||||||||
| Qwen2.5 1.5B | ||||||||
| Llama-3.2 1B | ||||||||
| Model | Family | Base | CPT | |
|---|---|---|---|---|
| CroissantLLM 1.35B | Croissant | 27.5 | 35.7 | |
| Qwen2.5 0.5B | Qwen | 30.2 | 31.7 | |
| Qwen2.5 1.5B | Qwen | 31.5 | 32.1 | |
| Llama-3.2 1B | Llama | 31.9 | 33.3 | |
| Llama-3.2 3B | Llama | 33.0 | 33.1 | |
| Llama-3.1 8B | Llama | 35.9 | 34.5 |
| Québec French | Standard French | PPL | ||||||
|---|---|---|---|---|---|---|---|---|
| Condition | BLiMP | CoRE | AlloCiné | PAWS | BoolQ | MMS | QC | FR |
| No replay (ref.) | 90.2 | 33.3 | 50.8 | 39.0 | 40.8 | 25.3 | 15.99 | 9.62 |
| 10% replay | 90.0 | 33.9 | 47.7 | 36.2 | 45.8 | 25.4 | 16.02 | 8.67 |
| 25% replay | 90.3 | 33.7 | 45.4 | 36.0 | 48.1 | 25.4 | 16.08 | 8.55 |
| 50% replay | 89.8 | 33.7 | 43.1 | 35.7 | 50.1 | 25.4 | 16.22 | 8.47 |
| 25% FR+EN | 90.4 | 33.7 | 45.7 | 36.0 | 48.0 | 25.4 | 16.08 | 8.55 |
Continued Pretraining
Continued pretraining adapts pretrained models to new domains, languages, modalities, or contexts by training further on targeted data.
See all 59 papers