cs.CLNov 18, 2026

When Does Continual Pre-Training Help? A Controlled Study of Dialectal Adaptation for Québec French

Authors: Eeham Khan, Richard Khoury, Leila Kosseim

Organizations: CLaC Lab, Department of Computer Science and Software Engineering, Concordia University · GRAAL, Department of Computer Science and Software Engineering, Université Laval

Abstract

Continual pre-training (CPT) is widely used to adapt pretrained language models to dialectal and non-standard language varieties, yet it remains unclear when this additional training is necessary, what drives its benefits, and whether aggregate evaluation scores adequately capture its effects. In this paper, we present a controlled study of CPT for Québec French language understanding across nine LLMs spanning different families and scales. Using a budget-matched protocol of 30M tokens per experiment, our results show that the benefit of CPT depends strongly on the starting model's existing competence in the target variety. When CPT is beneficial, the improvements are driven more by corpus composition than by data volume, with literary text accounting for most of the gain. Paired Québec and standard French evaluations further show that CPT primarily improves expression comprehension rather than grammatical competence, and that corpus register determines which regional expressions are acquired. These gains also involve capability-specific trade-offs, including declines in reading comprehension that aggregate scores can conceal. Overall, these findings show that CPT for dialectal adaptation should not be treated as a uniformly beneficial default. Instead, adaptation strategies should begin with a careful assessment of the base model's existing competence, accompanied by source-level data analysis and task-level evaluation.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustness

    Aug 6, 2026Aarohi Srivastava, David ChiangLanguage Model PretrainingMultilingual Language Models

  2. Diffract: Spectral View of LLM Domain Adaptation

    Aug 11, 2026Nikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain AdaptationDomain-Adaptive Pretraining

  3. DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

    Jul 8, 2026Jordan Painter, Dipankar Srirag, Adarsh Kappiyath +3Language ModelingLLM Alignment