A Catalog of Basque Dialectal Resources: Online Collections and Standard-to-Dialectal Adaptations

Large Language Models (LLMs) have achieved remarkable progress across natural language processing (NLP) tasks, yet their capabilities degrade sharply for low-resource languages and dialectally diverse settings. Bangla, the world's sixth most spoken language, exemplifies this gap: existing resources overwhelmingly target Standard Bangla, leaving its regional dialects without the benchmarks needed to develop or evaluate dialect-aware systems. We address this gap with 5-Dialects-BN, the first multi-annotation Bangla dialect benchmark to align Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five regional varieties. The dataset comprises 6,000 manually annotated entries spanning five major dialects: Chittagong, Barisal, Noakhali, Sylhet, and Rangpur (Chittagong 1,900; Noakhali 1,500; Sylhet 1,200; Barisal 700; Rangpur 700), reflecting natural online availability. Each entry is enriched with five aligned annotations: the original dialectal text, a Romanized transliteration, an English translation, a Standard Bangla translation, and a subjectivity label (subjective vs. objective). Annotations were produced and cross-validated by native speakers and undergraduate linguistics students to ensure dialectal authenticity and semantic fidelity. The resulting resource supports a diverse suite of tasks, including dialect identification, dialect-to-standard normalization, machine translation, subjectivity classification, and parameter-efficient fine-tuning (e.g., LoRA) of multilingual LLMs. By providing a standardized, multi-annotation benchmark, 5-Dialects-BN enables principled evaluation of LLMs on dialectally diverse Bangla and lays a foundation for further research in low-resource, dialect-aware NLP.

Large language models increasingly understand dialectal English, yet still produce only standard, US-leaning English, leaving dialectal generation, the harder half of the problem, largely unaddressed. We introduce DiaLLM, which continually pretrains three open-weight language model families on the International Corpus of English and applies implicit and explicit post-training paradigms, each combined with three model alignment strategies, giving the first controlled comparison of these components across Australian, Indian, and Northern British English. Our results reveal a robustness-generation gap: benchmarks are shaped by continual pretraining and SFT, while alignment visibly reshapes generation in ways benchmarks do not capture. Explicit variety-targeted adaptation produces output reliably recognised as dialectal and judged more dialectal than broad alignment, yet where human judgement was directly assessed, the method that most aggressively optimises the dialectal reward is not the one judged most dialectal. Independent linguistic analysis corroborates this reward-quality gap, most clearly on two of the three families. No single alignment method dominates, and closing the gap will require richer reward designs and continued investment in dialectal resources. We release all code, checkpoints, and preference datasets.

Neural Machine Translation (NMT) and Large Language Models (LLMs) excel at cross-lingual tasks but often fail to capture intra-lingual morphological variation, marginalizing dialectal speakers. In Bangla, existing translation frameworks commonly rely on Standard Colloquial Bangla (SCB) as an intermediate pivot, which can compound errors and reduce cross-dialectal nuance. To address this gap, we introduce a unified, multi-directional NMT system for direct translation between SCB and eleven regional variants. We first review prior dialectal NLP resources to identify existing technological gaps. As a foundational contribution, we construct and release a large multi-dialect parallel corpus for Bangla, comprising 14,562 aligned rows and 51,541 non-null sentence pairs through the integration of seven prior datasets and native-speaker-verified manual augmentation. Using this corpus, we benchmark state-of-the-art sequence-to-sequence architectures with parameter-efficient Weight-Decomposed Low-Rank Adaptation (DoRA). Results show that deep monolingual pre-training is more effective than large multilingual capacity for this task. The compact BanglaT5 model outperforms NLLB-200 and mBART-50 by up to 13.96 BLEU, achieving 29.26 BLEU, 57.26 chrF++, and 49.68 METEOR. A dataset scaling study shows diminishing returns beyond 3,000 parallel pairs and indicates that linguistic proximity to Standard Bangla is more important than raw data volume for translation quality. Finally, we deploy the optimized model as an INT8-quantized web application, providing a scalable, open-source framework for inclusive language technology and equitable digital access.