On the Effectiveness and Limitations of Open-Weight LLMs for Low-Resource Offensive Language Detection in Danish
Organizations: Texas A&M University-Corpus Christi · Mississippi State University · University of Miami
Abstract
We study the effectiveness and robustness of open-weight large language models (LLMs) for Danish offensive language detection, an underexplored low-resource task. We evaluate four model families (Llama, Mistral, Gemma, and Qwen) across eleven prompting strategies spanning five design categories, cross-prompt generalization, Danish orthographic variation, and robustness under balanced and naturally imbalanced class distributions. We compare against three BERT-family baselines with distinct pre-training strategies, evaluated via both masked language model prompting and supervised fine-tuning. Even without task-specific fine-tuning, LLMs can surpass all three fine-tuned BERT baselines, and lightweight finetuning yields substantial further gains. No single model or prompt category dominates, underscoring that prompt design interacts with model pre-training in ways that cannot be predicted a priori. Semantically grounded prompts transfer more robustly, while format-constrained patterns consistently fail to generalize. ASCII transliteration of Danish-specific characters (æ, ø, å) generally degrades performance, indicating these characters encode meaningful signal. At the same time, the gap between a balanced test set (best F1 of 0.84) and the corpus's natural class skew (0.62 trained on skewed data, 0.70 trained on balanced data) -- relative drops of 26.2% and 16.7% -- shows that class imbalance remains a critical open challenge. Our code and configurations are available at https://github.com/PASHAADC/Danish-LLM-HateSpeech.
Figures & tables
| Pattern | LLaMA | Mistral | Gemma | Qwen |
|---|---|---|---|---|
| vanilla QA | 3 | 2 | 41 | 13 |
| choice QA | 1 | 2 | 26 | 1 |
| cloze | 86 | 98 | 88 | 91 |
| cot | 7 | 2 | 4 | 3 |
| target | 4 | 2 | 5 | 14 |
| illocutionary | 5 | 3 | 96 | 8 |
| Setting | Model | Reported | Corrected | Coverage | |
| Balanced, base | |||||
| LLaMA | .53 | .55 | .89 | ||
| Mistral | .52 | .54 | .89 | ||
| Gemma | .59 | .66 | .75 | ||
| Qwen | .52 | .52 | .84 | ||
| Balanced, fine-tuned | |||||
| Balanced | Imbalanced | ||||||
|---|---|---|---|---|---|---|---|
| Model | Configuration | P | R | F1 | P | R | F1 |
| mBERT | (80/10/10) | .51 | .67 | .58 | .22 | .49 | .31 |
| DaBERT | (80/10/10) | .51 | .76 | .61 | .29 | .60 | .39 |
| ScandiBERT | (80/10/10) | .49 | 1.0 | .66 | .13 | 1.0 | .23 |
| mBERT | (60/10/30) | .60 | .51 | .55 | .27 | .38 | .31 |
| DaBERT | (60/10/30) | .47 | .48 | .48 | .26 | .55 | .35 |
| Balanced | Imbalanced | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLaMA | Mistral | Gemma | Qwen | LLaMA | Mistral | Gemma | Qwen | |||||||||||||||||
| Pattern | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 |
| vanilla | .46 | .45 | .46 | .54 | .52 | .53 | .71 | .40 | .52 | .58 | .26 | .36 | .15 | .65 | .25 | .19 | .63 | .29 | .30 | .35 | .32 | .35 | .44 | .39 |
| choice | .54 | .88 | .67 | .45 | .81 | .58 | .61 | .55 | .57 | .61 | .55 | .57 | .12 | .81 | .21 | .13 | .86 | .22 | .23 | .53 | .32 | .25 | .60 | .35 |
| cloze | .62 | .36 | .45 | .50 | .02 | .05 | .79 | .36 | .49 | .64 | .43 | .51 | .17 | .33 | .22 | .18 | .09 | .12 | .24 | .30 | .27 | .16 | .33 | .22 |
| cot | .50 | .36 | .42 | .53 | .71 | .61 | .66 | .60 | .62 | .50 | .12 | .19 | .12 | .37 | .18 | .16 | .74 | .26 | .22 | .60 | .33 | .19 | .19 | .19 |
| Balanced | Imbalanced | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLaMA | Mistral | Gemma | Qwen | LLaMA | Mistral | Gemma | Qwen | |||||||||||||||||
| Pattern | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 |
| vanilla | .77 | .55 | .64 | .53 | .83 | .65 | .82 | .86 | .84 | .61 | .90 | .73 | .00 | .00 | .00 | .00 | .00 | .00 | .21 | .88 | .33 | .14 | .84 | .24 |
| choice | .48 | .69 | .57 | .57 | .67 | .62 | .78 | .76 | .77 | .73 | .76 | .74 | .15 | .56 | .24 | .20 | .47 | .28 | .26 | .63 | .37 | .16 | .93 | .27 |
| cloze | .57 | .88 | .69 | .72 | .31 | .43 | .72 | .62 | .67 | .49 | .88 | .63 | .15 | .40 | .22 | .00 | .00 | .00 | .58 | .35 | .43 | .19 | .47 | .27 |
| cot | .57 | .86 | .69 | .53 | .79 | .63 | .64 | .88 | .74 | .74 | .88 | .80 | .00 | .00 | .00 | .21 | .21 | .21 | .13 | .67 | .22 | .36 | .63 | .45 |
| Balanced | Imbalanced | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLaMA | Mistral | Gemma | Qwen | LLaMA | Mistral | Gemma | Qwen | |||||||||||||||||
| Pattern | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 |
| vanilla | .50 | .99 | .67 | .48 | .38 | .42 | .52 | .50 | .51 | .50 | 1.00 | .67 | .20 | .02 | .04 | .50 | .01 | .02 | .79 | .21 | .33 | .13 | .73 | .22 |
| choice | .50 | .99 | .66 | .48 | .73 | .58 | .49 | .46 | .48 | .50 | .99 | .67 | .15 | .66 | .24 | .09 | .07 | .08 | .00 | .00 | .00 | .21 | .15 | .17 |
| cloze | .50 | .96 | .66 | .51 | .72 | .60 | .70 | .33 | .45 | .52 | .91 | .66 | .12 | .73 | .21 | .24 | .46 | .31 | .70 | .05 | .10 | .11 | .15 | .13 |
| cot | .55 | .77 | .64 | .33 | .02 | .03 | .65 | .50 | .57 | .50 | .98 | .66 | .00 | .00 | .00 | .30 | .41 | .35 | .18 | .25 | .21 | .80 | .12 | .22 |
| Configuration A (80/10/10) | Configuration B (60/10/30) | |||||||||||||||||||||||
| LLaMA | Mistral | Gemma | Qwen | LLaMA | Mistral | Gemma | Qwen | |||||||||||||||||
| Pattern | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 |
| vanilla | .89 | .57 | .70 | .64 | .33 | .44 | .60 | .62 | .61 | .52 | .76 | .62 | .73 | .82 | .77 | .55 | .76 | .63 | .73 | .54 | .62 | .56 | .95 | .71 |
| choice | .51 | .95 | .66 | .38 | .36 | .37 | .79 | .81 | .80 | .54 | .86 | .66 | .50 | 1.00 | .66 | .50 | .85 | .63 | .66 | .98 | .79 | .53 | .98 | .68 |
| cloze | .50 | .98 | .66 | .94 | .40 | .57 | .60 | .50 | .55 | .47 | .88 | .62 | .51 | .99 | .67 | .50 | .87 | .63 | .78 | .90 | .84 | .58 | .76 | .66 |
| cot | .92 | .57 | .71 | .65 | .26 | .37 | 1.00 | .29 | .44 | .73 | .86 | .79 | .50 | 1.00 | .66 | .00 | .00 | .00 | .62 | .87 | .73 | .53 | .95 | .68 |
| Configuration A (80/10/10) | Configuration B (60/10/30) | |||||||||||||||||||||||
| LLaMA | Mistral | Gemma | Qwen | LLaMA | Mistral | Gemma | Qwen | |||||||||||||||||
| Pattern | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 |
| vanilla | .25 | .70 | .37 | .20 | .42 | .27 | .16 | .86 | .28 | .34 | .70 | .45 | .15 | .17 | .16 | .14 | .70 | .24 | .14 | .41 | .21 | .17 | .11 | .13 |
| choice | .32 | .84 | .46 | .14 | .79 | .24 | .22 | .63 | .33 | .17 | .91 | .29 | .13 | .96 | .24 | .13 | .49 | .20 | .24 | .38 | .29 | .13 | .98 | .23 |
| cloze | .18 | .91 | .30 | .22 | .86 | .34 | .85 | .26 | .39 | .13 | 1.00 | .24 | .15 | .82 | .25 | .14 | .59 | .23 | .21 | .71 | .32 | .14 | .81 | .24 |
| cot | .21 | .30 | .25 | .13 | 1.00 | .23 | .28 | .60 | .39 | .31 | .81 | .45 | .20 | .45 | .27 | .13 | .36 | .19 | .29 | .34 | .32 | .17 | .80 | .29 |
| Model | Vocab | w/ æøå | w/o æøå | Ratio |
|---|---|---|---|---|
| LLaMA | 128K | 2.64 | 1.68 | 1.57 |
| Mistral | 32K | 3.18 | 1.89 | 1.68 |
| Gemma | 256K | 2.09 | 1.50 | 1.40 |
| Qwen | 152K | 2.65 | 1.70 | 1.56 |
| LLaMA | Mistral | Gemma | Qwen | |
| Orthographic variation (F1 averaged across patterns) | ||||
| Balanced | ||||
| Original (æøå) | .594 | .501 | .679 | .691 |
| ASCII (aeoeaa) | .597 | .562 | .550 | .579 |
| Imbalanced | ||||
| Original (æøå) | .267 | .283 | .486 | .227 |
Hate Speech Detection
Hate Speech Detection identifies hateful or abusive language, its targets and intent, to support safer and more accountable content moderation.
See all 107 papers