HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head

Thisen Ekanayake; Nisansa de Silva

cs.CLAug 24, 2026

HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head

Authors: Thisen Ekanayake, Nisansa de Silva

Organizations: Department of Computer Science & Engineering University of Moratuwa

Abstract

We present HelaBERT, a family of two BERT-based masked language models pre-trained from scratch on approximately 1 billion tokens of Sinhala text sourced from MADLAD-400, CulturaX, and a custom corpus comprising news articles, Sinhala Wikipedia, and web crawl data. HelaBERT-Small (~23.3M parameters, 6 layers) and HelaBERT-Large (~110M parameters, 12 layers) both use a SentencePiece Unigram tokenizer (vocabulary size 32,000) tailored to Sinhala's agglutinative morphology and complex script. We evaluate both models on four downstream Sinhala text classification tasks: news category classification, news source classification, sentiment analysis, and writing style classification, using 5 independent seed runs with stratified 80/20 train/test splits. We additionally propose a dual pooling classification head and evaluate it systematically across all four tasks, finding consistent improvements on sentiment analysis and a moderate gain on news category classification for HelaBERT-Small, while the standard [CLS]-linear head remains competitive on news source classification, a headline-level task with short average input length. We release both models to support further research in Sinhala NLP.

Figures & tables

Parameter	Small	Large
Parameters	$\sim$ 23.3M	$\sim$ 110M
Hidden size	384	768
Transformer layers	6	12
Attention heads	6	12
Intermediate size	1,536	3,072
Max sequence length	512	512

Table 1: Architecture comparison between HelaBERT-Small and HelaBERT-Large.

Metric	Small	Large
Final train loss	3.69	2.26
Final eval loss	3.49	2.17
Total training steps	$\sim$ 52,000	$\sim$ 90,700

Table 2: Pre-training results for HelaBERT-Small and HelaBERT-Large.

Figure 1: Training and validation loss curves for HelaBERT-Small.

Figure 2: Training and validation loss curves for HelaBERT-Large.

Dataset	Classes	Train	Test	Avg. W
News Category	5	2,596	640	23.8
News Source	9	18,280	4,571	8.3
Sentiment	3	2,049	513	16.8
Writing Style	4	10,008	2,462	182.6

Table 3: Summary statistics of the four fine-tuning datasets. Avg. Words computed over the training split.

	News Cat.	News Src.	Senti.	Style
LR (Small)	1e-5	5e-5	3e-5	1e-5
LR (Large)	3e-5	5e-5	5e-6	1e-5
Epochs (Small)	10	3	6	3
Epochs (Large)	3	3	10	3
Max seq. len.	512	512	512	512
Batch size	16

Table 4: Fine-tuning hyperparameters per task. LR = learning rate.

Model	Sentiment †	News Source	News Category	Writing Style
Baseline (majority class)	59.42 (w-F 1 )	—	—	—
LaBSE	20.63	11.85	24.09	—
LASER	54.07	28.84	48.54	87.06
XLM-R ${}_{\text{base}}$	58.08	58.29	85.12	96.89
XLM-R ${}_{\text{large}}$	60.45	61.84	89.54	98.41
SinBERT O	50.83	57.22	78.07	93.84

Table 5: Macro-F 1 (%) comparison on four Sinhala text classification tasks. All baseline results are taken from 8 . † Sentiment results for HelaBERT models are evaluated on a different publicly available 3-class dataset (positive/negative/neutral); all other models used a 4-class dataset (including conflict ) that is no longer publicly available. These columns are not directly comparable .

Panel of Figure 3 — Figure 3: Dual pooling classification head on top of HelaBERT. The encoder output is split into the [CLS] vector $\mathbf{c}$ , the token sequence $\mathbf{T}$ , and a padding mask $\mathbf{m}$ . A shared affinity vector is computed via additive attention. Dir 1 (blue): [CLS] attends over tokens via softmax, yielding $\tilde{\mathbf{c}}$ . Dir 2 (orange): tokens attend back to [CLS] via a sigmoid gate, yielding $\tilde{\mathbf{T}}$ . Both outputs are layer-normalized, concatenated to $\mathbb{R}^{2H}$ , and passed through a two-layer MLP to produce class logits.

Model	Head	News Cat.	News Src.	Sentiment	Writing
Small	Standard	85.97	60.16	65.34	95.92
	Co-attn	89.02	59.90	69.27	96.42
	$\Delta$	+3.05	$-$ 0.26	+3.93	+0.50
Large	Standard	90.38	63.65	64.60	97.73
	Co-attn	90.50	63.31	70.22	97.75
	$\Delta$	+0.12	$-$ 0.34	+5.62	+0.02

Table 6: Standard [CLS] -linear head vs. dual pooling classification head: macro-F 1 (%) on all four tasks (best run of 5 seeds).

\Delta

= dual pooling

-

standard. Positive

\Delta

favours dual pooling; negative values in italics .

Model / Head	Acc.	Macro-F 1	W-F 1
HelaBERT-Small, standard	0.6589	0.6534	0.6581
HelaBERT-Small, dual pooling	0.6881	0.6927	0.6904
$\Delta$ (dual pooling $-$ std.)	+0.029	+0.039	+0.032
HelaBERT-Large, standard	0.6530	0.6460	0.6539
HelaBERT-Large, dual pooling	0.7135	0.7022	0.7096
$\Delta$ (dual pooling $-$ std.)	+0.061	+0.056	+0.056

Table 7: Standard vs. dual pooling sentiment results (best run of 5). All figures on the same 3-class test set (

N=513

	Standard			Dual pooling
Model	NEG	NEU	POS	NEG	NEU	POS
Small	0.54	0.67	0.75	0.64	0.67	0.77
Large	0.55	0.68	0.70	0.59	0.73	0.78

Table 8: Per-class F 1 for standard vs. dual pooling on sentiment (best run;

N=513

). NEG = negative, NEU = neutral, POS = positive.

Figure 4: Co-attention vs. standard head: $\Delta$ macro-F 1 (percentage points) across all four tasks for HelaBERT-Small (blue) and HelaBERT-Large (green). Bars below zero indicate tasks where the standard head outperforms co-attention.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Small	Large
Sequence length	256 (stride 128)	512
Total training samples	$\sim$ 7.4M	$\sim$ 4.3M
Train / validation split	90% / 10%	90% / 10%
MLM probability	15%	15%
Per-device batch size	32	256
Gradient accumulation steps	8	1

Appendix

Table 9: Pre-training hyperparameters for HelaBERT-Small and HelaBERT-Large.

Label	Train	Test
0	418	102
1	365	91
2	675	166
3	797	199
4	341	82
Total	2,596	640

Appendix

Table 10: Per-label train/test distribution for News Category Classification.

Label	Train	Test
0	2,233	558
1	1,303	326
2	2,334	584
3	1,143	286
4	2,366	592
5	2,282	571

Appendix

Table 11: Per-label train/test distribution for News Source Classification.

Label	Train	Test
Neutral	899	225
Positive	599	150
Negative	551	138
Total	2,049	513

Appendix

Table 12: Per-label train/test distribution for Sentiment Analysis.

Label	Train	Test
News	3,568	885
Academic	2,834	701
Creative	1,919	458
Blog	1,687	418
Total	10,008	2,462

Appendix

Table 13: Per-label train/test distribution for Writing Style Classification.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

SinLlama -- A Large Language Model for Sinhala
Aug 12, 2025H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake +3Language Model Pretraining Multilingual Language Models
Dept. of Computer Science & Engineering, University of Moratuwa, Sri Lanka. · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand. · Central Scientific Instruments Organisation, Academy of Scientific and Innovative Research, Chandigarh, India. +1
DunbaaBERT: From Sacrifice to Semantics
May 26, 2026Iffat Maab, Waleed Jamil, Raphael SchmittLanguage Model Pretraining Language Modeling
Research and Development Center for Large Language Models (LLMC), National Institute of Informatics, Tokyo · Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany +1
SiNFluD: Creating and Evaluating Figurative Language Dataset for Sindhi
May 2, 2026Wazir Ali, Adeeb Noor, Saifullah TumraniFigurative Language Understanding Text Classification
Department of Data Science, Quaid-e-Awam University of Engineering, Science and Technology, Nawabshah, Pakistan. · SoloGenAI Pvt. Ltd. · Department of Information Technology, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia. +1

Aug 12, 2025J/K move · Enter open · S save

SinLlama -- A Large Language Model for Sinhala

H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake, Nisansa de Silva+2

Dept. of Computer Science & Engineering, University of Moratuwa, Sri Lanka. · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand. · Central Scientific Instruments Organisation, Academy of Scientific and Innovative Research, Chandigarh, India.+1

Language Model Pretraining Multilingual Language Models Language Modeling Tokenizer Adaptation

Open paper AskPDF arXiv

Low-resource languages such as Sinhala are often overlooked by open-source Large Language Models (LLMs). In this research, we extend an existing multilingual LLM (Llama-3-8B) to better serve Sinhala. We enhance the LLM tokenizer with Sinhala specific vocabulary and perform continual pre-training on a cleaned 10 million Sinhala corpus, resulting in the SinLlama model. This is the very first decoder-based open-source LLM with explicit Sinhala support. When SinLlama was instruction fine-tuned for three text classification tasks, it outperformed base and instruct variants of Llama-3-8B by a significant margin.

Abstract

Language Model Pretraining

Explore similar work

SinLlama -- A Large Language Model for Sinhala

DunbaaBERT: From Sacrifice to Semantics

SiNFluD: Creating and Evaluating Figurative Language Dataset for Sindhi