GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation
Authors: Quang Anh Kiet Tran, Nam Anh Tran, Manh Hung Trinh, Thanh Danh Le, Khang Nhat Hoang Vo, Huynh Dai Nhan Tran
Organizations: UIT, VNU-HCM · TwoHearts Technologies Co., Ltd., Ho Chi Minh City, Vietnam · IU, VNU-HCM · University of Eastern Finland · MBZUAI · University of Jyväskylä
Vietnamese pre-trained encoders have improved rapidly, but the strongest ones are large, multilingual-vocabulary models or depend on external word segmentation, which makes them costly to deploy. We present GoiBERT, a family of two compact Vietnamese encoders with 26M and 49M parameters that operate directly on raw text. GoiBERT learns from two complementary teachers. A same-tokenizer Vietnamese encoder first teaches general-purpose representations. A much larger teacher then provides task-level guidance through soft labels that are generated without ever labeling its own training data. Across eight benchmarks covering text classification, aspect-based sentiment analysis, natural language inference, and named entity recognition, GoiBERT-large achieves the highest average score among non-teacher models. It does so with fewer than half the parameters of the strongest compact baseline and twice its throughput. GoiBERT-base retains 99% of that baseline's performance at a quarter of its size. Our analysis shows that compression is nearly lossless for sentiment and classification tasks but remains costly for inference and entity recognition, which points to clear directions for future work. Our code and models are available at: https://github.com/Nhantran65/GoiBERT.git
Figures & tables
Figure 1: Average score over eight Vietnamese benchmarks versus model size. Both GoiBERT variants lie on the accuracy–efficiency frontier of non-teacher models: GoiBERT-large has the highest average with less than half of BamiBERT’s parameters, and GoiBERT-base stays close with about a quarter of them.
Figure 2: Overview of the GoiBERT training pipeline. (1) General distillation: the student matches the hidden states and attention maps of a frozen BamiBERT, which shares its tokenizer, on unlabeled Vietnamese text. (2) Out-of-fold soft labels: five CafeBERT teachers, each fine-tuned on four folds, label the held-out fold, so that no example is labeled by a teacher that saw it. (3) Task fine-tuning: the distilled backbone is trained on gold labels and CafeBERT soft labels with layer-wise learning-rate decay.
Task
Metric
Cafe-
XLM-R †
ViSo-
PhoBERT
Bami-
GoiBERT
GoiBERT
BERT ⋄
BERT
-v2
BERT
-base
-large
ViNLI
M-F1
86.19
77.01
67.67
78.87
80.18
75.86
78.52
ViSpam
M-F1
77.22
76.55
77.10
77.86
77.55
76.52
78.13
PhoNER
Ent. F1
95.47
92.50
93.07
95.44 ‡
94.32
92.97
93.47
ABSA-Rest.
Pair
77.41
71.58
74.68
74.12
73.53
74.15
76.00
ABSA-Hotel
Pair
74.78
71.23
74.17
72.35
73.94
75.01
75.67
Table 1: Test results (%) on eight Vietnamese benchmarks, one primary metric per task, averaged over three seeds. M-F1: macro-F1; Pair: aspect–sentiment pair F1; Ent. F1: strict entity-level F1. Bold marks the best non-teacher model in each row. ⋄ Task teacher of GoiBERT (shown for reference). † Reported results. ‡ Uses word-segmented input. All 15 metrics and seed ranges appear in Appendices E – F .
Figure 3: Per-task difference from BamiBERT (primary metric). Gains concentrate on ABSA and topic classification, whereas NLI and NER carry the cost of compression, most visibly for the smaller GoiBERT-base.
Family
Bami
Goi-L
Cafe
ΔL−C
Classification (3)
79.60
80.19
79.98
+ 0.21
ABSA (3)
77.29
78.82
79.39
− 0.57
NER (1)
94.32
93.47
95.47
− 2.00
NLI (1)
80.18
78.52
86.19
− 7.67
Table 2: Primary scores averaged by task family. ΔL−C is the gap between GoiBERT-large and its 11 × larger task teacher CafeBERT.
Model
Weights
Throughput
VRAM
CPU lat.
BamiBERT
206 MB
2,397
801 MB
31.0 ms
Goi-large
99 MB
4,827 (2.0 × )
410 MB
19.4 ms
Goi-base
53 MB
8,285 (3.5 × )
248 MB
14.2 ms
Table 3: Inference efficiency of the encoders (bf16 weights). Throughput (sentences/s) and peak VRAM: L40S, batch 32, length 256. CPU latency: one 128-token input, eight threads, fp32. Throughput speed-ups relative to BamiBERT are in parentheses.
Setting
ABSA-R
VSFC-T
ViNLI
GoiBERT-large
CE only (no GD)
59.35
77.42
45.02
+ General KD
72.89 + 13.5
78.06 + 0.6
76.18 + 31.2
+ Task KD (OOF)
74.81 + 1.9
78.94 + 0.9
77.60 + 1.4
+ LLRD (full)
76.00 + 1.2
80.42 + 1.5
78.52 + 0.9
GoiBERT-base
Table 4: Cumulative ablation (test, mean of three seeds). ABSA-R: Restaurant pair F1; VSFC-T and ViNLI: macro-F1. Small numbers give the change from the previous row.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Detailed encoder architecture of GoiBERT-base and GoiBERT-large.
Task
Metric
Cafe-
XLM-R †
ViDe-
ViSo-
PhoBERT
Bami-
GoiBERT
GoiBERT
BERT
BERTa †
BERT
-v2
BERT
-base
-large
ViNLI
Acc.
86.08
76.83
61.08
67.55
78.75
80.07
75.75
78.39
M-F1
86.19
77.01
60.71
67.67
78.87
80.18
75.86
78.52
ViSpam
Acc.
90.50
90.16
86.21
90.78
90.82
90.19
89.88
90.75
M-F1
77.22
76.55
67.04
77.10
77.86
77.55
76.52
78.13
PhoNER
Ent. F1
95.47
92.50
94.50 ‡
93.07
95.44 ‡
94.32
92.97
93.47
Appendix
Table 5: Complete test results (%) across all 15 evaluation metrics. Bold indicates the best value in each row; lower is better for parameter counts and GFLOPs. † denotes reported results and ‡ denotes results requiring word-segmented input. Approximate values and lower bounds are marked by ∼ and ≥ , respectively. ViNLI uses the community mirror presencesw/vinli_4_label .
Task
Metric
BamiBERT
GoiBERT-large
ViNLI
Acc.
80.07 [79.94, 80.24]
78.39 [78.33, 78.47]
M-F1
80.18 [80.04, 80.33]
78.52 [78.46, 78.57]
ViSpam
Acc.
90.19 [89.98, 90.41]
90.75 [90.44, 91.07]
M-F1
77.55 [75.58, 78.62]
78.13 [77.78, 78.32]
PhoNER
Ent. F1
94.32 [94.24, 94.39]
93.47 [93.44, 93.48]
ABSA-Rest.
Det.
87.03 [86.45, 87.52]
88.14 [87.90, 88.39]
Appendix
Table 6: Test-score variation for BamiBERT and GoiBERT-large. Values are means with [min, max] across seeds 42, 43, and 44.
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
In this paper, we introduce BamiBERT, a new BERT-based pre-trained language model for Vietnamese that addresses key limitations of PhoBERT -- the current de facto Vietnamese text encoder. Trained from scratch on a 129GB corpus of general-domain Vietnamese text for 20 epochs, BamiBERT supports an extended context length of up to 2048 tokens and operates directly on raw input, eliminating the need for external word segmentation. Across 8 Vietnamese benchmarks, it achieves the best score on 11 of 15 metrics and the second-best on 3 others, setting a new state of the art among "base"-sized Vietnamese encoders and demonstrating strong cross-domain generalization. We release BamiBERT at: https://huggingface.co/Qualcomm-AI-Research/BamiBERT
Encoder-only transformer models remain essential for production NLP pipelines. We introduce moBERTo, a Portuguese adaptation of ModernBERT obtained through continued pretraining of the ModernBERT-base checkpoint on 60 billion tokens (5 epochs over a 12-billion-token corpus curated from FineWeb2 and filtered with educational and STEM classifiers). We preserve the original architecture, including rotary positional embeddings, alternating local-global attention, flash attention, and unpadding. We evaluate moBERTo across information retrieval (including long-context retrieval at up to 8,192 tokens), document classification, named entity recognition, and natural language understanding. Our best variant, which combines a Portuguese tokenizer with subword-matching embedding transfer and long-context post-training, achieves the highest average reranking nDCG@10 across three Portuguese retrieval benchmarks and the best results on PLUE-PT. Through ablation studies, we show that (i) continued pretraining is strongly preferable to training from scratch, particularly for preserving long-context capabilities; (ii) tokenizer adaptation improves token-level tasks but degrades long-context retrieval; (iii) a dedicated long-context post-training phase at 8,192 tokens further improves reranking and NER; and (iv) encoder-only architectures remain competitive with larger decoder-only alternatives for discriminative tasks. We publicly release the model weights at https://huggingface.co/Tropic-AI/moBERTo and training data at https://huggingface.co/datasets/Tropic-AI/moberto-pretraining-dataset-c4-compatible on Hugging Face.
Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos +1
UNICAMP, Campinas, Brazil · Tropic AI · Maritaca AI
Encoder-only Transformers remain effective for discriminative and representation-learning tasks, yet Polish encoders still largely rely on BERT/RoBERTa-style architectures. We introduce Polish ModernBERT, a family of four Polish encoders available at Base and Large scales, each with 512-token and 8K context variants. We adapt the ModernBERT pretraining recipe through staged selection experiments and release a long-context benchmark covering legal topic classification, ideological decision-direction prediction, factual-consistency assessment over literary plot summaries, and human-rights violation assessment. Across 30 tasks, Polish ModernBERT achieves the best overall performance among the evaluated Polish encoders, reaching 83.99 and 85.11 for the Base-8K and Large-8K models, respectively. On long-context tasks, the 8K variants improve over matched Polish RoBERTa-8K baselines from 67.47 to 77.15 and from 75.88 to 78.49 at the Base and Large scales, respectively. The Base-8K model achieves this gain with 22% fewer parameters (149M vs.\ 190M). Efficiency measurements in representative inference setups show lower peak memory usage and latency than matched Polish RoBERTa baselines in both 512-token and 8K settings. Polish ModernBERT-8K-Base additionally achieves the best result on a Polish retrieval benchmark among the evaluated encoders below 300M parameters.
Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1
National Information Processing Institute, Warsaw, Poland