GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation
Authors: Quang Anh Kiet Tran, Nam Anh Tran, Manh Hung Trinh, Thanh Danh Le, Khang Nhat Hoang Vo, Huynh Dai Nhan Tran
Organizations: UIT, VNU-HCM · TwoHearts Technologies Co., Ltd., Ho Chi Minh City, Vietnam · IU, VNU-HCM · University of Eastern Finland · MBZUAI · University of Jyväskylä
Vietnamese pre-trained encoders have improved rapidly, but the strongest ones are large, multilingual-vocabulary models or depend on external word segmentation, which makes them costly to deploy. We present GoiBERT, a family of two compact Vietnamese encoders with 26M and 49M parameters that operate directly on raw text. GoiBERT learns from two complementary teachers. A same-tokenizer Vietnamese encoder first teaches general-purpose representations. A much larger teacher then provides task-level guidance through soft labels that are generated without ever labeling its own training data. Across eight benchmarks covering text classification, aspect-based sentiment analysis, natural language inference, and named entity recognition, GoiBERT-large achieves the highest average score among non-teacher models. It does so with fewer than half the parameters of the strongest compact baseline and twice its throughput. GoiBERT-base retains 99% of that baseline's performance at a quarter of its size. Our analysis shows that compression is nearly lossless for sentiment and classification tasks but remains costly for inference and entity recognition, which points to clear directions for future work. Our code and models are available at: https://github.com/Nhantran65/GoiBERT.git
Figures & tables
Figure 1: Average score over eight Vietnamese benchmarks versus model size. Both GoiBERT variants lie on the accuracy–efficiency frontier of non-teacher models: GoiBERT-large has the highest average with less than half of BamiBERT’s parameters, and GoiBERT-base stays close with about a quarter of them.
Figure 2: Overview of the GoiBERT training pipeline. (1) General distillation: the student matches the hidden states and attention maps of a frozen BamiBERT, which shares its tokenizer, on unlabeled Vietnamese text. (2) Out-of-fold soft labels: five CafeBERT teachers, each fine-tuned on four folds, label the held-out fold, so that no example is labeled by a teacher that saw it. (3) Task fine-tuning: the distilled backbone is trained on gold labels and CafeBERT soft labels with layer-wise learning-rate decay.
Task
Metric
Cafe-
XLM-R †
ViSo-
PhoBERT
Bami-
GoiBERT
GoiBERT
BERT ⋄
BERT
-v2
BERT
-base
-large
ViNLI
M-F1
86.19
77.01
67.67
78.87
80.18
75.86
78.52
ViSpam
M-F1
77.22
76.55
77.10
77.86
77.55
76.52
78.13
PhoNER
Ent. F1
95.47
92.50
93.07
95.44 ‡
94.32
92.97
93.47
ABSA-Rest.
Pair
77.41
71.58
74.68
74.12
73.53
74.15
76.00
ABSA-Hotel
Pair
74.78
71.23
74.17
72.35
73.94
75.01
75.67
Table 1: Test results (%) on eight Vietnamese benchmarks, one primary metric per task, averaged over three seeds. M-F1: macro-F1; Pair: aspect–sentiment pair F1; Ent. F1: strict entity-level F1. Bold marks the best non-teacher model in each row. ⋄ Task teacher of GoiBERT (shown for reference). † Reported results. ‡ Uses word-segmented input. All 15 metrics and seed ranges appear in Appendices E – F .
Figure 3: Per-task difference from BamiBERT (primary metric). Gains concentrate on ABSA and topic classification, whereas NLI and NER carry the cost of compression, most visibly for the smaller GoiBERT-base.
Family
Bami
Goi-L
Cafe
ΔL−C
Classification (3)
79.60
80.19
79.98
+ 0.21
ABSA (3)
77.29
78.82
79.39
− 0.57
NER (1)
94.32
93.47
95.47
− 2.00
NLI (1)
80.18
78.52
86.19
− 7.67
Table 2: Primary scores averaged by task family. ΔL−C is the gap between GoiBERT-large and its 11 × larger task teacher CafeBERT.
Model
Weights
Throughput
VRAM
CPU lat.
BamiBERT
206 MB
2,397
801 MB
31.0 ms
Goi-large
99 MB
4,827 (2.0 × )
410 MB
19.4 ms
Goi-base
53 MB
8,285 (3.5 × )
248 MB
14.2 ms
Table 3: Inference efficiency of the encoders (bf16 weights). Throughput (sentences/s) and peak VRAM: L40S, batch 32, length 256. CPU latency: one 128-token input, eight threads, fp32. Throughput speed-ups relative to BamiBERT are in parentheses.
Setting
ABSA-R
VSFC-T
ViNLI
GoiBERT-large
CE only (no GD)
59.35
77.42
45.02
+ General KD
72.89 + 13.5
78.06 + 0.6
76.18 + 31.2
+ Task KD (OOF)
74.81 + 1.9
78.94 + 0.9
77.60 + 1.4
+ LLRD (full)
76.00 + 1.2
80.42 + 1.5
78.52 + 0.9
GoiBERT-base
Table 4: Cumulative ablation (test, mean of three seeds). ABSA-R: Restaurant pair F1; VSFC-T and ViNLI: macro-F1. Small numbers give the change from the previous row.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Detailed encoder architecture of GoiBERT-base and GoiBERT-large.
Task
Metric
Cafe-
XLM-R †
ViDe-
ViSo-
PhoBERT
Bami-
GoiBERT
GoiBERT
BERT
BERTa †
BERT
-v2
BERT
-base
-large
ViNLI
Acc.
86.08
76.83
61.08
67.55
78.75
80.07
75.75
78.39
M-F1
86.19
77.01
60.71
67.67
78.87
80.18
75.86
78.52
ViSpam
Acc.
90.50
90.16
86.21
90.78
90.82
90.19
89.88
90.75
M-F1
77.22
76.55
67.04
77.10
77.86
77.55
76.52
78.13
PhoNER
Ent. F1
95.47
92.50
94.50 ‡
93.07
95.44 ‡
94.32
92.97
93.47
Appendix
Table 5: Complete test results (%) across all 15 evaluation metrics. Bold indicates the best value in each row; lower is better for parameter counts and GFLOPs. † denotes reported results and ‡ denotes results requiring word-segmented input. Approximate values and lower bounds are marked by ∼ and ≥ , respectively. ViNLI uses the community mirror presencesw/vinli_4_label .
Task
Metric
BamiBERT
GoiBERT-large
ViNLI
Acc.
80.07 [79.94, 80.24]
78.39 [78.33, 78.47]
M-F1
80.18 [80.04, 80.33]
78.52 [78.46, 78.57]
ViSpam
Acc.
90.19 [89.98, 90.41]
90.75 [90.44, 91.07]
M-F1
77.55 [75.58, 78.62]
78.13 [77.78, 78.32]
PhoNER
Ent. F1
94.32 [94.24, 94.39]
93.47 [93.44, 93.48]
ABSA-Rest.
Det.
87.03 [86.45, 87.52]
88.14 [87.90, 88.39]
Appendix
Table 6: Test-score variation for BamiBERT and GoiBERT-large. Values are means with [min, max] across seeds 42, 43, and 44.
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.