PhoPunct: A Large-Scale Benchmark of Transformer Architectures for Vietnamese Punctuation Restoration
Authors: Khoa D. Nguyen, Thai Bao Huynh, Binh T. Nguyen
Organizations: AISIA Research Lab, Ho Chi Minh City, Vietnam · University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Information Technology, Ho Chi Minh City, Vietnam
Vietnamese punctuation restoration is often evaluated on relatively small benchmarks, limiting conclusions about how architectural choices scale with data volume. We present a controlled benchmark on News and Novels corpora containing 22.1 million labeled tokens (19.9M News and 2.2M Novels). We evaluate 17 architectures formed from four pretrained Transformer backbones and optional bidirectional long short-term memory (Bi-LSTM) and conditional random field (CRF) layers, together with a PhoBERT-large-Bi-LSTM-CRF model, PhoPunct. Every model is trained for eight epochs with the same learning rate and without architecture-specific tuning. Under this fixed budget, adding Bi-LSTM and CRF layers does not reliably improve macro F1: in seven of eight matched domain-backbone comparisons, a plain Transformer classifier matches or outperforms its Bi-LSTM-CRF counterpart. PhoPunct obtains the best News result (61.35), whereas XLM-R with a CRF obtains the best Novels result (50.18), exceeding PhoPunct (48.97). These results show that the value of recurrent and structured-prediction heads depends on their optimization regime and motivate explicit separation between architectural improvements and improvements due to architecture-specific tuning. Our source code is available at: https://github.com/aisia-research-lab/PhoPunct
Figures & tables
Study
Data scale
Focus
Shared budget
Matched heads
Pham et al. (2019)
N/A
CRF baseline
–
✗
Pham et al. (2020)
22.1M tokens
Seven-label DNN
✗
✗
Tran et al. (2021)
∼ 742K sentences
Transformer heads
✗
∼
Uyen et al. (2022)
425,350 sentences
Case + punctuation
✗
✗
Ours
22.1M tokens
4×4+1 grid
✓
✓
Table 1: Vietnamese punctuation-restoration studies. ✓ : criterion satisfied; ✗ : not satisfied or not reported; ∼ : partial comparison; –: not applicable (columns indicate a shared optimization budget across compared architectures and a matched Plain/Bi-LSTM/CRF/Bi-LSTM–CRF head set, respectively). Data scale is reported in each study’s own unit (sentences or tokens) and is not directly comparable across rows. Pham et al. (2020) and Ours report identical token counts because both use the same underlying files.
Domain
Split
Tokens
News
Train
11,939,762
Validation
4,003,912
Test
3,958,621
Total
19,902,295
Novels
Train
1,350,500
Validation
358,791
Table 2: Dataset scale by domain and split.
Figure 1: Validation workflow for the inherited corpus. The study checks the distributed format and label alignment; it does not claim to recreate or re-annotate the original data.
Figure 2: Punctuation-label counts by domain over the complete train, validation, and test splits. Counts are displayed on a logarithmic scale.
Figure 3: Matched prediction-head framework (dashed boxes denote optional components). Each backbone produces one contextual representation per word through first-subword gathering. The Plain, Bi-LSTM, CRF, and Bi-LSTM–CRF configurations use the same classifier-module parameterization across the four base-sized backbones; PhoBERT-large supplies the backbone for PhoPunct.
Hyperparameter
Value
Train/eval batch size
16 / 32
Learning rate
2×10−5
Optimizer
AdamW
Warmup ratio
0.06
Epochs
8
Maximum length
256 subwords
Table 3: Training hyperparameters shared by all architectures.
Figure 4: Macro F1 for the controlled backbone–head grid. The same four head configurations are compared within each base-sized backbone; PhoPunct is reported separately in Table 4 because it uses PhoBERT-large.
ID
Model
News
Novels
1
mBERT
53.71
43.84
2
vELECTRA
53.80
40.70
3
ViBERT
57.47
49.82
4
XLM-R
58.25
49.79
5
mBERT + Bi-LSTM
53.90
42.56
6
vELECTRA + Bi-LSTM
53.42
40.46
Table 4: Macro F1 (%) across all 17 architectures. Bold indicates the best result per domain.
Backbone
Domain
Plain
B+CRF
Δ
mBERT
News
53.71
53.54
+0.17
mBERT
Novels
43.84
42.37
+1.47
vELECTRA
News
53.80
53.45
+0.35
vELECTRA
Novels
40.70
40.22
+0.48
ViBERT
News
57.47
57.66
−0.19
ViBERT
Novels
49.82
45.91
+3.91
Table 5: Matched Plain and Bi-LSTM–CRF (B+CRF) results. Δ is Plain minus B+CRF macro F1.
Figure 5: PhoPunct architecture and decoding flow.
Figure 6: Diagnostic evidence. Left: per-class F1 for the two principal systems by domain, excluding O . Right: Plain minus Bi-LSTM–CRF macro F1; positive values favor Plain.