PhoPunct: A Large-Scale Benchmark of Transformer Architectures for Vietnamese Punctuation Restoration
Authors: Khoa D. Nguyen, Thai Bao Huynh, Binh T. Nguyen
Organizations: AISIA Research Lab, Ho Chi Minh City, Vietnam · University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Information Technology, Ho Chi Minh City, Vietnam
Vietnamese punctuation restoration is often evaluated on relatively small benchmarks, limiting conclusions about how architectural choices scale with data volume. We present a controlled benchmark on News and Novels corpora containing 22.1 million labeled tokens (19.9M News and 2.2M Novels). We evaluate 17 architectures formed from four pretrained Transformer backbones and optional bidirectional long short-term memory (Bi-LSTM) and conditional random field (CRF) layers, together with a PhoBERT-large-Bi-LSTM-CRF model, PhoPunct. Every model is trained for eight epochs with the same learning rate and without architecture-specific tuning. Under this fixed budget, adding Bi-LSTM and CRF layers does not reliably improve macro F1: in seven of eight matched domain-backbone comparisons, a plain Transformer classifier matches or outperforms its Bi-LSTM-CRF counterpart. PhoPunct obtains the best News result (61.35), whereas XLM-R with a CRF obtains the best Novels result (50.18), exceeding PhoPunct (48.97). These results show that the value of recurrent and structured-prediction heads depends on their optimization regime and motivate explicit separation between architectural improvements and improvements due to architecture-specific tuning. Our source code is available at: https://github.com/aisia-research-lab/PhoPunct
Figures & tables
Study
Data scale
Focus
Shared budget
Matched heads
Pham et al. (2019)
N/A
CRF baseline
–
✗
Pham et al. (2020)
22.1M tokens
Seven-label DNN
✗
✗
Tran et al. (2021)
∼ 742K sentences
Transformer heads
✗
∼
Uyen et al. (2022)
425,350 sentences
Case + punctuation
✗
✗
Ours
22.1M tokens
4×4+1 grid
✓
✓
Table 1: Vietnamese punctuation-restoration studies. ✓ : criterion satisfied; ✗ : not satisfied or not reported; ∼ : partial comparison; –: not applicable (columns indicate a shared optimization budget across compared architectures and a matched Plain/Bi-LSTM/CRF/Bi-LSTM–CRF head set, respectively). Data scale is reported in each study’s own unit (sentences or tokens) and is not directly comparable across rows. Pham et al. (2020) and Ours report identical token counts because both use the same underlying files.
Domain
Split
Tokens
News
Train
11,939,762
Validation
4,003,912
Test
3,958,621
Total
19,902,295
Novels
Train
1,350,500
Validation
358,791
Table 2: Dataset scale by domain and split.
Figure 1: Validation workflow for the inherited corpus. The study checks the distributed format and label alignment; it does not claim to recreate or re-annotate the original data.
Figure 2: Punctuation-label counts by domain over the complete train, validation, and test splits. Counts are displayed on a logarithmic scale.
Figure 3: Matched prediction-head framework (dashed boxes denote optional components). Each backbone produces one contextual representation per word through first-subword gathering. The Plain, Bi-LSTM, CRF, and Bi-LSTM–CRF configurations use the same classifier-module parameterization across the four base-sized backbones; PhoBERT-large supplies the backbone for PhoPunct.
Hyperparameter
Value
Train/eval batch size
16 / 32
Learning rate
2×10−5
Optimizer
AdamW
Warmup ratio
0.06
Epochs
8
Maximum length
256 subwords
Table 3: Training hyperparameters shared by all architectures.
Figure 4: Macro F1 for the controlled backbone–head grid. The same four head configurations are compared within each base-sized backbone; PhoPunct is reported separately in Table 4 because it uses PhoBERT-large.
ID
Model
News
Novels
1
mBERT
53.71
43.84
2
vELECTRA
53.80
40.70
3
ViBERT
57.47
49.82
4
XLM-R
58.25
49.79
5
mBERT + Bi-LSTM
53.90
42.56
6
vELECTRA + Bi-LSTM
53.42
40.46
Table 4: Macro F1 (%) across all 17 architectures. Bold indicates the best result per domain.
Backbone
Domain
Plain
B+CRF
Δ
mBERT
News
53.71
53.54
+0.17
mBERT
Novels
43.84
42.37
+1.47
vELECTRA
News
53.80
53.45
+0.35
vELECTRA
Novels
40.70
40.22
+0.48
ViBERT
News
57.47
57.66
−0.19
ViBERT
Novels
49.82
45.91
+3.91
Table 5: Matched Plain and Bi-LSTM–CRF (B+CRF) results. Δ is Plain minus B+CRF macro F1.
Figure 5: PhoPunct architecture and decoding flow.
Figure 6: Diagnostic evidence. Left: per-class F1 for the two principal systems by domain, excluding O . Right: Plain minus Bi-LSTM–CRF macro F1; positive values favor Plain.
Punctuation restoration is essential for improving the readability and downstream utility of automatic speech recognition (ASR) outputs, yet remains underexplored for Persian despite its importance. We introduce PersianPunc, a large-scale, high-quality dataset of 17 million samples for Persian punctuation restoration, constructed through systematic aggregation and filtering of existing textual resources. We formulate punctuation restoration as a token-level sequence labeling task and fine-tune ParsBERT to achieve strong performance. Through comparative evaluation, we demonstrate that while large language models can perform punctuation restoration, they suffer from critical limitations: over-correction tendencies that introduce undesired edits beyond punctuation insertion (particularly problematic for speech-to-text pipelines) and substantially higher computational requirements. Our lightweight BERT-based approach achieves a macro-averaged F1 score of 91.33% on our test set while maintaining efficiency suitable for real-time applications. We make our dataset (https://huggingface.co/datasets/MohammadJRanjbar/persian-punctuation-restoration) and model (https://huggingface.co/MohammadJRanjbar/parsbert-persian-punctuation) publicly available to facilitate future research in Persian NLP and provide a scalable framework applicable to other morphologically rich, low-resource languages.
Mohammad Javad Ranjbar Kalahroodi, Heshaam Faili, Azadeh Shakery
University of Tehran, Tehran, Iran · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran
In this paper, we introduce BamiBERT, a new BERT-based pre-trained language model for Vietnamese that addresses key limitations of PhoBERT -- the current de facto Vietnamese text encoder. Trained from scratch on a 129GB corpus of general-domain Vietnamese text for 20 epochs, BamiBERT supports an extended context length of up to 2048 tokens and operates directly on raw input, eliminating the need for external word segmentation. Across 8 Vietnamese benchmarks, it achieves the best score on 11 of 15 metrics and the second-best on 3 others, setting a new state of the art among "base"-sized Vietnamese encoders and demonstrating strong cross-domain generalization. We release BamiBERT at: https://huggingface.co/Qualcomm-AI-Research/BamiBERT
Punctuation restoration improves ASR (Automatic Speech Recognition) readability. However streaming ASR requires online decisions with limited future context. In streaming ASR, the system predicts punctuation incrementally, which makes generation-based approaches prone to latency and alignment failures under boundary-wise evaluation. This paper proposes a non-autoregressive scoring method (no free-form generation) that preserves the input transcript and makes a decision at each word boundary. Our method compares punctuation insertion hypotheses against a no-insertion baseline under a bounded K-subword-token lookahead, and calibrates decisions using a weight α and a validation-calibrated threshold τ (no parameter updates during inference). On IWSLT 2017, our scoring method achieves a 4-class macro F1 of 0.893 in the no fine-tuning setting (validation-calibrated, K=2) and 0.937 after fine-tuning (K=2), outperforming the prompt-based baseline (0.566) and a fine-tuned ELECTRA baseline (0.913) under the same lookahead budget. We analyze the impact of the lookahead budget through ablation studies on K.
Sungmook Woo, Hyungu Kang, Chanwoo Kim
Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea