cs.CLNov 6, 2026

PhoPunct: A Large-Scale Benchmark of Transformer Architectures for Vietnamese Punctuation Restoration

Authors: Khoa D. Nguyen, Thai Bao Huynh, Binh T. Nguyen

Organizations: AISIA Research Lab, Ho Chi Minh City, Vietnam · University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Information Technology, Ho Chi Minh City, Vietnam

Abstract

Vietnamese punctuation restoration is often evaluated on relatively small benchmarks, limiting conclusions about how architectural choices scale with data volume. We present a controlled benchmark on News and Novels corpora containing 22.1 million labeled tokens (19.9M News and 2.2M Novels). We evaluate 17 architectures formed from four pretrained Transformer backbones and optional bidirectional long short-term memory (Bi-LSTM) and conditional random field (CRF) layers, together with a PhoBERT-large-Bi-LSTM-CRF model, PhoPunct. Every model is trained for eight epochs with the same learning rate and without architecture-specific tuning. Under this fixed budget, adding Bi-LSTM and CRF layers does not reliably improve macro F1: in seven of eight matched domain-backbone comparisons, a plain Transformer classifier matches or outperforms its Bi-LSTM-CRF counterpart. PhoPunct obtains the best News result (61.35), whereas XLM-R with a CRF obtains the best Novels result (50.18), exceeding PhoPunct (48.97). These results show that the value of recurrent and structured-prediction heads depends on their optimization regime and motivate explicit separation between architectural improvements and improvements due to architecture-specific tuning. Our source code is available at: https://github.com/aisia-research-lab/PhoPunct

Figures & tables

Explore similar work

CardsList
  1. PersianPunc: A Large-Scale Dataset and BERT-Based Approach for Persian Punctuation Restoration

    Mar 5, 2026Mohammad Javad Ranjbar Kalahroodi, Heshaam Faili, Azadeh ShakeryBidirectional Encoder Representations from TransformersLow-Resource Language Processing

  2. BamiBERT: A New BERT-based Language Model for Vietnamese

    Jul 2, 2026Dat Quoc Nguyen, Thinh Pham, Chi Tran +1Language Model PretrainingTransformer

  3. Efficient Punctuation Restoration via Weighted Lookahead Scoring Method for Streaming ASR Systems

    Apr 18, 2026Sungmook Woo, Hyungu Kang, Chanwoo KimStreaming ASR