How Well Do Decision Models Transfer to Vietnamese? An Empirical Study of Text Classification

Phat Huu Ha; Nguyen Xuan Phuc; Nong Nhut Quy; Trinh Tran Tran; Nguyen Minh Triet; Dang Van Thin

cs.CLOct 22, 2026

How Well Do Decision Models Transfer to Vietnamese? An Empirical Study of Text Classification

Authors: Phat Huu Ha, Nguyen Xuan Phuc, Nong Nhut Quy, Trinh Tran Tran, Nguyen Minh Triet, Dang Van Thin

Organizations: University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University HoChiMinh City, Vietnam

Abstract

Recent advances in specialized decision models have expanded the possibilities for zero-shot classification, yet their effectiveness on Vietnamese remains underexplored. This study evaluates four specialized decision models across 13 classification settings spanning six task groups, examining zero-shot performance, sensitivity to label-description language, and robustness to Vietnamese linguistic variations. The evaluation considers English, Vietnamese, and bilingual label descriptions, alongside common linguistic perturbations involving teencode, typographical errors, and diacritic removal. Under English label descriptions, JEV achieves the strongest overall performance, ranking first in 11 classification settings. The effects of label-description language vary across models and datasets, with bilingual descriptions providing no consistent improvement. Robustness experiments reveal that diacritic removal causes the most substantial performance degradation, reducing mean macro- $F_1$ by 9.51--15.47 points across model configurations, considerably more than teencode or typographical errors. These findings highlight the importance of label-description choices and Vietnamese orthographic variation when evaluating specialized decision models for zero-shot text classification.

Figures & tables

Topic classification
Dataset	LAYA			GLiNER			Quyet			JEV			Fine-tuning BERT
Dataset	Acc	WF1	MF1	Acc	WF1	MF1	Acc	WF1	MF1	Acc	WF1	MF1	Acc	WF1	MF1
NEU-ESC	26.49	30.02	14.79	62.30	58.80	27.10	47.19	49.19	23.08	51.79	55.83	37.31	78.65	–	62.73
UIT-VSFC	49.15	57.08	46.02	44.91	55.03	48.99	82.56	81.86	63.63	83.77	84.61	74.36	88.22	–	78.08
Emotion recognition
UIT-VSMEC	34.34	35.80	31.89	18.76	6.13	5.10	56.28	56.59	51.76	55.70	54.58	49.99	64.65	63.80	59.06

Table 1: Zero-shot classification performance (%) using English label descriptions and a supervised fine-tuned PhoBERT baseline. Acc, WF1, and MF1 denote accuracy, weighted-

F_{1}

, and macro-

F_{1}

. Bold and underlined MF1 indicate the highest and second-highest reported scores. Dashes denote unavailable values. Means cover the 12 settings shared by all four zero-shot models.

Figure 1: Zero-shot macro-

F_{1}

(%) of specialized decision models and general-purpose LLMs across 11 Vietnamese classification settings.

Figure 2: Comparison of JEV and Quyet confusion matrices on UIT-VSMEC.

Topic classification
Dataset	LAYA		GLiNER		Quyet		JEV
Dataset	VI	EN+VI	VI	EN+VI	VI	EN+VI	VI	EN+VI
NEU-ESC	$+6.06$	$-4.07$	$-1.37$	$+1.32$	$+3.63$	$-3.18$	$-0.67$	$-0.32$
UIT-VSFC	$-5.24$	$-15.08$	$-11.65$	$-13.68$	$+2.96$	$-1.42$	$+0.26$	$+0.84$
Emotion recognition
UIT-VSMEC	$-2.36$	$-4.83$	$+0.57$	$+0.59$	$+3.38$	$-11.86$	$+1.42$	$+1.55$

Table 2: Changes in macro-

F_{1}

relative to English label descriptions (percentage points).

Perturbation	Changed (%)	Edit burden (%)
Teencode	61.70	2.14
Typo	90.18	4.93
No accents	98.10	18.28

Table 3: Perturbation coverage and character edit burden across the 13 evaluation settings.

Topic classification
Dataset	LAYA			GLiNER			Quyet			JEV
Dataset	NoAcc	Teen	Typo	NoAcc	Teen	Typo	NoAcc	Teen	Typo	NoAcc	Teen	Typo
NEU-ESC	$-9.93$	$-0.31$	$-3.42$	$-12.74$	$-0.10$	$-2.30$	$-9.68$	$+0.02$	$-1.38$	$-2.48$	$-0.38$	$-0.62$
UIT-VSFC	$-27.08$	$-0.59$	$-4.31$	$-46.60$	$+0.50$	$-7.86$	$-29.42$	$+0.08$	$-4.08$	$-13.22$	$+0.36$	$-3.36$
Emotion recognition
UIT-VSMEC	$-15.85$	$-1.74$	$-6.85$	$-0.62$	$-0.62$	$+0.59$	$-26.38$	$-4.93$	$-9.80$	$-18.64$	$-5.13$	$-6.43$

Table 4: Changes in macro-

F_{1}

under linguistic perturbations relative to original inputs (percentage points).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Dataset	Description change	Flips / $n$ (%)	W $\rightarrow$ C	C $\rightarrow$ W	W $\rightarrow$ W	$\Delta$ MF1
LAYA	ViMQ	VI $\rightarrow$ EN+VI	760 / 1,000 (76.00)	88	393	279	$-16.33$
GLiNER	ViNLI	EN $\rightarrow$ VI	2,266 / 2,991 (75.76)	580	699	987	$-13.85$
Quyet	ViNLI	EN $\rightarrow$ VI	930 / 2,991 (31.09)	682	151	97	$+21.06$
Quyet	ViHSD	EN $\rightarrow$ VI	1,555 / 6,680 (23.28)	147	1,247	161	$-14.23$
JEV	ViNLI	EN $\rightarrow$ VI	123 / 2,991 (4.11)	84	28	11	$+1.50$

Appendix

Table 5: Selected instance-level prediction changes across label-description configurations.

Model	Flips	W $\rightarrow$ C	C $\rightarrow$ W	W $\rightarrow$ W
LAYA	54.29	12.15	27.95	14.19
GLiNER	19.83	3.33	12.29	4.21
Quyet	40.60	8.13	23.33	9.15
JEV	19.57	4.21	11.73	3.63

Appendix

Table 6: Instance-level prediction transitions under diacritic removal (%).

	Contradiction (%)		Accuracy (%)
Model	Original	Teencode	Original	Teencode
LAYA	52.06	31.44	67.01	44.85
GLiNER	23.71	9.28	35.05	24.23
Quyet	58.76	56.70	74.23	70.10
JEV	50.00	46.91	82.99	78.35

Appendix

Table 7: Effects of rewriting không on NLI predictions (194 paired instances).

Token-level operations
Operation / parameter	Teencode	Typo
Slang substitution	0.80	–
Telex-related errors	–	0.08
Full diacritic removal	–	0.05
Partial diacritic removal	–	0.03
Confusion-set substitution	–	0.02

Appendix

Table 8: Stochastic perturbation generator parameters.

Topic classification
Dataset	$\boldsymbol{n}$	Teencode		Typo		No accents
Dataset	$\boldsymbol{n}$	Changed	Edit	Changed	Edit	Changed	Edit
NEU-ESC	6,613	68.30	3.57	90.20	7.42	99.17	20.42
UIT-VSFC	3,166	42.61	2.20	90.24	7.49	99.87	20.44
Emotion recognition
UIT-VSMEC	693	71.72	5.05	87.73	7.73	99.86	21.62

Appendix

Table 9: Perturbation coverage and character edit burden by evaluation setting (%).

Topic classification
Dataset	Large Language Models					Decision Models
Dataset	Gemma-3	Qwen3	Llama-3.3	GPT-4o	GPT-4.1	LAYA	GLiNER	Quyet	JEV
NEU-ESC	37.32	32.12	36.49	33.24	34.15	14.79	27.10	23.08	37.31
UIT-VSFC	73.60	70.27	73.11	68.99	77.07	46.02	48.99	63.63	74.36
Emotion recognition
UIT-VSMEC	52.10	49.98	46.39	56.51	57.65	31.89	5.10	51.76	49.99

Appendix

Table 10: Zero-shot macro-

F_{1}

on 11 overlapping classification settings. Decision models use English label descriptions, while LLM results are taken from prior zero-shot prompting experiments. Bold and underlined values indicate the highest and second-highest reported scores per setting, respectively. Unreported results are denoted by –.

Zero-Shot Text Classification

Zero-shot text classification assigns texts to categories without task-specific labeled training data, using language models, embeddings, or label descriptions.

See all 35 papers

VietBinoculars: A Zero-Shot Approach for Detecting Vietnamese LLM-Generated Text
Sep 30, 2025Trieu Hai Nguyen, Sivaswamy AkileshAI-Generated Text Detection Language Model Robustness
Faculty of Information Technology, Nha Trang University, 02 Nguyen Dinh Chieu Street, North Nha Trang 57100, Vietnam · Faculty of Business Administration, Swiss School of Business and Management Geneva, 12 Avenue des Morgines, Geneva 1213, Switzerland
How Robust Are LLMs to Vietnamese Dialects?
Aug 11, 2026Minh Tran, Trinh Chau, Thanh-Nhan Le +4LLM Evaluation Language Model Robustness
University of Science, VNU-HCM · VNU University of Engineering and Technology · New York University Abu Dhabi +4
Chinese-Jev: Bringing System One Model to Chinese-Language Tasks
Sep 29, 2026Zexiao Wang, Zihao Zhang, Xudong Wang +5LLM Decision-Making Efficient Language Model Inference
Fudan University · National University of Singapore · University of Chinese Academy of Sciences

Sep 30, 2025J/K move · Enter open · S save

VietBinoculars: A Zero-Shot Approach for Detecting Vietnamese LLM-Generated Text

Trieu Hai Nguyen, Sivaswamy Akilesh

Faculty of Information Technology, Nha Trang University, 02 Nguyen Dinh Chieu Street, North Nha Trang 57100, Vietnam · Faculty of Business Administration, Swiss School of Business and Management Geneva, 12 Avenue des Morgines, Geneva 1213, Switzerland

AI-Generated Text Detection Language Model Robustness Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

The rapid proliferation of Large Language Models has intensified the challenge of distinguishing LLM-generated text from human writing in non-English languages. This study introduces VietBinoculars, a zero-shot detection framework coupling PhoGPT-4B observer and performer models with calibrated global decision thresholds. By utilizing specialized Vietnamese BPE tokenization, the method eliminates byte-level fragmentation and probability dilution common in massive multilingual backbones. Evaluated across multi-domain benchmarks, VietBinoculars achieves an area under the ROC curve exceeding 0.99. Under optimal Youden's J thresholds and greedy decoding, detection accuracy reaches at least 98.78%, while significantly outperforming baseline Binoculars, zero-shot detectors, and commercial tools on creative Capybara prompts. Even under a strict false positive rate constraint of 0.06%, the detector maintains F1-scores between 83.15% and 94.70%. Detection performance consistently improves with sequence length, stabilizing at optimal accuracy for passages containing 450 to 550 tokens. Extended stress testing across 48 distinct model-decoding configurations and three post-generation rewriting strategies delineates practical operational boundaries. VietBinoculars exhibits robust resilience against single-pass paraphrasing and human-style revisions, but experiences notable performance degradation under high-entropy sampling and iterative double paraphrasing.

Abstract

Zero-Shot Text Classification

Explore similar work

VietBinoculars: A Zero-Shot Approach for Detecting Vietnamese LLM-Generated Text

How Robust Are LLMs to Vietnamese Dialects?

Chinese-Jev: Bringing System One Model to Chinese-Language Tasks