ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks

Tin Van Huynh; Kiet Van Nguyen; Ngan Luu-Thuy Nguyen

cs.CLMar 22, 2026

ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks

Authors: Tin Van Huynh, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen

Organizations: University of Information Technology, Ho Chi Minh City, Vietnam. · Vietnam National University, Ho Chi Minh City, Vietnam.

Abstract

High-quality text representations are crucial for natural language understanding (NLU), but low-resource languages like Vietnamese face challenges due to limited annotated data. While pre-trained models like PhoBERT and CafeBERT perform well, their effectiveness is constrained by data scarcity. Contrastive learning (CL) has recently emerged as a promising approach for improving sentence representations, enabling models to effectively distinguish between semantically similar and dissimilar sentences. We propose ViCLSR (Vietnamese Contrastive Learning for Sentence Representations), a novel supervised contrastive learning framework specifically designed to optimize sentence embeddings for Vietnamese, leveraging existing natural language inference (NLI) datasets. Additionally, we propose a process to adapt existing Vietnamese datasets for supervised learning, ensuring compatibility with CL methods. Our experiments demonstrate that ViCLSR significantly outperforms the powerful monolingual pre-trained model PhoBERT on five benchmark NLU datasets such as ViNLI (+6.97% F1), ViWikiFC (+4.97% F1), ViFactCheck (+9.02% F1), UIT-ViCTSD (+5.36% F1), and ViMMRC2.0 (+4.33% Accuracy). ViCLSR shows that supervised contrastive learning can effectively address resource limitations in Vietnamese NLU tasks and improve sentence representation learning for low-resource languages. Furthermore, we conduct an in-depth analysis of the experimental results to uncover the factors contributing to the superior performance of contrastive learning models. ViCLSR is released for research purposes in advancing natural language processing tasks.

Figures & tables

Figure 1: Overview of Typical Text Representation Models from 2013 to 2024.

Figure 2: Overview of our supervised contrastive learning framework (ViCLSR) for Vietnamese NLU tasks, including data preparation, contrastive training, and fine-tuning for downstream tasks.

Dataset	premise (x)	Hypothesis		(x, x+, x-)
Dataset	premise (x)	Entailment (x+)	Contradiction (x-)	(x, x+, x-)
ViNLI	6,094	6,094	6,094	6,094
XNLI	2,500	2,500	2,500	2,500
Total	8,594	8,594	8,594	8,594

Table 1: Statistics of Data Generated for Contrastive Learning from NLI Datasets.

Figure 3: Our Contrastive Learning Architecture with a shared XLM-R encoder.

Figure 4: Architecture for Fine-tuning the ViCLSR Model on Various Downstream NLU Tasks. Figure 4(a) Represents Fine-tuning for Natural Language Inference and Fact Checking Tasks, Where the Model Predicts the Semantic Relationship Between a Premise (Context) and Hypothesis (Claim). Figure 4(b) Illustrates Fine-tuning for Constructive Speech Detection Tasks, Processing Single Input Sentences. Figure 4(c) Demonstrates Fine-tuning for Multiple-Choice Machine Reading Comprehension Tasks, Where the Model Predicts the Correct Answer Choice Based on the Given Context and Question.

Dataset	Train	Dev	test	Task	Source	Domain	Metrics	Classes
ViNLI	24,376	3,009	2,991	NLI	Online news	Open domain	Acc/F1	3
ViWikiFC	16,738	2,090	2,091	Fact checking	Wikipedia	Open domain	Acc/F1	3
ViFactCheck	5,062	723	1,447	Fact checking	Online news	Open domain	F1	3
UIT-ViCTSD	7,000	1,000	2,000	Constructive speech detection	Social media	Open domain	Acc/F1	2
ViMMRC2.0	3,600	564	1,109	Multiple-choice MRC	Student textbooks	Literature	Acc	-

Table 2: Summary of the Benchmark Datasets and NLU Tasks Used to Evaluate the Performance of the ViCLSR Model.

Model	#Params	#Layers	#Heads	#Hidden	#Vocab	Language	Data Source	CL Strategy
PhoBERT Base	135M	12	12	768	64K	Monolingual	Wikipedia, News	No
PhoBERT Large	370M	24	16	1024	64K	Monolingual	Wikipedia, News	No
mBERT	179M	12	12	768	119K	Multilingual	Wikipedia	No
XLM-R Base	270M	12	8	768	250K	Multilingual	CommonCrawl	No
XLM-R Large	550M	24	16	1024	250K	Multilingual	CommonCrawl	No
CafeBERT	550M	24	16	1024	250K	Multilingual	Wikipedia, News	No

Table 3: Summarizing the Detailed Information of Previous Pre-Trained Models Compared to Our Model (ViCLSR).

Models	NLI		Fact Checking			Constructive Speech Detection		MRC
	ViNLI		ViWikiFC		ViFactCheck	UIT-ViCTSD		ViMMRC2.0
	Acc	F1	Acc	F1	F1	Acc	F1	Acc
PhoBERT Base	72.87	72.79	80.67	80.70	77.76	79.40	78.59	53.92
PhoBERT Large	75.93	75.87	81.63	81.62	79.76	80.70	76.86	54.73
mBERT	64.84	64.83	75.93	76.01	69.94	78.90	76.70	53.38
XLM-R Base	71.59	71.51	78.53	78.56	81.10	80.40	78.10	29.39

Table 4: The Result of Model Performance with Our ViCLSR Across NLU Tasks.

$\tau$	0.001	0.01	0.05	0.1	0.5	1
ViNLI	82.66	82.48	83.28	82.04	82.75	81.46
ViWikiFC	87.70	86.65	88.56	86.79	86.75	87.03
ViFactCheck	88.38	88.11	87.97	87.00	87.23	85.20
UIT-ViCTSD	81.59	82.30	82.75	82.35	80.55	82.70
ViMMRC2.0	55.67	55.62	56.38	55.85	55.11	55.02

Table 5: Impact of Temperature Hyperparameter

\tau

in Contrastive Loss on Fine-Tuning Accuracy for Downstream NLU Tasks.

$\lambda$	w/o	0.001	0.01	0.05	0.1	0.5	1
ViNLI	83.28	82.08	83.10	82.04	82.57	82.35	82.44
ViWikiFC	88.56	86.41	86.99	88.18	86.89	87.99	87.13
ViFactCheck	87.97	87.11	87.28	88.11	86.86	87.55	86.58
UIT-ViCTSD	82.75	81.69	82.30	82.49	82.23	82.15	80.40
ViMMRC2.0	56.38	54.28	55.27	55.73	54.78	54.78	54.28

Table 6: Effect of Auxiliary MLM Objective on Downstream NLU Tasks Performance Under Different

\lambda

Settings.

Figure 5: Impact of masking rate during contrastive learning on the downstream performance of ViCLSR across five Vietnamese NLU tasks.

Pooling Type	[CLS]	Mean	First-Last	Top-2
ViNLI	83.28	82.93	82.66	82.50
ViWikiFC	88.56	87.37	87.51	87.56
ViFactCheck	87.97	87.00	87.14	87.00
UIT-ViCTSD	82.75	82.64	82.24	82.02
ViMMRC2.0	56.38	55.49	55.16	55.92

Table 7: Impact of Different Pooling Strategies on Downstream NLU Tasks.

Pooling Type	Models	Top 1	Top 3	Top 5	Top 10
[CLS] Pooling	ViCLSR	82.62	90.96	93.50	96.04
	XLM-R Large	16.52	19.06	19.35	21.18
	PhoBERT Large	24.43	28.38	29.51	31.92
	mBERT	48.72	59.32	63.55	67.65
	CafeBERT	42.37	50.00	54.94	59.60
	DiffCSE	40.68	49.01	52.54	56.50

Table 8: Comparison of IR System Accuracy Between ViCLSR and Other Models on ViWikiFC Using [CLS] Pooling and Mean Pooling.

Panel of Figure 6 — Figure 6: Analysis of Alignment and Uniformity on ViNLI Dataset Comparing ViCLSR with Other Models Using Sentence Embeddings Derived from CLS Pooling (E: Alignment for Positive Pairs, C: Alignment for Negative Pairs).

ViCLSR	XLM-R Large	PhoBERT Large
Query 1 : ” Tô Định là kẻ vô cùng xấu xa và tàn độc. ” (Tô Định was an extremely wicked and cruel man.) Gold evidence : ” Năm 39, thái thú quận Giao Chỉ là Tô Định tàn ác, giết chồng của Trưng Trắc là Thi Sách. ” (In the year 39, Tô Định, the cruel governor of Giao Chỉ district, killed Thi Sách, the husband of Trưng Trắc.)
#1. Năm 39, thái thú quận Giao Chỉ là Tô Định tàn ác, giết chồng của Trưng Trắc là Thi Sách. (In the year 39, Tô Định, the cruel governor of Giao Chỉ district, killed Thi Sách, the husband of Trưng Trắc.)	#1. Nhà Thương thường phái quân đội đi chiến đấu chống lại những bộ tộc lân cận. (The Shang dynasty often sent its armies to fight against neighboring tribes.)	#1. Điều này hiển nhiên rất khó khiến người ta tin tưởng. (This is obviously very hard for people to believe.)
#2. Năm Cam, Khánh Trắng là một số ví dụ về băng nhóm tội phạm có tổ chức. (Năm Cam and Khánh Trắng are examples of organized criminal gangs.)	#2. Các trang phục của người dân tộc thiểu số cũng có thể sử dụng. (The traditional costumes of ethnic minorities can also be used.)	#2. Theo cư dân địa phương, bà rất linh thiêng. (According to local residents, she is considered very sacred.)
#3. Năm 931, Dương Đình Nghệ là tướng cũ của Khúc Hạo đem quân đánh phủ thành Đại La. (In 931, Dương Đình Nghệ, a former general of Khúc Hạo, led his army to attack the Đại La citadel.)	#3. Chính quyền này kiểm soát chặt chẽ mọi mặt của đời sống. (This government tightly controls all aspects of life.)	#3. Chính quyền này kiểm soát chặt chẽ mọi mặt của đời sống. (This government tightly controls all aspects of life.)
Query 2 : ” Singapore sở hữu 12 khu vực công nghiệp. ” (Singapore has 12 industrial zones.) Gold_evidence : ” Singapore có 12 khu vực công nghiệp lớn, trong đó lớn nhất là Khu công nghiệp Jurong. ” (Singapore has 12 major industrial zones, the largest of which is the Jurong Industrial Estate.)
#1. Singapore có 12 khu vực công nghiệp lớn, trong đó lớn nhất là Khu công nghiệp Jurong. (Singapore has 12 major industrial zones, the largest of which is the Jurong Industrial Estate.)	#1. Tại đó, người dân bầu cho mọi việc. (At that place, the people vote on everything.)	#1. Kinh tế của Palawan chủ yếu phụ thuộc vào nông nghiệp. (Palawan’s economy mainly depends on agriculture.)

Table 9: Comparison of Retrieved Top-3 Examples by ViCLSR Against XLM-R and PhoBERT on the ViWikiFC Dataset. Retrieved sentences highlighted in blue-green text represent correct retrievals, whereas those in bittersweet text denote incorrect retrievals.

Figure 7: Impact of training data size during contrastive learning on the downstream performance of ViCLSR across five Vietnamese NLU tasks.

Panel of Figure 8 — Figure 8: Visualization of the Attention Map Between a Premise and Hypothesis Sentence Using the Pre-Trained XLM-R Model and Our ViCLSR Model.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Training-validation loss curves of the ViCLSR model on downstream tasks.

Sentence Embeddings

Sentence embeddings map sentences to vectors for semantic similarity, retrieval, clustering, and downstream language tasks.

See all 35 papers

Feb 26, 2026

ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport

Image-text retrieval has become a fundamental component in intelligent multimedia systems; however, most existing vision-language models are optimized for highresource languages and remain suboptimal for low-resource settings such as Vietnamese. This work introduces ViCLIP-OT, a foundation vision-language model specifically designed for Vietnamese image-text retrieval. The proposed framework integrates CLIP-style contrastive learning with a Similarity-Graph Regularized Optimal Transport (SIGROT) loss to enhance global cross-modal consistency and mitigate modality gap issues. Extensive experiments on three Vietnamese benchmarks (UITOpenViIC, KTVIC, and Crossmodal-3600) demonstrate that ViCLIP-OT consistently outperforms CLIP and SigLIP baselines in both in-domain and zero-shot settings. On UIT-OpenViIC, the model achieves an average Recall@K of 67.34%, improving upon CLIP by 5.75 percentage points. In zero-shot evaluation on Crossmodal-3600, ViCLIPOT surpasses CLIP by 11.72 percentage points. Embedding-space analysis further confirms improved alignment and reduced modality gap. The results indicate that integrating SIGROT provides an effective and scalable strategy for cross-modal retrieval in low-resource languages, offering practical implications for intelligent multimedia retrieval systems in Vietnamese and other underrepresented linguistic contexts.

Quoc-Khang Tran, Minh-Thien Nguyen, Nguyen-Khang Pham

Can Tho University, Can Tho, Vietnam

Vision-Language Retrieval Cross-Modal Retrieval Vision-Language Alignment Modality Gap in VLMs

Ask PDF

Jul 2, 2026

BamiBERT: A New BERT-based Language Model for Vietnamese

In this paper, we introduce BamiBERT, a new BERT-based pre-trained language model for Vietnamese that addresses key limitations of PhoBERT -- the current de facto Vietnamese text encoder. Trained from scratch on a 129GB corpus of general-domain Vietnamese text for 20 epochs, BamiBERT supports an extended context length of up to 2048 tokens and operates directly on raw input, eliminating the need for external word segmentation. Across 8 Vietnamese benchmarks, it achieves the best score on 11 of 15 metrics and the second-best on 3 others, setting a new state of the art among "base"-sized Vietnamese encoders and demonstrating strong cross-domain generalization. We release BamiBERT at: https://huggingface.co/Qualcomm-AI-Research/BamiBERT

Dat Quoc Nguyen, Thinh Pham, Chi Tran +1

Qualcomm AI Research · Virginia Tech

Language Model Pretraining Transformer Long-Context Language Modeling Language Modeling

Ask PDF

Jun 9, 2026

ViP-VL: Vietnamese Self-supervised Speech Pretraining Model with Vector-Quantization Learning

We present ViP-VL, an efficient Vietnamese Self-supervised speech Pretraining model leveraging Vector-quantization Learning. To bridge the gap between high-resolution audio and efficient processing, ViP-VL incorporates Acoustic Stacking and Receptive Field Alignment to enable a synchronized 8x subsampling rate within the ChunkFormer architecture, while further enhancing representation robustness through a specialized Mask Selection Strategy during pretraining on the BEST-RQ framework. Pretrained on 17,000 hours of unlabeled Vietnamese speech, our model establishes new state-of-the-art results across four major downstream tasks: Automatic Speech Recognition, Speech Emotion Recognition, Dialect Classification, and Speaker Verification. To facilitate future research and the development of high-performance Vietnamese speech technologies, we publicly release our pretrained weights and implementation at github.com/khanld/chunkformer.

Khanh Le, Kiet Anh Hoang, Bao Nguyen +5

VinUniversity, Vietnam · UNEY, Switzerland

Audio Representation Learning Speech Processing Audio Understanding Automatic Speech Recognition Self-Supervised Speech Representation Learning Discrete Speech Representation Learning

Ask PDF

Abstract

Sentence Embeddings

Explore similar work

ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport

BamiBERT: A New BERT-based Language Model for Vietnamese

ViP-VL: Vietnamese Self-supervised Speech Pretraining Model with Vector-Quantization Learning