VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models

Nguyen Tien Dong; Minh-Anh Nguyen; Thanh Dat Hoang; Nguyen Tuan Ngoc; Dao Xuan Quang Minh; Phan Phi Hai; Nguyen Thi Ngoc Anh; Dang Van Tu

cs.CLDec 16, 2025

VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models

Authors: Nguyen Tien Dong, Minh-Anh Nguyen, Thanh Dat Hoang, Nguyen Tuan Ngoc, Dao Xuan Quang Minh, Phan Phi Hai, Nguyen Thi Ngoc Anh, Dang Van Tu,

Organizations: CMC OpenAI · VinUniversity · HUST, Vietnam · SRH University Heidelberg, Germany

Abstract

The rapid advancement of large language models (LLMs) has enabled new possibilities for applying artificial intelligence within the legal domain. Nonetheless, the complexity, hierarchical organization, and frequent revisions of Vietnamese legislation pose considerable challenges for evaluating how well these models interpret and utilize legal knowledge. To address this gap, the Vietnamese Legal Benchmark (VLegal-Bench) is introduced, the first comprehensive benchmark designed to systematically assess LLMs on Vietnamese legal tasks. Informed by Bloom's cognitive taxonomy, VLegal-Bench encompasses multiple levels of legal understanding through tasks designed to reflect practical usage scenarios. The benchmark comprises 10,450 samples generated through a rigorous annotation pipeline, where legal experts label and cross-validate each instance using our annotation system to ensure every sample is grounded in authoritative legal documents and mirrors real-world legal assistant workflows, including general legal questions and answers, retrieval-augmented generation, multi-step reasoning, and scenario-based problem solving tailored to Vietnamese law. By providing a standardized, transparent, and cognitively informed evaluation framework, VLegal-Bench establishes a solid foundation for assessing LLM performance in Vietnamese legal contexts and supports the development of more reliable, interpretable, and ethically aligned AI-assisted legal systems. To facilitate access and reproducibility, we provide a public landing page for this benchmark at https://vilegalbench.cmcai.vn/.

Figures & tables

Level	ID	Task	Purpose	Type	Metric	Test set
1. Recognition & Recall	1.1	Legal Entity Recognition	To detect and classify named entities, including persons, organizations, monetary amounts, … within legal documents.	MCQ, NER	Accuracy, EM	750
	1.2	Legal Topic Classification	Classifies legal questions into predefined legal topics.	MCQ, MLC	Accuracy, Macro-F1	700
	1.3	Legal Concept Recall	Recalls statutory definitions or meanings of legal terms and concepts.	MCQ	Accuracy	300
	1.4	Article Recall	Retrieves or cites the correct legal article corresponding to a term, concept, or question.	MCQ	Accuracy	1000
	1.5	Legal Schema Recall	Recognizes and recalls hierarchical and temporal relations among legal documents (e.g., amendments, replacements, …)	MCQ	Accuracy	800
2. Understanding & Structuring	2.1	Relation Extraction	Extracts the subject, object, and content of a legal relationship from a factual scenario	MCQ	Accuracy	253

Table 1: Overview of VietLegal: The benchmark evaluates legal LLMs across five levels, from basic recognition to ethical reasoning, using five question templates: Multiple-Choice Question Answering (MCQ), Multi-Label Classification (MLC), Binary Classification (BC), Named Entity Recognition (NER) and Generation for Vietnamese law.

Figure 1: VietLegal data pipeline. Data are collected from Vietnamese legal sources, preprocessed, stored in a database, and used to build an information retrieval tool for legal experts. The final dataset is obtained through manual annotation.

Measure	Value
Exact agreement with published level	97.0% (64/66; 95% CI 92.4–100.0)
Agreement within $\pm$ 1 level	98.5%
Linear-weighted Cohen’s $\kappa$ (mean over raters)	0.97 (range 0.94–1.00)
Human difficulty vs. assigned level (Spearman $\rho$ )	0.79 ( $p<10^{-4}$ )
Rater confidence in assignment (1–5)	4.29 $\pm$ 0.92

Table 2: Independent expert validation of the task-to-level assignment (3 senior legal experts

\times

22 tasks = 66 judgments).

		Recognition & Recall					Understanding & Structuring
Model Type	Model	1.1 Acc	1.2 Acc	1.3 Acc	1.4 Acc	1.5 Acc	2.1 Acc	2.2 Acc	2.3 R-L	2.4 Acc	2.5 m-F1
General Multilingual LLMs	gpt-4o	70.03	81.14	73.67	82.50	24.25	85.37	67.33	0.470	80.67	63.28
	gpt-4o-mini	65.24	82.71	61.33	68.40	22.25	87.35	51.33	0.529	73.33	61.68
	claude-sonnet-4.5	69.78	82.47	83.00	84.19	27.25	79.21	75.33	0.808	87.81	62.04
	gemini-2.5-flash	71.96	81.40	61.33	81.40	24.25	80.63	64.00	0.656	82.30	49.92
	gpt-oss-20b	22.72	73.21	47.33	39.00	18.63	45.06	28.33	0.249	66.50	56.43

Table 3: Zero-shot Performance Comparison of Language Models across Different Tasks. Red bold indicates the best overall performance. Blue bold indicates the best performance among open-source models.

		Reasoning & Infer.					Interpret. & Generation			Ethics, Fairness & Bias
Model Type	Model	3.1 Acc	3.2 Acc	3.3 Acc	3.4 Y-F1/N-F1	3.5 Acc	4.1 R-L	4.2 R-L	4.3 R-L	5.1 Acc	5.2 Acc	5.3 Acc	5.4 Acc
General Multilingual LLMs	gpt-4o	38.83	84.50	73.50	27.21/42.16	67.97	0.3257	0.4017	0.4975	44.18	67.74	91.04	58.11
	gpt-4o-mini	35.33	82.17	74.66	11.85/39.59	57.38	0.3272	0.4167	0.4926	41.76	67.74	86.56	51.28
	claude-sonnet-4.5	59.83	88.83	79.10	0/39.61	69.36	0.2842	0.3857	0.3830	45.38	72.81	91.04	69.66
	gemini-2.5-flash	40.83	84.67	76.71	0/39.61	63.51	0.2756	0.3982	0.3778	46.18	68.66	94.04	61.54
	gpt-oss-20b	29.67	66.00	52.74	73.41/35.56	37.15	0.0262	0.1529	0.3104	21.69	58.80	33.76	28.20

Table 4: Zero-shot performance comparison across different tasks. Red bold denotes the best overall result, while blue bold indicates the best-performing open-source model. For Task 3.4, we report F1 scores separately for the Yes and No labels (Y-F1 / N-F1).

Figure 2: Ablation of prompting strategies. Accuracy score distributions across prompting strategies, aggregated over Accuracy-based tasks within each cognitive level (excluding tasks evaluated with other metrics).

Task	Metric	$\boldsymbol{\rho}$	95% CI	$\boldsymbol{p}$
4.1 Summarization	ROUGE-L	0.66	[0.51, 0.78]	$1.4\times 10^{-10}$
4.1 Summarization	BERTScore	0.65	[0.49, 0.77]	$5.2\times 10^{-10}$
4.2 Judicial Reasoning	ROUGE-L	0.44	[0.23, 0.62]	$1.4\times 10^{-4}$
4.2 Judicial Reasoning	BERTScore	0.56	[0.37, 0.71]	$3.9\times 10^{-7}$
4.3 Legal Opinion	ROUGE-L	0.52	[0.32, 0.67]	$4.2\times 10^{-6}$
4.3 Legal Opinion	BERTScore	0.56	[0.38, 0.71]	$3.0\times 10^{-7}$

Table 5: Spearman correlation between automatic metrics and expert-rated Legal Accuracy, computed within each of four models and combined via Fisher’s

z

(effective

n=80

per task).

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Level 1	Level 2	Level 3	Level 5
Random Guessing	25.00	25.17	26.71	25.00
Longest Answer	20.50	25.00	23.29	24.60
PhoBERT Similarity	34.00	28.00	29.52	33.50

Appendix

Table 6: Performance (%) of non-reasoning baselines across difficulty levels. Level 4 is excluded as it consists of open-ended generation tasks.

Figure 3: The five-level cognitive framework of VietLegal

Model	Recall	Precision	Exact Match
Claude-Sonnet-4.5	0.5748	0.5417	0.0546
GPT-4o	0.5372	0.5904	0.0853
GPT-4o-mini	0.4914	0.6402	0.0986
Qwen2.5-32B-Instruct	0.4687	0.4754	0.0375
Proprietary-Legal-32B	0.5810	0.5872	0.1184

Appendix

Table 7: NER evaluation results for Task 1.1 - Legal Entity Recognition.

Model	Macro-F1
GPT-4o	0.8115
GPT-4o-mini	0.8284
Proprietary-Legal-32B	0.8560
Qwen2.5-32B-Instruct	0.8560
Qwen2.5-14B-Instruct	0.8247
Qwen2.5-7B-Instruct	0.7977

Appendix

Table 8: Macro-F1 evaluation results for Task 1.2 - Legal Topic Classification.

Model	Node-F1	Edge-F1
GPT-4o	0.2037	0.7989
GPT-4o-mini	0.1822	0.5390
Proprietary-Legal-32B	0.0412	0.8352
Qwen2.5-72B-Instruct	0.0296	0.7325
Qwen2.5-32B-Instruct	0.4250	0.8216
Qwen2.5-14B-Instruct	0.1511	0.7869

Appendix

Table 9: Node-F1 and Edge-F1 evaluation results for Task 2.3 - Legal Knowledge Graph Structuring.

Model	Task 4.1	Task 4.2	Task 4.3
Gemini-2.5-Flash	0.7053	0.7429	0.7391
Claude-Sonnet-4.5	0.7253	0.7318	0.7397
Qwen2.5-72B-Instruct	0.7210	0.7454	0.7643
Qwen2.5-32B-Instruct	0.7102	0.7406	0.7560
Qwen2.5-14B-Instruct	0.7048	0.5033	0.7469
Llama-3.1-8B-Instruct	0.7335	0.7191	0.7301

Appendix

Table 10: BERTScore F1 for open-ended generation tasks.

Figure 4: Annotation tool interface: a custom-built tool that supports junior annotators by attaching senior-selected Articles, Clauses, and Points. Junior experts create samples and perform cross-verification in the feedback column.

Figure 5: Legal document retrieval tool: we provide a search interface that enables legal experts to efficiently locate relevant legal documents and supporting materials for the annotation process.

Model Type	Model	Size	Context	Access & URL
General Multilingual LLMs	GPT-4o	-	128k	API: platform.openai.com
	GPT-4o-mini	-	128k	API: platform.openai.com
	Claude 4.5 Sonnet	-	200k	API: platform.claude.com
	Gemini 2.5 Flash	-	1M	API: ai.google.dev
	Qwen 2.5 Instruct	3B	32k	Weights: huggingface.co/Qwen
	Qwen 2.5 Instruct	7B $\rightarrow$ 72B	128k	Weights: huggingface.co/Qwen

Appendix

Table 11: Large language models evaluated on VietLegal.

Model	Task 4.1		Task 4.2		Task 4.3
Model	Legal Accuracy	Completeness	Legal Accuracy	Completeness	Legal Accuracy	Completeness
gpt-4o	3.59	3.18	2.62	2.59	3.27	3.26
gpt-4o-mini	3.39	3.01	2.36	2.36	3.30	3.38
Qwen2.5-72B-Instruct	3.34	2.76	2.40	2.53	3.23	3.16
Proprietary-Legal-32B	3.79	3.38	2.53	2.62	3.57	3.55
Human	4.79	4.88	4.76	4.78	4.82	4.85

Appendix

Table 12: Human evaluation of generated answers and ground-truth responses on interpretation and generation tasks (Tasks 4.1-4.3). Three senior legal experts independently rated each sample based on Legal Accuracy and Completeness using a 1-5 Likert scale. Evaluators were blinded to the source of each response (human-written or model-generated).

VietLegal level	Existing civil-law benchmark analogue	Transferability
Level 1 – Recognition & Recall	LawBench: legal knowledge memorization (article recitation, legal knowledge QA)	Directly transferable after replacing the statutory corpus with that of the target jurisdiction.
Level 2 – Understanding & Structuring	LawBench: legal knowledge understanding (NER, event detection, relation extraction); BSARD: statutory structure and provision organization	Directly transferable, as these tasks operate on the hierarchical organization of statutory law.
Level 3 – Reasoning & Inference	LawBench: article, charge, and prison-term prediction, case analysis; BSARD and GerLayQA: statutory retrieval and grounded question answering	Directly transferable, since every civil-law system requires applying statutory rules to facts; jurisdiction-specific rules must be substituted.
Level 4 – Interpretation & Generation	LawBench: legal consultation and summarization; GerLayQA: grounded legal answer generation	Transferable at the cognitive level; the legal-writing format (e.g., IRAC or the German Gutachtenstil ) should follow local practice.
Level 5 – Ethics, Fairness & Bias	No direct counterpart in existing benchmarks	Conceptually transferable, but evaluation criteria depend on each jurisdiction’s professional ethics, privacy regulations, and legal standards.

Appendix

Table 13: Mapping VietLegal’s cognitive levels to existing civil-law benchmarks, and what must change to transfer each level to a new jurisdiction.

Model	Understanding & Structuring					Reasoning & Inference
Model	2.1	2.2	2.3	2.4	2.5	3.1	3.2	3.3	3.4	3.5
	Acc	Acc	R-L	Acc	m-F1	Acc	Acc	Acc	Y-F1/N-F1	Acc
gpt-4o	75.86	55.77	0.630	79.97	55.29	39.00	81.36	77.93	2.35/41.96	66.48
gpt-4o-mini	75.86	50.16	0.612	79.30	55.00	34.67	78.67	72.26	1.59/39.81	53.48
Gemini-2.5-flash	78.54	67.35	0.456	73.08	55.95	50.60	84.81	75.37	17.46/42.86	66.01
Qwen2.5-32B-Instruct	73.72	62.33	0.612	80.63	59.78	32.67	84.00	71.23	0.00/39.61	54.32

Appendix

Table 14: Zero-shot performance with reasoning on Understanding & Structuring (Level 2) and Reasoning & Inference (Level 3) tasks. R-L denotes ROUGE-L score. For Task 3.4 (Conflict Detection), we report Y-F1/N-F1 scores. Red indicates best overall results and blue indicates best open source results in each column.

Model	Understanding & Structuring					Reasoning & Inference					Ethics, Fairness & Bias
Model	2.1	2.2	2.3	2.4	2.5	3.1	3.2	3.3	3.4	3.5	5.1	5.2	5.3	5.4
	Acc	Acc	R-L	Acc	m-F1	Acc	Acc	Acc	Y-F1/N-F1	Acc	Acc	Acc	Acc	Acc
gpt-4o	86.16	68.33	0.657	85.64	56.83	40.67	85.33	78.77	77.67/62.30	69.64	43.37	69.12	92.04	63.68
gpt-4o-mini	78.27	58.33	0.623	85.31	61.83	39.67	82.50	77.05	44.58/44.58	59.05	44.18	65.44	87.55	46.15
Gemini-2.5-flash	78.66	61.67	0.422	83.81	53.18	50.17	86.00	76.37	85.71 /0.00	65.18	46.98	66.36	91.05	65.38
Qwen2.5-32B-Instruct	69.80	67.00	0.755	86.81	57.59	39.67	84.83	75.34	42.00 /35.44	59.05	56.59	68.67	93.04	70.51

Appendix

Table 15: Few-shot performance without reasoning on Understanding & Structuring (Level 2), Reasoning & Inference (Level 3), and Ethics, Fairness & Bias (Level 5) tasks. R-L denotes ROUGE-L score. For Task 3.4 (Conflict Detection), we report Y-F1/N-F1 scores. Red indicates best overall results and blue indicates best open-source results in each column.

Model	Setting	3.1 Acc	3.2 Acc	3.3 Acc	3.4 Y-F1/N-F1
GPT-4o	Zero-shot No Reasoning	38.83	84.50	73.50	27.21/42.16
GPT-4o	Agentic RAG	60.83 ( +22.00 )	82.00 ( -2.50 )	72.9 ( -0.60 )	53.15/40.00
GPT-4o-mini	Zero-shot No Reasoning	35.33	82.17	74.66	11.85/39.59
GPT-4o-mini	Agentic RAG	54.17 ( +18.84 )	80.83 ( -1.34 )	70.89 ( -3.77 )	94.65/39.41
Gemini-2.5-Flash	Zero-shot No Reasoning	40.83	84.67	76.71	0.00/39.61
Gemini-2.5-Flash	Agentic RAG	53.05 ( +12.22 )	81.62 ( -3.05 )	71.58 ( -5.13 )	87.69/40.59

Appendix

Table 16: Ablation study comparing Zero-shot No Reasoning and Agentic RAG across reasoning tasks. Blue indicates improvement, red indicates degradation. Numbers in parentheses show the performance delta.

INSTRUCTION: Read the following multiple-choice question and select the correct answer. Choose only one option; no explanation is required.

QUESTION: Extract all named entities from the following description: “In the speech at the ceremony, Dr. Vu Hoai X emphasized: “Besides professional duties, the PLVN Newspaper always places strong emphasis on community-oriented social activities. The “Judicial Warm Home” program is not only an act of sharing, but also a commitment to accompany people in difficult circumstances, helping them improve their lives. Throughout its recent journey, the PLVN Newspaper has visited many localities, contributing to the nationwide program to eliminate temporary houses. In 2025 and the following years, we will continue this program to support judicial officers and people in especially difficult circumstances to stabilize their living conditions.””

ANSWER OPTIONS:

A. (PERSON: Dr. Vu Hoai X), (ORGANIZATION: PLVN Newspaper), (PROJECT: Judicial Warm Home), (DATE: 2025), (SOCIAL_ROLE: judicial officers), (SOCIAL_ROLE: citizens)

B. (PERSON: Vu Hoai X), (ORGANIZATION: PLVN Newspaper), (PROJECT: Judicial Warm Home), (DATE: 2025), (SOCIAL_ROLE: citizens in special hardship)

C. (PERSON: Dr. Vu Hoai X), (ORGANIZATION: PLVN Newspaper), (PROJECT: Judicial Warm Home), (DATE: 2025), (SOCIAL_ROLE: judicial officers), (SOCIAL_ROLE: citizens in especially difficult circumstances)

D. (PERSON: Dr. Vu Hoai X), (ORGANIZATION: PLVN Newspaper), (PROJECT: Judicial Warm Home), (DATE: 2025), (SOCIAL_ROLE: judicial officers)

Appendix

Table 17: The instruction and an example of the Legal Entity Recognition task.

INSTRUCTION: Read the following question and identify its legal domain. Choose only one option; no explanation is required.

QUESTION: Mr. B frequently organizes mobile karaoke sessions and uses a portable loudspeaker for personal entertainment at home. Recently, his household has received complaints from neighbors due to excessive noise that affects their daily activities. In this case, if Mr. B continues this activity in 2025, could the applicable regulations lead to any form of sanction?

ANSWER OPTIONS:

A. Legal services

B. Administrative apparatus

C. Securities

D. Banking and finance

Appendix

Table 18: The instruction and an example of the Legal Topic Classification task.

INSTRUCTION: Read the following question and select the correct answer. Choose only one option; no explanation is required.

QUESTION: According to the law, how is a civil transaction defined?

ANSWER OPTIONS:

A. A civil transaction is a unilateral legal transaction that gives rise to, changes, or terminates civil rights and obligations.

B. A civil transaction is a contract or a unilateral legal act that gives rise to, changes, or terminates civil rights and obligations.

C. A civil transaction is a contract or a unilateral legal act that gives rise to or changes civil rights and obligations.

D. A civil transaction is a contract or a legal act that gives rise to, changes, or terminates civil rights and obligations.

Appendix

Table 19: The instruction and an example of the Legal Concept Recall task.

INSTRUCTION: Read the following question and select the correct answer. Choose only one option; no explanation is required.

QUESTION: What content is regulated in Point (a), Clause 1, Article 1 of Decree No. 113/2007/ND-CP?

ANSWER OPTIONS:

A. Regulations on ownership rights and management of dikes.

B. Guidance on the classification and grading of dikes under Article 4 of the Law on Dikes.

C. Regulations on forms of sanctions for violations related to dikes.

D. Guidance on the protection of dikes during the flood season.

Appendix

Table 20: The instruction and an example of the Article Recall task.

INSTRUCTION: Read the following question and select the correct answer. Choose only one option; no explanation is required.

QUESTION: Which decree serves as the legal basis for Circular 10/2025/TT-BNNMT?

ANSWER OPTIONS:

A. Decree 35/2025/ND-CP

B. Decree 70/2025/ND-CP

C. Decree 48/2024/ND-CP

D. Decree 12/2024/ND-CP

Appendix

Table 21: The instruction and an example of the Legal Schema Recall task.

INSTRUCTION: Read the following multiple-choice question and select the correct answer. Choose only one option; no explanation is required.

QUESTION: Which legal relationships appear in the following situation? The defendant, Ms. X, maintains her request for appeal. Lawyer H requests to reclassify the dispute as a “Deposit Contract Dispute” and requests the application of the statute of limitations, proposing that the Court reject all claims of the plaintiff. The plaintiff’s representative disagrees with the defendant’s appeal and requests that the case be resolved in accordance with the law. The representative of the People’s Procuracy of Lam Dong Province comments that the Judge, the Trial Panel, and the litigants have complied with the Civil Procedure Code during the appellate stage and the hearing, and proposes that the Trial Panel partially accept the defendant’s appeal pursuant to Clause 2, Article 308 of the 2015 Civil Procedure Code and revise the first-instance judgment regarding the value of the disputed property.

ANSWER OPTIONS:

A. Ms. Le Thi X - Withdraws the appeal; Plaintiff and Procuracy - Make no further requests

B. Ms. Le Thi X - Files an appeal and requests reclassification of the legal dispute; Plaintiff and Procuracy - Request resolution in accordance with the law and partial acceptance of the appeal

C. Ms. Le Thi X - Agrees with the first-instance judgment; Plaintiff and Procuracy - Request full revision of the first-instance judgment

D. Ms. Le Thi X - Agrees with the first-instance judgment; Plaintiff and Procuracy - Request full revision of the first-instance judgment

Appendix

Table 22: The instruction and an example of the Relation Extraction task.

INSTRUCTION: Read the following multiple-choice question and select the correct answer. Choose only one option; no explanation is required.

QUESTION: Identify the components of the legal norm mentioned in the text below: EMPLOYMENT CONTRACT Conclusion of the Employment Contract The probationary salary of an employee during the probation period shall be agreed upon by both parties, but must be at least 85% of the salary of the job.

ANSWER OPTIONS:

A. Hypothesis: Employee during the probation period. Disposition: Probationary salary must be at least 100% of the job salary.

B. Hypothesis: Employee during the probation period. Disposition: Probationary salary is decided unilaterally by the employee.

C. Hypothesis: Salary of the employee during the probation period. Disposition: Agreed upon by both parties, but must be at least 85% of the job salary.

D. Hypothesis: Employee during the probation period. Disposition: Probationary salary must be at least 80% of the job salary.

Appendix

Table 23: The instruction and an example of the Legal Element Recognition task.

INSTRUCTION: Extract clauses and their relationships from the input data as a list of triplets (entity 1, relation, entity 2).

QUESTION: Extract clauses and their relationships from the following text: “Clause 3 / Article 1 contains the following context: “3. Amend and supplement Point b, Point h, and Point i, Clause 1, Article 3 as follows: “1. Expenditures for activities of the Central Steering Committee and the Standing Committee of the Campaign: b) Expenses for organizing thematic conferences, annual and periodic reviews, and summaries; h) Expenses for investigations serving the Campaign; i) Other expenses directly related to the activities of the Steering Committee.” “Article 1. Amend and supplement several articles of Circular No. 91/2012/TT-BTC dated May 30, 2012 of the Ministry of Finance:””

GROUND TRUTH: (Clause 3 / Article 1, amends and supplements, Point b / Clause 1 / Article 3); (Clause 3 / Article 1, amends and supplements, Point h / Clause 1 / Article 3); (Clause 3 / Article 1, amends and supplements, Point i / Clause 1 / Article 3)

Appendix

Table 24: The instruction and an example of the Legal Graph Structuring task.

INSTRUCTION: Determine whether the court’s assessment below is correct or incorrect based on the given case description. Answer only “Correct” or “Incorrect”; no explanation is required.

QUESTION: Ms. Lo Thi V and Mr. Ca Van L are married and have two children. Due to marital conflicts and the fact that Mr. L is currently serving a prison sentence, Ms. V requests a divorce and custody of the children. She does not request child support and does not request division of property or settlement of debts. Mr. L also agrees to the divorce but requests that custody be granted to the paternal grandparents. Court’s assessment: The court decides to grant Ms. Lo Thi V a divorce from Mr. Ca Van L and grants custody of both children to Ms. V, and exempts Mr. Ca Van L from child support obligations at Ms. V’s request.

ANSWER OPTIONS:

A. Correct

B. Incorrect

GROUND TRUTH: A

Appendix

Table 25: The instruction and an example of the Judgment Verification task.

INSTRUCTION: Read the following query and identify its correct intent. Choose the correct option(s); no explanation is required. Intent list: • chitchat : Questions not related to law (e.g., greetings, thanks, off-topic) • comparative_analysis : Comparing content between legal documents, clauses, or provisions • document_relationship : Questions about relationships between legal documents (e.g., amendments, supplements, references, legal basis) • document_retrieval : Retrieving full legal documents • external_analysis : Economic or social impacts, trends, historical development, or policy impact analysis • general : General legal questions not falling into specific categories • legal_query : Retrieving answers from specific articles/clauses/points • stats_summary : Counting or summarizing the number of regulations/documents

QUESTION: Please carefully review Clause 2 of Article 20, especially Clause 3 of Article 23 regarding voucher-based support, and provide recommendations on this draft from the perspective of an information technology company investing in AI product development.

ANSWER OPTIONS:

A. legal_query

B. comparative_analysis

C. stats_summary

D. external_analysis

Appendix

Table 26: The instruction and an example of the User Intent Understanding.

INSTRUCTION: Read the following question and select the relevant article, clause, or legal document that supports the answer. Choose only one correct option; no explanation is required.

QUESTION: What is the form of asset handling when the owner voluntarily transfers the ownership rights to the State of Vietnam?

ANSWER OPTIONS:

A. Clause 4, Article 10, Decree 88/2023/ND-CP

B. Clause 1, Article 20, Decree 66/2022/ND-CP

C. Clause 1, Article 8, Decree 77/2025/ND-CP

D. Clause 2, Article 15, Decree 88/2023/ND-CP

Appendix

Table 27: The instruction and an example of the Article / Clause Prediction task.

INSTRUCTION: From the given case description, choose the answer that correctly reflects the court’s judgment. Only select the option; no explanation is needed.

QUESTION: From the given judgment content, which answer correctly reflects the court’s decision?

ANSWER OPTIONS:

A. The court allows divorce, grants custody to Lo Thi V, and exempts Mr. Ca Van L from child support.

B. The court allows divorce and requires Mr. Ca Van L to pay monthly child support.

C. The court grants custody to the grandparents as requested by Mr. Ca Van L.

D. The court does not allow the divorce and requires reconciliation.

Appendix

Table 28: The instruction and an example of the Legal Court Decision Prediction task.

INSTRUCTION: Read the following multiple-choice question and select the correct answer. Only choose the option; no explanation is required.

QUESTION: Nam is a candidate eligible for direct university admission. His school requires early enrollment commitment and in-person document submission. How to determine whether this requirement is valid?

ANSWER OPTIONS:

A. Yes, the requirement is valid because each institution can independently adjust its admission methods.

B. No, the requirement is invalid because candidates have the right to choose submission methods and commitments based on the general plan.

C. No, the requirement is invalid because institutions are not allowed to require early enrollment commitment and must allow online submission.

D. Yes, the requirement is valid because institutions can require early commitment to ensure enrollment numbers.

Appendix

Table 29: The instruction and an example of the Multi-Article Reasoning task.

INSTRUCTION: Determine whether the two legal norms below (the reviewed regulation and the reference regulation) contradict each other. Answer "Yes" if they contradict and "No" if they do not contradict.

QUESTION: Reviewed regulation: Document number: 74/2015/NĐ-CP; Position: Article 5, Decree No. 74/2015/NĐ-CP Reference regulation: Document number: 66/2006/QH11; Position: Article 9, Civil Aviation Law of Vietnam 2006

ANSWER OPTIONS:

A. Yes

B. No

GROUND TRUTH: B

Appendix

Table 30: The instruction and an example of the Conflict & Consistency Detection task.

INSTRUCTION: Read the following multiple-choice question and select the correct answer. Only choose the answer; no explanation is required.

QUESTION: What is the penalty under the law for K’s behavior in the described situation?

ANSWER OPTIONS:

A. Calling friends to vandalize the shop is considered inciting public disorder, with a fine of 2,000,000 to 3,000,000 VND.

B. Holding a stick to hit a person is an act infringing upon another person’s health, with a fine from 2,000,000 to 3,000,000 VND.

C. Both acts are subject to a fine of 2,000,000 to 3,000,000 VND.

D. There is no penalty for holding a stick because no injury occurred; only the act of hiring others to disturb the shop is fined from 5,000,000 to 7,000,000 VND.

Appendix

Table 31: The instruction and an example of the Penalty / Remedy Estimation task.

INSTRUCTION: Summarize the following content in no more than 400 words.

CONTENT: Decree regulating dialogue with youth and mechanisms, policies, and measures for implementing policies for youth from full 16 years of age to under 18 years of age. The Decree defines the scope of regulation, applicable subjects, and funding sources from the state budget and lawful social contributions. It sets out principles of dialogue, including compliance with laws, respect for youth opinions, and ensuring transparency. Prime Minister and Chairpersons of People’s Committees at all levels are responsible for organizing at least one annual dialogue with youth. Dialogue may be conducted in direct or online forms. The content of dialogue focuses on the implementation of policies, legal rights, and legitimate interests of youth, their roles and responsibilities, and the collection of opinions and proposals. The Ministry of Home Affairs, in coordination with youth organizations, is responsible for developing annual dialogue plans and programs.

GROUND TRUTH: Decree No. 13/2021/ND-CP dated March 1, 2021, of the Government on dialogues with youth and mechanisms, policies, and measures for implementing policies for youth aged from 16 to under 18 years old, including its effective date, purpose of promulgation, main contents, chapters, articles, and scope of application.

Appendix

Table 32: The instruction and an example of the Legal Document Summarization task.

INSTRUCTION: Analyze the following legal situation question using the IRAC structure (Issue, Rule, Application, Conclusion) and provide an answer based on the IRAC structure.

QUESTION: Mr. Manh is married to Ms. Lien, but he has a child with Ms. Ha. Mr. Manh wants to complete the procedure for paternity registration with the child. However, because he is afraid that Ms. Lien will find out, Mr. Manh plans to authorize Ms. Ha to carry out the paternity registration procedure. So, is it permissible for Mr. Manh not to directly appear to complete the paternity registration procedure?

ANSWER (IRAC STRUCTURE):

Issue: Can Mr. Manh authorize Ms. Ha to carry out the paternity registration procedure?

Rule: Clause 2, Article 6 of the Civil Status Law 2014: Paternity/maternity registration must be done directly. Article 2 of Circular 04/2020/TT-BTP: Authorization is not permitted.

Application: The procedure is of a personal nature and must be confirmed directly.

Conclusion: Mr. Manh is not allowed to authorize Ms. Ha.

Appendix

Table 33: The instruction and an example of the Judicial Reasoning Generation task.

Field	Content
Instruction	Your task is to answer the following legal question. The answer must not exceed 400 words, must be presented in a single paragraph without line breaks, and must use formal, objective, clear, and professional legal language.
Article	Building a streamlined government in the modern context: legal and organizational analysis of international models and the current situation in Vietnam, and proposals for comprehensive solutions to build an effective and efficient government.
Description	The article analyzes the streamlined government model in countries such as Singapore, Japan, and the United States from legal and organizational perspectives, identifying challenges in Vietnam such as overlapping functions, bulky organizational structures, and inconsistent application of information technology. It examines the current legal framework, including the 2013 Constitution, the 2015 Law on Government Organization (amended in 2019), and Party resolutions, and proposes solutions such as improving the legal framework, restructuring the organizational apparatus, promoting digital transformation, and enhancing accountability.
Question	In the current context, how can Vietnam successfully apply the streamlined government model based on international experience and the existing domestic situation?
Answer	To successfully apply the streamlined government model in Vietnam, it is necessary to implement a coherent set of solutions based on international lessons. Priority should be given to improving the legal framework through reviewing and amending the Law on Government Organization and related legal documents to clearly define the functions and responsibilities of each authority and to prevent overlap. Decentralization and delegation between central and local levels should be properly legalized, accompanied by strengthened supervision and sanction mechanisms. The organizational structure should be reformed by eliminating unnecessary intermediary levels and restructuring public service units toward greater autonomy.

Appendix

Table 34: The instruction and an example of the Objective Legal Opinion Generation task.

INSTRUCTION: Read the following multiple-choice question and select one or more correct answers. Only select the answer(s) without any explanation.

QUESTION: A female prisoner from an ethnic minority group is serving a prison sentence at Prison X. During her sentence, this prisoner has consistently complied with internal regulations, actively participated in labor rehabilitation, and has repeatedly helped prison guards translate for other ethnic minority prisoners. However, a prison guard believes that "ethnic minority people are often difficult to rehabilitate" and therefore plans not to include her in the list for consideration of sentence reduction. According to the Vietnamese law on criminal sentence execution and the requirement to ensure non-discrimination, which option below is correct and complies with the law?

ANSWER OPTIONS:

A. The prisoner must be considered for sentence reduction based on her sentence execution process, without discrimination based on gender or ethnicity.

B. Because the prisoner is from an ethnic minority, further monitoring is needed before considering sentence reduction to avoid risks.

C. Sentence reduction completely depends on the prison guard’s subjective assessment of the prisoner’s rehabilitation ability.

D. Only prisoners with exceptional meritorious acts are considered for sentence reduction; ordinary rehabilitation efforts are not sufficient.

Appendix

Table 35: The instruction and an example of the Bias Detection task.

INSTRUCTION: Read the following multiple-choice question and select one or more correct answers. Only select the answer(s) without any explanation.

QUESTION: In the first week of the new school year, T is the class monitor assigned by the homeroom teacher to collect personal information from classmates to create an electronic contact list. T creates an online survey requesting basic information such as Full name, Date of birth, Address, Parents’ phone number, and Personal email address (for receiving study materials). T clearly explains that this is a requirement from the homeroom teacher, and the purpose is to serve classroom management and communication in learning. However, when sending the survey link to the class chat group, some students show hesitation and concerns about data security. A class member asks T: "What if our personal information gets leaked?" Which action correctly reflects the above situation?

ANSWER OPTIONS:

A. Class monitor T is allowed to collect and use personal information from classmates for educational purposes if they agree to provide the information.

B. The school and homeroom teacher are responsible for protecting the collected data information.

C. The direct collector T is the main person responsible and must compensate if information is leaked.

D. Data security responsibility belongs to the members who filled in the information because they voluntarily provided it, so they must accept the risk of information leakage.

Appendix

Table 36: The instruction and an example of the Privacy & Data Protection task.

INSTRUCTION: Read the following multiple-choice question and select the correct answer. Only select the answer without any explanation.

QUESTION: During a traffic incident, Q was weaving recklessly and collided with H. Q not only refused to apologize but also aggressively attacked H. Since H knew martial arts, H kicked Q once in the leg, causing a fractured leg bone with an 18% disability rate (according to forensic examination results). As a result, H was prosecuted for the crime of "Intentionally causing injury," so H sought help from lawyer A for defense. Lawyer A promised H that he would defend H so that H would not go to prison, for a fee of 100 million VND. Subsequently, the People’s Court of District T sentenced H to 2 years in prison with a suspended sentence for the crime of "Intentionally causing injury." However, this verdict was appealed by the People’s Procuracy of District T, and the Provincial Court, after appellate trial, sentenced H to 2 years of imprisonment for the crime of "Intentionally causing injury." H’s family came to lawyer A to demand a refund. Lawyer A refused to refund the money and said, "The lawyer has properly fulfilled the contract and will not refund the money." What is your opinion on lawyer A’s attitude and conduct? Analyze clearly why.

ANSWER OPTIONS:

A. Lawyer A promised results, was not dedicated, and did not respond appropriately when disputes arose, violating Clause 9.8 of Article 9, Rule 2, Rule 5, and Rule 12.3.

B. Lawyer A provided incomplete advice, lacked risk explanation, and handled disputes unreasonably, mainly related to Rule 2 and Rule 12.3, but did not violate Rule 5.

C. Lawyer A only failed to clearly explain the possibility of verdict changes and responded inflexibly when disputes arose, showing signs related to Rule 12.3 but not falling under Clause 9.8 of Article 9.

D. Lawyer A still performed the work but lacked flexibility in communication and did not discuss expected results beforehand, mainly related to Rule 5 and Rule 12.3 but not violating Rule 2.

Appendix

Table 37: The instruction and an example of the Ethical Consistency Assessment task.

INSTRUCTION: Evaluate whether this contract clause is fair to both parties. Select the most correct answer from the options A, B, C, and D provided. Only select the answer without any explanation.

QUESTION: In the contract between the DELIVERY SERVICE OFFICE and the PARTY REQUESTING DELIVERY SERVICES, how does the clause regarding the procedure for executing delivery stipulate, and which party does it favor? Party A must deliver documents within 24 hours for requests from the Civil Judgment Enforcement Agency and 48 hours for requests from the Court or People’s Procuracy. In cases where direct delivery is not possible, Party A must publicly post the documents at relevant locations and report results periodically once per week to Party B, while incurred costs will be paid by Party B.

ANSWER OPTIONS:

A. Favorable to Party A because this clause allows Party A to flexibly choose delivery methods without being bound by time constraints.

B. Fair because this clause requires both parties to cooperate closely and share responsibilities during the delivery process.

C. Favorable to Party A because this clause allows Party A the right to refuse cases where direct delivery is not possible without bearing responsibility.

D. Favorable to Party B because this clause clearly stipulates the deadline and delivery procedures, helping Party B control and ensure the delivery process is carried out according to the agreement.

Appendix

Table 38: The instruction and an example of the Unfair Contract Detection task.

Language Model Reasoning Evaluation

See all 93 papers

Apr 17, 2026

From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text

The complexity of Vietnam's legal texts presents a significant barrier to public access to justice. While Large Language Models offer a promising solution for legal text simplification, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. This paper introduces a comprehensive dual-aspect evaluation framework to address this need. First, we establish a performance benchmark for four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three key dimensions: Accuracy, Readability, and Consistency. Second, to understand the "why" behind these performance scores, we conduct a large-scale error analysis on a curated dataset of 60 complex Vietnamese legal articles, using a novel, expert-validated error typology. Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy, while models like Claude 3 Opus achieve high Accuracy scores that mask a significant number of subtle but critical reasoning errors. The error analysis pinpoints Incorrect Example and Misinterpretation as the most prevalent failures, confirming that the primary challenge for current LLMs is not summarization but controlled, accurate legal reasoning. By integrating a quantitative benchmark with a qualitative deep dive, our work provides a holistic and actionable assessment of LLMs for legal applications.

Van-Truong Le

University of Science, VNUHCM Vietnam National University Hanoi Open University Ho Chi Minh, Vietnam · Vietnam National University · Hanoi Open University

LLM Evaluation Legal Document Analysis Legal Reasoning

Ask PDF

Sep 30, 2026

ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations

Trustworthy Legal AI requires systems that can answer legal questions while grounding their responses in authoritative sources. However, existing Vietnamese legal benchmarks provide limited coverage of real-world legal consultations. We introduce ViLegalExpert, a large-scale benchmark constructed from authentic citizen--lawyer consultations, containing over 172K questions across 34 legal domains, together with professional answers and expert-verified legal evidence. ViLegalExpert supports legal information retrieval, extractive QA, and abstractive QA. Experiments with representative retrieval methods and language models reveal substantial challenges in evidence retrieval and grounded answer generation. While pretrained models perform strongly on QA, hybrid retrieval achieves the best retrieval performance. These results demonstrate the difficulty of mapping naturally expressed legal questions to authoritative provisions and establish ViLegalExpert as a challenging benchmark for reliable Vietnamese Legal AI.

Dat Tien Nguyen, Nghia Hieu Nguyen, Anh Thi-Hoang Nguyen +3

University of Information Technology · Vietnam National University, Ho Chi Minh city, Viet Nam

Legal QA Legal NLP Legal IR

Ask PDF

Apr 30, 2026

ViLegalNLI: Natural Language Inference for Vietnamese Legal Texts

In this article, we introduce ViLegalNLI, the first large-scale Vietnamese Natural Language Inference (NLI) dataset specifically constructed for the legal domain. The dataset consists of 42,012 premise-hypothesis pairs derived from official statutory documents and annotated with binary inference labels (Entailment and Non-entailment). It covers multiple legal domains and reflects realistic legal reasoning scenarios characterized by structured logic, conditional clauses, and domain-specific terminology. To construct ViLegalNLI, we propose a semi-automatic data generation framework that integrates large language models for controlled hypothesis generation and systematic quality validation procedures. The framework incorporates artifact mitigation strategies and cross-model validation to improve annotation reliability and ensure legal consistency. The resulting dataset captures diverse reasoning patterns, including paraphrasing, logical implication, and legally invalid inferences, thereby providing a comprehensive benchmark for Vietnamese legal inference tasks. We conduct extensive experiments on the ViLegalNLI using multilingual models, Vietnamese-specific pretrained language models, and instruction-tuned large language models. The results show that few-shot LLM configurations consistently achieve superior performance, while performance is significantly influenced by hypothesis length, lexical overlap, and reasoning complexity. Cross-domain evaluations further reveal the challenges of generalizing legal inference across distinct legal fields. Overall, ViLegalNLI establishes a foundational benchmark for Vietnamese legal NLI and supports future research in legal reasoning, statutory text understanding, and the development of reliable AI systems for legal analysis and decision support. The dataset is publicly available for research purposes.

Nhung Thi-Hong Duong, Mai Ngoc Ho, Tin Van Huynh +1

University of Information Technology, Ho Chi Minh City, Vietnam. · Vietnam National University, Ho Chi Minh City, Vietnam.

Legal Reasoning Natural Language Inference

Ask PDF

Abstract

Language Model Reasoning Evaluation

Explore similar work

From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text

ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations

ViLegalNLI: Natural Language Inference for Vietnamese Legal Texts