Legal information assistants could help Urdu speakers understand their constitutional rights, but building reliable systems requires test data and evaluation methods suited to the language. We present LEGAL-UQA, a parallel English-Urdu benchmark of 619 question-answer pairs linked to passages from Pakistan's Constitution. Each example includes the question, reference answer, and source passage in both languages. We evaluate two components separately: answering Urdu questions when the source passage is supplied, and retrieving the passage from which each question was generated. We also compare JEV, a decision model used as an automated judge, with human grades. The findings show that model rankings depend on how answer similarity is measured, while a simple word-matching method retrieves source passages more effectively than the embedding methods tested. At the evaluated cutoff, JEV agrees with human grades less often than a baseline that gives every answer full credit. An audit also identifies cases where generated references omit information supported by the source. Together, these findings highlight the need to test retrieval and answering separately, include simple retrieval baselines, and validate automated judges and reference answers with human review when engineering legal QA software for Urdu speakers.
Figures & tables
Fig. 1: Dataset construction and supplied-article QA evaluation. The diagram shows the English and Urdu source documents, question generation, translation, review, and answer evaluation. Answering models receive the question and source passage; JEV also receives the reference and model answer.
Reference similarity
Human grades and JEV judge
Answer set
F1
METEOR
SacreBLEU
Human full (%)
JEV pos. (%)
Agreement (%)
κ
Claude-3.5-Sonnet
66.57
0.690
44.87
99.19
93.55
94.35
0.211
Gemini-1.5-Flash-002
35.17
0.279
18.00
98.39
83.87
83.87
0.063
Gemini-1.5-Pro-002
41.32
0.342
24.60
98.39
91.13
92.74
0.288
Mistral-Large
65.91
0.615
51.45
96.77
84.68
84.68
0.127
mT5
62.22
0.605
42.81
74.19
65.32
79.84
0.527
TABLE I: Supplied-article Urdu answering on 124 validation questions per answer set. F1, METEOR, and SacreBLEU measure similarity to generated references; column maxima are bold. “Human full” is the full-credit rate after the third reviewer resolved grading disagreements; “JEV pos.” is the percentage JEV judged equivalent at the 0.5 cutoff; agreement and Cohen’s κ compare JEV with the human grades.
Method
Query language
R@1 (%)
R@3 (%)
R@5 (%)
text-embedding-3-large
Urdu
53.23
72.58
77.42
mistral-embed
Urdu
29.03
45.97
52.42
BM25
English
62.90
83.87
89.52
BM25
Urdu
71.77
84.68
87.90
TABLE II: Originating-article retrieval for 124 validation questions over 305 passages (Recall@ k , %). Embedding scores come from saved Urdu retrieval results; BM25 scores are calculated from the dataset.
Legal NLP benchmarks are overwhelmingly English-centric, leaving failure modes in morphologically rich, non-Latin-script languages undetected. We introduce UA-Legal-Bench, a five-task benchmark for evaluating large language models on Ukrainian legal reasoning, built from the Unified State Register of Court Decisions (EDRSR) -- one of the world's largest open judicial corpora (99.5 million decisions). The benchmark comprises: (1) case-type classification (4 classes, n=2,000), (2) judgment form classification (4 classes, n=2,000), (3) case-outcome prediction (6 classes, n=800), (4) legal norm extraction (n=1,794), and (5) cause category prediction (22 classes, n=1,871). We evaluate 11 LLMs (3B--675B) from five families under zero-shot and 3-shot prompting via AWS Bedrock with 158K API calls. Our results reveal sharply task-dependent few-shot effects: few-shot prompting improves judgment form classification by up to +38.6 pp but has mixed effects on outcome prediction. We show that accuracy is misleading on imbalanced legal tasks: the model with highest COP accuracy (62%) is a majority-class predictor (macro-F1: 23%), while the genuinely best model scores only 44% macro-F1. Within-family scaling analysis reveals that 8B models can match frontier performance on surface-level tasks but scaling thresholds vary dramatically across families. We release all data, prompts, and model predictions.
The rapid adoption of Large Language Models (LLMs) has raised important concerns about the factual reliability of their outputs, particularly in low-resource languages such as Urdu. Existing automated fact-checking systems are predominantly developed for English, leaving a significant gap for the more than 200 million Urdu speakers worldwide. In this work, we present UrduFactBench and UrduFactQA, two novel hand-annotated benchmarks designed to enable fact-checking and factual consistency evaluation in Urdu. While UrduFactBench focuses on claim verification, UrduFactQA targets the factuality of LLMs in question answering. These resources, the first of their kind for Urdu, were developed through a multi-stage annotation process involving native Urdu speakers. To complement these benchmarks, we introduce UrduFactCheck, a modular fact-checking framework that incorporates both monolingual and translation-based evidence retrieval strategies to mitigate the scarcity of high-quality Urdu evidence. Leveraging these resources, we conduct an extensive evaluation of twelve LLMs and demonstrate that translation-augmented pipelines consistently enhance performance compared to monolingual ones. Our findings reveal persistent challenges for open-source LLMs in Urdu and underscore the importance of developing targeted resources. All code and data are publicly available at https://github.com/mbzuai-nlp/UrduFactCheck.
Legal domains in high-resource languages like English have widely adopted artificial intelligence for legal question answering. However, data scarcity in low resource languages such as Nepali has limited the training of large language models on Nepali legal texts. This study presents the first application of a Retrieval Augmented Generation based model for Nepali legal question answering using case laws extracted from the Nepal Kanun Patrika digital archive. Using BM25 on chunked documents, the approach achieved a top precision at one of 91 percent, and up to 75 percent with the multilingual E5 large model. Evaluation of generated answers showed 74 percent groundedness, 85 percent truthfulness according to an automated judge model, and 84 percent human evaluated truthfulness when using BM25 document retrieval, with a 92 percent successful answer generation rate. These results demonstrate that the RAG pipeline can effectively address the gap in legal question answering for low resource languages and provide a foundation for reliable AI systems in the Nepali legal domain.
Samir Wagle, Abiral Adhikari, Reewaj Khanal +4
Information and Language Processing Research Lab Kathmandu University , Dhulikhel, Kavre, Nepal · School of Law Kathmandu Universit; Dhulikhel, Kavre, Nepal · Dublin City University Dublin, Ireland