Cross Lingual Transfer in Tulu Legal Comprehension: Script-Dependent Improvement and RAG-Induced Knowledge Conflict

Sindhu Shetty; Spurthi Setty; Natan Vidra

cs.CLAug 13, 2026

Cross Lingual Transfer in Tulu Legal Comprehension: Script-Dependent Improvement and RAG-Induced Knowledge Conflict

Authors: Sindhu Shetty, Spurthi Setty, Natan Vidra

Abstract

Low-resource languages without an adequate training corpus often use a related, higher-resource language as a scaffold for comprehension. Still, there is a need to develop rigorous evaluation methods to identify when models fail in cross lingual low-resource environments. Using the legal domain as a backdrop, three models (Llama3, Hex-1, Sarvam) were tested on the ability to classify legal complaints written in a low resource Dravidian language (Tulu). Transliterating queries across Dravidian scripts allowed models to gain a preliminary understanding of speakers' complaints without the use of wide scale training, though the level of comprehension was heavily script dependent (with Kannada - another relatively low-resource language - producing the strongest positive trend). Retrieving from a corpus of Kannada legal papers across a RAG framework caused mixed results. Some models had a weak positive trend in comprehension under certain conditions, but when models failed, it was often across two axes: fact substitution (fixating on specific passage excerpts that skewed reasoning) and confabulation (hallucination that had no basis in either query or corpus). Within low resource domains, results identify the model's parsing of information and subsequent reasoning as the source of reasoning failure, rather than corpus contents. Script-dependent comprehension and RAG robustness also seem to travel together. This is further supported by the reasoning-trace analysis and a statistical-honesty framework deployed - techniques that are more broadly applicable to low-resource multilingual RAG evaluation.

Figures & tables

Condition	Script	Priming	RAG
A	English gloss	–	✓
B	Romanized Tulu	–	✓
C	Romanized Tulu	✓	✓
D	Kannada script	✓	✓
E	Kannada script	✓	–
F	Kannada script	–	✓

Table 1: Experimental conditions, defined by query script, presence of linguistic priming, and use of retrieval-augmented generation (RAG). Conditions with priming (C, D, E, K, L, M) were each run at both sparse and extensive tiers.

Figure 1: Hits@1 across five conditions per model (

n=60

each): bare comprehension (H), no-RAG full-priming ceiling (E_full), RAG without priming (F), RAG with full priming (D_full), English-gloss query with RAG (A). Only Llama3 drops below its own no-RAG ceiling once RAG is added. Full 19-condition breakdown: Appendix D .

Figure 2: Query language under RAG: English-gloss queries (Condition A) substantially outperform Kannada-script Tulu queries under identical retrieval (Condition D), for all three models (Fisher’s exact,

p<.001

in every comparison). This is the one RAG-related effect that generalizes across models.

Query (gloss)	Reasoning trace
“I don’t know which court I should go to”	“The situation explicitly mentions ‘court’, indicating a problem related to the legal system.”
“Someone is living on my land without asking and won’t leave”	“…classic housing/trespass issue.”

Table 2: Sarvam, Condition H, ids 14 and 17. Row 1: human chosen and predicted category both Courts/Legal-System, reasoning names the specific recognized term rather than a generic paraphrase. Row 2: human chosen - Land/Property, predicted - Housing. The model parses the core action but assigns an adjacent category, indicating partial rather than absent comprehension.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix


ID	Primary	Secondary	English Gloss	Romanized Tulu
1	Housing	Consumer/Money/Debt	My landlord did not give back my money	enna dhani kasn vapas kortijer.
2	Work/Employment	–	My employer did not pay me this month	enna dhani i tingolda sanbalan kortijer.
3	Torts/Individuals	–	Someone broke my fence and did not fix it	era enna belin hal maltd nanla sari maltijer.
4	Consumer/Money/Debt	–	I gave someone money and they disappeared	yan erega kas kori bokka akulu tappayer.
5	Land/Property	–	My neighbor took my land	enna baritakulu enna jagan detter.

Appendix

Table 3: Full 60-sentence query set: human chosen LIST categories, English gloss, and Romanized Tulu.


ID	\kannadafont Kannada
1	\kannadafont ಎನ್ನ ಧನಿ ಕಾಸ್ನ್ ವಾಪಸ್ ಕೊರ್ತಿಜೇರ್.
2	\kannadafont ಎನ್ನ ಧನಿ ಈ ತಿಂಗೋಲ್ದ ಸಂಬಳನ್ ಕೊರ್ತಿಜೇರ್.
3	\kannadafont ಏರಾ ಎನ್ನ ಬೇಲಿನ್ ಹಾಳ್ ಮಲ್ತ್ದ್ ನನ್ಲಾ ಸರಿ ಮಲ್ತಿಜೇರ್.
4	\kannadafont ಯಾನ್ ಏರೇಗಾ ಕಾಸ್ ಕೊರಿ ಬೊಕ್ಕ ಅಕುಲು ತಪ್ಪಾಯೇರ್.
5	\kannadafont ಎನ್ನ ಬರೀತಕುಲು ಎನ್ನ ಜಾಗನ್ ದೆತ್ತೇರ್.

Appendix

Table 4: Full 60-sentence query set in native script (Kannada).

\kannadafont
\kannadafont Kannada	Romanized	Meaning
\kannadafont ಆಂಜೊ	aanjo	man
\kannadafont ಆಲ್	al	she
\kannadafont ಬಾಲೆ	baale	child
\kannadafont ಮೀನ್	meen	fish
\kannadafont ಪೋಪಿನಿ	popini	to go

Appendix

Table 5: Safe-tier glossary items (grammatical/function vocabulary, no category-revealing content).

\kannadafont
\kannadafont Kannada	Romanized	Meaning
\kannadafont ಯಜಮಾನ	yejamaana	landlord/owner
\kannadafont ಧನಿ	dhani	landlord/employer (authority figure over you)
\kannadafont ಕಾಸು	kas	money
\kannadafont ಜಾಗ	jagga	land
\kannadafont ಇಲ್ಲ್	ille	house

Appendix

Table 6: Leaky-tier glossary items (content vocabulary overlapping with LIST category terms).

		Llama3		Hex-1		Sarvam
Script	Tier	Hits@1	Hits@3	Hits@1	Hits@3	Hits@1	Hits@3
Romanized	bare	16.7%	46.7%	16.7%	65.0%	33.3%	61.7%
	sparse	13.3%	43.3%	18.3%	55.0%	30.0%	60.0%
	full	31.7%	56.7%	25.0%	61.7%	40.0%	71.7%
Kannada	bare	46.7%	76.7%	41.7%	68.3%	48.3%	76.7%
	sparse	51.7%	75.0%	45.0%	76.7%	46.7%	78.3%

Appendix

Table 7: Comprehension accuracy by script and priming tier, no RAG, all three models (

n=60

/cell).

Figure 3: Hits@1 by script and priming tier, per model (

n=60

/bar).

Condition	Llama3 H1	Llama3 H3	Hex-1 H1	Hex-1 H3	Sarvam H1	Sarvam H3	n
A	61.7%	95.0%	86.7%	98.3%	91.7%	100.0%	60
B	16.7%	56.7%	16.7%	50.0%	28.3%	63.3%	60
C_sparse	16.7%	46.7%	11.7%	46.7%	31.7%	61.7%	60
C_full	18.3%	53.3%	25.0%	63.3%	45.0%	73.3%	60
D_sparse	26.7%	66.7%	41.7%	68.3%	55.0%	75.0%	60
D_full	23.3%	68.3%	46.7%	80.0%	60.0%	80.0%	60

Appendix

Table 8: Hits@1 / Hits@3 across all 19 conditions, per model (

n=60

each).

Figure 4: Hits@1 across all 19 conditions, per model (

n=60

/bar). Each panel’s dashed line marks that model’s own no-RAG, full-priming ceiling (E_full).

Baseline comprehension (binomial test vs. chance/majority)
Comparison (Hits@1)	Llama3	Hex-1	Sarvam
H (Kannada bare) vs. chance (12.5%) / majority (25%)	$p<.001$ / $p<.001$	$p<.001$ / $p<.001$	$p<.001$ / $p<.001$
D_full vs. majority-class (25%)	$p=.665$ (ns)	$p<.001$ ***	$p<.001$ ***
Priming effects (Fisher’s exact)
E_full vs. H (Kannada full-priming effect)	$p=.200$ (ns)	$p=.067$ (ns)	$p=.064$ (ns)
L_full vs. J (Malayalam full-priming effect)	$p=.047$ *	$p=.013$ *	$p=.001$ **

Appendix

Table 9: Fisher’s exact and binomial tests for pairwise condition comparisons at Hits@1,

n=60

/condition. Grouped by comparison type: baseline comprehension checks against chance/majority guessing, priming-effect checks (no RAG), and RAG-effect checks (Kannada bare, sparse, and full priming, plus query-language effect under RAG).

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

Model	Bare	Sparse prime	Full prime	RAG only	RAG+sparse	RAG+full
Llama3	46.7¿16.7***	51.7¿13.3***	60.0¿31.7**	25.0¿16.7 (ns)	26.7¿16.7 (ns)	23.3¿18.3 (ns)
Hex-1	41.7¿16.7**	45.0¿18.3**	60.0¿25.0***	38.3¿16.7*	41.7¿11.7***	46.7¿25.0*
Sarvam	48.3¿33.3 (ns)	46.7¿30.0 (ns)	66.7¿40.0**	53.3¿28.3**	55.0¿31.7*	60.0¿45.0 (ns)

Appendix

Table 10: Kannada-script vs. Romanized-script comprehension (Hits@1, %) across all six priming/RAG tiers, per model (

n=60

/condition). Values shown as Kannada%

>

Romanized%. Fisher’s exact test, same methodology as Table 9 . Computed from condition-level results in Table 8 ; no new model calls were made.

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

Model	Tier	E-right/D-wrong	D-right/E-wrong	Ratio
Llama3	full	26	4	6.5:1
Llama3	sparse	20	5	4.0:1
Hex-1	full	10	2	5.0:1
Hex-1	sparse	8	6	1.3:1
Sarvam	full	7	3	2.3:1
Sarvam	sparse	4	9	0.4:1

Appendix

Table 11: Disagreement-level analysis: among cases where full/sparse-priming no-RAG (E) and RAG (D) predictions diverge, how often each condition was uniquely correct, per model and priming tier (

n=60

/condition).

Judge language	Hits@1	Hits@3	Passages retained	No-RAG fallback
Kannada script	25.0% (5/20)	65.0% (13/20)	2.85 / 3	0/20
English gloss	60.0% (12/20)	80.0% (16/20)	1.65 / 3	3/20

Appendix

Table 12: CRAG relevance-verification results, Llama3, Kannada-script D-condition queries (

n=20

). Fisher’s exact test on Hits@1 (English-judge vs. Kannada-judge):

p=.054

. Reference points from the full 60-query set (Table 8 ): D_full (RAG, full priming) = 23.3%; E_full (no-RAG, full priming) = 60.0%.

Case	Case No.	Court	CNR / Date
Sridhar P. v. Ramesh	C.C. No. 5343/2024	4th Addl. Small Causes Court & ACJM, Bengaluru Urban	KABC020187742024, 3 Jun 2025
Amith Ramachandra v. Shantha	P.C.R. No. 3981/2025	41st Addl. Chief Judicial Magistrate, Bengaluru Urban	KABC030166202025, 6 May 2025
Sadashiva Shetty v. Vijaya Bank	R.A. No. 43-2001	Principal Civil Judge (Sr. Div.) & CJM, Udupi	KAUP020002942001, 24 Jan 2009
State v. Sudhakar Pai	C.C. No. 1357/2016	JMFC 2nd Court, Mangaluru	KADK070014602016, 18 Oct 2016
Nikitha A. v. Santhosh Ghatke	Crl.Misc. No. 127/2023	JMFC Traffic Court VI, Bengaluru Urban	KABC080042862023, 4 Jul 2025
Akkamma v. Ramesh Patil	Cri.Misc. No. 17/2018	Principal Civil Judge & JMFC, Byadgi, Haveri	KAHV210000762018, 1 Aug 2018

Appendix

Table 13: Court cases named in Appendix C , with case number, deciding court, eCourts CNR identifier, and order date. All are public lower-court filings from ecourtsindia.com .

Explore similar work

Mar 22, 2026

Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models

We investigate how large language models perform on low-resource languages by benchmarking eight LLMs across five experimental conditions in English, Kazakh, and Mongolian. Using 50 hand-crafted questions spanning factual, reasoning, technical, and culturally grounded categories, we evaluate 2,000 responses on accuracy, fluency, and completeness. We find a consistent performance gap of 13.8-16.7 percentage points between English and low-resource language conditions, with models maintaining surface-level fluency while producing significantly less accurate content. Cross-lingual transfer-prompting models to reason in English before translating back-yields selective gains for bilingual architectures (+2.2pp to +4.3pp) but provides no benefit to English-dominant models. Our results demonstrate that current LLMs systematically underserve low-resource language communities, and that effective mitigation strategies are architecture-dependent rather than universal.

Abdul-Salem Beibitkhan

North American University

Low-Resource Language Processing Multilingual Language Model Evaluation Cross-Lingual Transfer LLM Prompting

Ask PDF

Apr 20, 2026

Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answering

Legal domains in high-resource languages like English have widely adopted artificial intelligence for legal question answering. However, data scarcity in low resource languages such as Nepali has limited the training of large language models on Nepali legal texts. This study presents the first application of a Retrieval Augmented Generation based model for Nepali legal question answering using case laws extracted from the Nepal Kanun Patrika digital archive. Using BM25 on chunked documents, the approach achieved a top precision at one of 91 percent, and up to 75 percent with the multilingual E5 large model. Evaluation of generated answers showed 74 percent groundedness, 85 percent truthfulness according to an automated judge model, and 84 percent human evaluated truthfulness when using BM25 document retrieval, with a 92 percent successful answer generation rate. These results demonstrate that the RAG pipeline can effectively address the gap in legal question answering for low resource languages and provide a foundation for reliable AI systems in the Nepali legal domain.

Samir Wagle, Abiral Adhikari, Reewaj Khanal +4

Information and Language Processing Research Lab Kathmandu University , Dhulikhel, Kavre, Nepal · School of Law Kathmandu Universit; Dhulikhel, Kavre, Nepal · Dublin City University Dublin, Ireland

Retrieval-Augmented Generation Legal NLP Legal QA Low-Resource Language Processing Knowledge-Intensive QA

Ask PDF

Sep 30, 2026

LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian

Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenge for Low-Resource Languages (LRLs), where standard reference-based metrics fall short. This paper investigates the viability of the "LLM-as-a-Judge" paradigm for Romanian by adapting the Ragas framework using next-generation models (Gemini 2.5 and Gemini 3). We introduce AdminRo-Eval, a curated dataset of Romanian administrative documents annotated by native speakers, to serve as a ground truth for benchmarking automated evaluators. We compare three evaluation methodologies - direct scoring, comparative ranking, and granular decomposition - across metrics for Faithfulness, Answer Relevance, and Context Relevance. Our findings reveal that evaluation strategies must be metric-specific: granular decomposition achieves the highest human alignment for Faithfulness (96% with Gemini 2.5 Pro), while comparative ranking outperforms in Answer Relevance (90%). Furthermore, we demonstrate that while lightweight models struggle with complex reasoning in LRLs, the Gemini 2.5 Pro architecture establishes a robust, transferable baseline for automated Romanian RAG evaluation.

Claudiu Creanga, Liviu P. Dinu

Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania · HLT Research Center, University of Bucharest, Romania · Faculty of Mathematics and Computer Science, University of Bucharest, Romania

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation RAG Evaluation Automated Evaluation Low-Resource Language Processing

Ask PDF