Domain-Specific Quality Estimation for Machine Translation in Low-Resource Scenarios

Namrata Patil Gurav; Akashdeep Ranu; Archchana Sindhujan; Diptesh Kanojia

cs.CLMar 7, 2026

Domain-Specific Quality Estimation for Machine Translation in Low-Resource Scenarios

Authors: Namrata Patil Gurav, Akashdeep Ranu, Archchana Sindhujan, Diptesh Kanojia

Organizations: Institute for People-Centred AI, University of Surrey, United Kingdom

Abstract

Quality Estimation (QE) is essential for assessing machine translation quality in reference-less settings, particularly for domain-specific and low-resource language scenarios. In this paper, we investigate sentence-level QE for English to Indic machine translation across four domains (Healthcare, Legal, Tourism, and General) and five language pairs. We systematically compare zero-shot, few-shot, and guideline-anchored prompting across selected closed-weight and open-weight LLMs. Findings indicate that while closed-weight models achieve strong performance via prompting alone, prompt-only approaches remain fragile for open-weight models, especially in high-risk domains. To address this, we adopt ALOPE, a framework for LLM-based QE that uses Low-Rank Adaptation with regression heads attached to selected intermediate Transformer layers. We also extend ALOPE with recently proposed Low-Rank Multiplicative Adaptation (LoRMA). Our results show that intermediate-layer adaptation consistently improves QE performance, with gains in semantically complex domains, indicating a path toward more robust QE in practical scenarios. We release code and domain-specific QE datasets publicly to support further research.

Figures & tables

Domain	Langs	Train	Test
Healthcare	Hi/Mr/Ta/Gu	13,280	1,660
Legal	Gu/Ta/Te	6,160	770
Tourism	Hi/Mr/Te	13,840	1,730
General	Hi/Mr/Ta/Te/Gu	18,880	2,360

Table 1: Indic-Domain-QE dataset composition, showing domain coverage, language pairs, and the number of instances in the train and test splits aggregated across languages.

Figure 1: Methodological Framework uses open-weight models for (i) prompt-only approaches (zero-shot, few-shot, guideline-anchored), and (ii) ALOPE adaptation with LoRA/LoRMA.

	Layer	$R=32$ , $\alpha=16$						$R=64$ , $\alpha=32$						$R=128$ , $\alpha=32$
		En-Hi	En-Mr	En-Ta	En-Te	En-Gu	Avg	En-Hi	En-Mr	En-Ta	En-Te	En-Gu	Avg	En-Hi	En-Mr	En-Ta	En-Te	En-Gu	Avg
\Block 4-1 General	L $-1$	0.363	-0.026	0.385	0.180	0.190	0.218	0.476	0.155	0.605	0.267	0.435	$\dagger$ 0.388	0.384	0.105	0.524	0.196	0.391	0.320
	L $-7$	0.171	0.157	0.389	0.176	0.212	0.221	0.213	0.059	0.469	0.252	0.371	0.273	0.057	0.057	0.346	0.134	0.320	0.183
	L $-9$	0.123	0.090	0.531	0.163	0.458	0.273	0.311	0.038	0.531	0.279	0.485	0.329	0.051	0.107	0.300	0.061	0.343	0.172
	L $-11$	0.178	0.061	0.492	-0.029	0.484	0.237	0.321	0.034	0.568	-0.021	0.480	0.276	0.308	0.007	0.610	0.292	0.457	0.335
\Block 4-1 Healthcare	L $-1$	0.520	0.025	0.201	NA	0.377	0.281	0.382	0.033	0.248	NA	0.490	0.288	0.346	0.156	0.165	NA	0.452	0.280

Table 2: Spearman’s (

\rho

) scores obtained for ALOPE layer-wise (L) experiments with LoRA across different domains and language pairs. The bolded values represent the highest spearman scores obtained for each language pair in each domain. The (

\dagger

) represents the highest average obtained in each domain. ‘NA’ indicates that data from that specific domain and language pair is unavailable.

Domain	Prompt setting	Model family	En-Hi	En-Mr	En-Ta	En-Te	En-Gu	Average
General	Zero-shot	Closed	0.424	0.597	0.848	0.392	0.924	$\dagger$ 0.637
	Zero-shot	Open	0.390	-0.058	0.772	0.382	0.812	0.460
	Few-shot + Guidelines	Closed	0.475	0.582	0.886	0.238	0.776	0.591
	Few-shot + Guidelines	Open	0.408	0.038	0.832	0.442	0.849	0.514
	Few-shot (No Guidelines)	Closed	0.563	0.314	0.940	-0.031	0.860	0.529
	Few-shot (No Guidelines)	Open	0.418	0.375	0.867	0.486	0.752	0.580

Table 4: Reports best Spearman’s correlation (

\rho

) achieved for each domain and language pair across all evaluated models, including both open- and closed-weight LLMs. The bolded values represent the highest spearman scores obtained for each language pair in each domain. The (

\dagger

) represents the highest average obtained in each domain. Detailed model-wise results are provided in Appendices F , G , and H .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Few-shot QE Prompt (Without Guidelines)

Figure 3: Few-shot QE Prompt (With Guidelines)

Domain	Model	En-Hi		En-Mr		En-Ta		En-Te		En-Gu		Avg
		$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
General	Gemini-1.5-Pro	-0.082	-0.232	0.120	0.159	0.856	0.848	0.191	0.073	0.774	0.924	0.372	0.354
	Gemini-2.5-Pro	0.328	0.424	0.747	0.597	0.715	0.805	0.542	0.392	0.918	0.786	0.650	0.601
	Qwen3-14B	0.319	0.390	-0.159	-0.058	0.730	0.772	0.393	0.328	0.499	0.649	0.356	0.416
	Gemma3-27B	0.079	0.116	-0.259	-0.290	0.776	0.701	0.532	0.382	0.195	0.102	0.265	0.202
	LLaMA-3.2-3B	0.065	0.056	-0.184	-0.164	0.099	0.249	-0.366	-0.381	-0.174	-0.024	-0.112	-0.053

Appendix

Table 7: Zero-shot prompt-only QE performance. Spearman’s (

\rho

) and Pearson’s (

r

) scores are reported for all language pairs. Best

\rho

per language pair in bold .‘NA’ indicates that a language pair is unavailable for that specific domain.

Domain	Model	En-Hi		En-Mr		En-Ta		En-Te		En-Gu		Avg
		$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
General	Gemini-1.5-Pro	-0.004	-0.094	-0.063	0.087	0.881	0.850	-0.319	-0.169	0.859	0.761	0.271	0.287
	Gemini-2.5-Pro	0.625	0.475	0.638	0.582	0.834	0.886	0.255	0.238	0.926	0.776	0.656	0.591
	Qwen3-14B	0.247	0.279	-0.159	-0.058	0.720	0.783	0.498	0.442	0.850	0.849	0.431	0.459
	Gemma3-27B	0.489	0.376	0.041	0.038	0.770	0.832	0.502	0.352	0.799	0.649	0.520	0.449
	LLaMA-3.2-3B	-0.637	-0.527	-0.101	-0.251	0.479	0.329	0.150	0.186	-0.332	-0.200	-0.088	-0.093

Appendix

Table 8: Few-shot with guidelines QE performance. Spearman’s (

\rho

) and Pearson’s (

r

) scores are reported for all language pairs. Best

\rho

per language pair in bold . ‘NA’ indicates that a language pair is unavailable for that specific domain.

Domain	Model	En-Hi		En-Mr		En-Ta		En-Te		En-Gu		Avg
		$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
General	Gemini-1.5-Pro	0.107	0.022	-0.213	-0.103	0.744	0.743	-0.035	-0.185	0.941	0.791	0.309	0.254
	Gemini-2.5-Pro	0.713	0.563	0.447	0.314	0.888	0.940	-0.095	-0.031	0.939	0.860	0.578	0.529
	Qwen3-14B	-0.128	-0.040	-0.235	-0.234	0.761	0.867	0.319	0.235	0.658	0.752	0.275	0.316
	Gemma3-27B	0.177	0.234	0.489	0.375	0.519	0.450	0.636	0.486	0.765	0.627	0.517	0.434
	LLaMA-3.2-3B	0.497	0.418	-0.101	0.014	0.393	0.477	0.223	0.188	0.078	0.228	0.218	0.265

Appendix

Table 9: Few-shot without guidelines QE performance. Spearman’s (

\rho

) and Pearson’s (

r

) scores are reported for all language pairs. Best

\rho

per language pair in bold . ‘NA’ indicates that a language pair is unavailable for that specific domain.

Domain	Prompt setting	En-Hi	En-Mr	En-Ta	En-Te	En-Gu	Avg
\Block 5-1 General	Zero-shot	0.056	-0.164	0.249	-0.381	-0.024	-0.053
	Few-shot + Guidelines	-0.527	-0.251	0.329	0.186	-0.200	-0.093
	Few-shot (No Guidelines)	0.418	0.014	0.477	0.188	0.228	0.265
	ALOPE (LoRA)	0.476	0.157	0.610	0.292	0.485	$\dagger$ 0.404
	ALOPE (LoRMA)	0.392	0.314	0.507	0.168	0.421	0.360
\Block 5-1 Healthcare	Zero-shot	-0.234	0.174	0.406	NA	-0.055	0.073

Appendix

Table 10: Comparison of Spearman’s correlation (

\rho

) achieved by LLaMA-3.2-3B across Prompt-only approaches and ALOPE. The bolded values represent the highest Spearman scores obtained for each language pair in each domain. The (

\dagger

) represents the highest average obtained in each domain. ‘NA’ indicates that a language pair is unavailable for that specific domain.

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

Toward domain-specific machine translation and quality estimation systems
Mar 26, 2026Javad Pourmostafa Roshan SharamiMachine Translation Domain Adaptation
LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Edits
Sep 16, 2026Kathy Hämmerl, Gabriel Bretschner, Joern WuebkerDomain Adaptation Machine Translation
Technical University of Munich · Munich Center for Machine Learning · LILT
CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs
May 15, 2026Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1Small Language Models MT Evaluation
Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland

Mar 26, 2026J/K move · Enter open · S save

Toward domain-specific machine translation and quality estimation systems

Javad Pourmostafa Roshan Sharami

Machine Translation Domain Adaptation MT Evaluation Tokenizer Adaptation In-Context Learning

Open paper AskPDF arXiv

Machine Translation (MT) and Quality Estimation (QE) perform well in general domains but degrade under domain mismatch. This dissertation studies how to adapt MT and QE systems to specialized domains through a set of data-focused contributions. Chapter 2 presents a similarity-based data selection method for MT. Small, targeted in-domain subsets outperform much larger generic datasets and reach strong translation quality at lower computational cost. Chapter 3 introduces a staged QE training pipeline that combines domain adaptation with lightweight data augmentation. The method improves performance across domains, languages, and resource settings, including zero-shot and cross-lingual cases. Chapter 4 studies the role of subword tokenization and vocabulary in fine-tuning. Aligned tokenization-vocabulary setups lead to stable training and better translation quality, while mismatched configurations reduce performance. Chapter 5 proposes a QE-guided in-context learning method for large language models. QE models select examples that improve translation quality without parameter updates and outperform standard retrieval methods. The approach also supports a reference-free setup, reducing reliance on a single reference set. These results show that domain adaptation depends on data selection, representation, and efficient adaptation strategies. The dissertation provides methods for building MT and QE systems that perform reliably in domain-specific settings.

Abstract

MT Evaluation

Explore similar work

Toward domain-specific machine translation and quality estimation systems

LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Edits

CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs