Translation Circuit Discovery and Targeted Optimization for Low-Resource Multilingual Translation

Atai Waris Khan; Abdullah Mazhar; Raghu Kumar; Md Shad Akhtar

cs.CLOct 15, 2026

Translation Circuit Discovery and Targeted Optimization for Low-Resource Multilingual Translation

Authors: Atai Waris Khan, Abdullah Mazhar, Raghu Kumar, Md Shad Akhtar

Organizations: Indraprastha Institute of Information Technology Delhi

Abstract

Recent multilingual large language models achieve strong translation performance, yet the internal mechanisms responsible for low-resource multilingual translation remain insufficiently understood. In this paper, we study translation circuits in multilingual decoder-only LLMs for Hindi-to-Urdu and Hindi-to-Sindhi (Devanagari). We propose TITAN, a mechanistic framework that combines causal intervention, sparse feature decomposition, and targeted preference optimization to identify and optimize crucial translation components in the transformer. Unlike conventional full-parameter fine-tuning, TITAN restricts optimization to sparse, causally important circuits, enabling scalable and parameter-efficient adaptation while updating fewer than 5% of model parameters. Experimental results show that TITAN achieves BLEU improvements of 33.75% on Hindi-to-Urdu and 45.86% on Hindi-to-Sindhi over the strongest multilingual baselines. Human expert evaluation further confirms substantial gains in adequacy and fluency across both language pairs. Additional analyses reveal that multilingual translation is implemented through sparse, structured circuits operating over a shared semantic subspace rather than through an English-centric pivot representation. Overall, our findings demonstrate that mechanistic circuit discovery can directly improve low-resource multilingual translation while providing interpretable insights into multilingual transformer behavior.

Figures & tables

Pair	Split	#Sent	Hindi		Target Language		Avg. Len.
Pair	Split	#Sent	#Tokens	#Types	#Tokens	#Types	Avg. Len.
Hi–Sd	SFT Train	20K	404.6k	36.1K	398.2K	49.7K	20.1
	DPO Train	15K	302.8K	30.5K	297.8K	41.6K	20.0
	Contrastive-From SFT	10K	220.6K	15.8K	217.6K	20.5K	21.9
Hi–Ur	SFT Train	30K	562.3K	47.3K	599.2K	43.7K	19.4
	DPO Train	30K	567.1K	47.4K	603.1K	43.8K	19.5

Table 1: Corpus statistics for the Hindi–Sindhi (Devanagari) and Hindi–Urdu parallel corpora used in TITAN. Each corpus is partitioned into SFT, DPO, and contrastive analysis splits. We sample a 10K contrastive subset from the SFT split of each language pair to construct contrastive-based intervention prompts for causal translation circuit identification. Hi–Sd : Hindi–Sindhi (Devanagari); Hi–Ur : Hindi–Urdu. Avg. Len. denotes the mean number of whitespace-tokenized tokens per sentence.

		Hindi–Urdu								Hindi–Sindhi (Devanagari)
		H $\rightarrow$ U				U $\rightarrow$ H				H $\rightarrow$ S				S $\rightarrow$ H
	Models	BL	CF2	TER	COM	BL	CF2	TER	COM	BL	CF2	TER	COM	BL	CF2	TER	COM
ZS	NLLB-200	0.01	0.09	105.65	57.08	0.31	0.61	108.63	55.39	0.44	1.05	101.23	41.34	2.31	10.58	91.36	43.08
	IndicTrans2	4.33	25.19	80.69	62.85	2.12	23.63	87.16	60.57	0.59	3.08	119.88	41.76	1.15	7.38	109.90	55.16
	Mistral-3	0.32	2.40	103.46	33.70	0.19	2.56	102.95	30.12	0.17	1.92	296.8	33.43	0.68	5.21	238.11	27.63
	LLaMA-3.1	1.46	3.78	117.57	43.83	0.11	4.20	100.79	44.42	0.51	3.63	273.22	37.52	1.21	6.93	250.01	33.43

Table 2: Translation performance comparison across Hindi-Urdu and Hindi-Sindhi (Devanagari) under zero-shot (ZS), supervised fine-tuning (SFT), and Direct Preference Optimization (DPO) settings. TITAN consistently outperforms multilingual translation baselines across both language pairs, demonstrating the effectiveness of sparse causal translation circuits for low-resource multilingual translation. All CIRCE and TITAN results use Gemma-2 9B as the underlying base model. BL : BLEU, CF2 : ChrF2, TER : Translation Edit Rate, COM : COMET.

Setting	Direction	BLEU	chrF++	TER	COMET
Hindi–English
SFT	Hi $\rightarrow$ En	27.96	52.36	70.24	73.71
DPO	Hi $\rightarrow$ En	29.33	54.37	66.35	75.43
CIRCE	Hi $\rightarrow$ En	33.48	58.11	66.34	76.39
TITAN	Hi $\rightarrow$ En	34.21	60.04	60.22	76.72
SFT	En $\rightarrow$ Hi	26.08	50.88	70.49	78.15

Table 3: Cross-lingual generalization on unseen high-resource Hindi–English and English–French translation tasks. TITAN consistently outperforms SFT and DPO baselines, using Gemma-2 9B as the base model; CIRCE applies targeted SFT, while TITAN further applies DPO to the same identified components.

Setting	Direction	BLEU	chrF++	TER	COMET
Hindi–Tamil
SFT	Hi $\rightarrow$ Ta	22.61	40.66	81.66	77.20
DPO	Hi $\rightarrow$ Ta	23.01	42.87	77.20	79.00
CIRCE	Hi $\rightarrow$ Ta	25.04	49.46	75.80	81.20
TITAN	Hi $\rightarrow$ Ta	26.71	50.60	72.20	82.70
English–Irish

Table 4: Performance on additional typologically diverse low-resource translation tasks. TITAN consistently outperforms SFT, DPO, and CIRCE on Hindi–Tamil and English–Irish, demonstrating that the observed gains extend beyond the closely related Indic language pairs used for primary evaluation.

Figure 1: Progressive knockout analysis for Hindi $\rightarrow$ Sindhi translation. CIRCE-identified components produce substantially larger degradation than randomly selected components.

Hindi $\rightarrow$ Urdu
Direction	System	Adequacy	Fluency
Gemma-2 (SFT)	Baseline	33.3	20.0
Gemma-2 (DPO)	Baseline	36.7	53.3
CIRCE	Ours	56.7	56.7
Titan	Ours	76.7	86.7
Hindi $\rightarrow$ Sindhi

Table 5: Human expert evaluation (% of ratings

\geq 4

on a 1–5 scale) for adequacy and fluency across Hindi–Urdu and Hindi–Sindhi translation tasks. Best results per direction are highlighted in bold.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Perturbation Type	Direction	$X^{+}$ / $X^{-}$ Example	Ablated Component
Target Nullification	Hi $\rightarrow$ Sd	Hindi: ... $\rightarrow$ Sindhi:	Target language identity
		Hindi: ... $\rightarrow$ Nothing:
	Hi $\rightarrow$ Ur	Hindi: ... $\rightarrow$ Urdu:	Target language identity
		Hindi: ... $\rightarrow$ Nothing:
Action Distortion	Hi $\rightarrow$ Sd	Hindi: ... $\rightarrow$ Sindhi:	Source-side instruction frame
		Eat the following: ... $\rightarrow$ Sindhi:

Appendix

Table 6: Contrastive perturbation taxonomy with representative examples. Each

X^{-}

alters a single semantic component of the directive while preserving structural isomorphism with

X^{+}

Role	Tokens	Function
IND
(Indicator)	Hindi , Sindhi , Urdu , $\rightarrow$ , :	Translation direction and structural scaffold
SRC
(Source)	Content tokens of the source sentence	Semantic payload
TGT
(Target)	First generated token(s) at query position	Translation output signal

Appendix

Table 7: Token role definitions used in attention and SAE analyses.

Figure 2: Progressive knockout analysis for Hindi $\rightarrow$ Urdu translation. CIRCE-identified components produce substantially larger degradation than randomly selected components.

Figure 3: Dominant CIRCE-identified attention heads for Hindi $\rightarrow$ Sindhi translation. Different heads specialize in structural anchoring, source-language identification, translation-direction routing, and target-language commitment.

Figure 4: Dominant CIRCE-identified attention heads for Hindi $\rightarrow$ Urdu translation. Compared to Hindi $\rightarrow$ Sindhi, the Urdu circuit exhibits broader mid-layer routing behavior associated with script-conversion complexity.

Figure 5: MLP representation dynamics for Hindi $\rightarrow$ Sindhi translation. Top: source-language reception behavior. Middle: target-language generation behavior. Bottom: latent pivot analysis across Hindi, Sindhi, and English representation directions.

Figure 6: MLP representation dynamics for Hindi $\rightarrow$ Urdu translation. Similar to Hindi $\rightarrow$ Sindhi, translation behavior evolves through shared Indic semantic directions rather than explicit English pivots.

Figure 7: Sparse feature analysis for Hindi $\rightarrow$ Sindhi translation. Top: layer-wise distribution of Hindi-specific and Sindhi-specific SAE features. Middle: mean feature selectivity across layers. Bottom: joint selectivity–lift feature space highlighting sparse causally relevant translation features.

Figure 8: Sparse feature analysis for Hindi $\rightarrow$ Urdu translation. Similar to Hindi $\rightarrow$ Sindhi, causally important translation behavior is concentrated within a sparse subset of high-lift features.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malay
Jun 10, 2026Joanito Agili Lopo, Yunita Sari, Guntur Budi HerwantoLLM Fine-Tuning Low-Resource MT
Department of Computer Science and Electronics Universitas Gadjah Mada
Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?
Mar 17, 2026Aishwarya Ramasethu, Niyathi Allu, Rohin Garg +2Machine Translation Low-Resource MT
Prediction Guard · Independent · Scale AI +1
MERIT: Multilingual Expert-Reward Informed Tuning for Chinese-Centric Low-Resource Machine Translation
Apr 6, 2026Zhixiang Lu, Chong Zhang, Chenyu Xue +4Machine Translation Low-Resource MT
Xi’an Jiaotong-Liverpool University

Jun 10, 2026J/K move · Enter open · S save

Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malay

Joanito Agili Lopo, Yunita Sari, Guntur Budi Herwanto

Department of Computer Science and Electronics Universitas Gadjah Mada

LLM Fine-Tuning Low-Resource MT Instruction Tuning Low-Resource Language Processing

Open paper AskPDF arXiv

Large Language Models (LLMs) offer new potential for translation tasks but often experience performance degradation when handling low-resource languages. To address this limitation, we propose an approach for fine-tuning LLMs on a low-resource language, Kupang Malay. Our approach involves designing a set of instructions by leveraging explicit lexical and semantic features from a bilingual dictionary, and introducing Continual Instruction Tuning (CIT), a training paradigm that enables iterative instruction-based training. Experimental results demonstrate that our model, named Lius, yields notable improvements over standard instruction-tuned models by outperforming 4-6 points, and surpassing both Neural Machine Translation (NMT) and Multilingual LLM models by 10-13 points on several evaluation metrics. These findings highlight the potential of our approach to mitigate the reliance on large-scale parallel data in low-resource language translation.

Abstract

Multilingual Language Models

Explore similar work

Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malay

Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?

MERIT: Multilingual Expert-Reward Informed Tuning for Chinese-Centric Low-Resource Machine Translation