Recent multilingual large language models achieve strong translation performance, yet the internal mechanisms responsible for low-resource multilingual translation remain insufficiently understood. In this paper, we study translation circuits in multilingual decoder-only LLMs for Hindi-to-Urdu and Hindi-to-Sindhi (Devanagari). We propose TITAN, a mechanistic framework that combines causal intervention, sparse feature decomposition, and targeted preference optimization to identify and optimize crucial translation components in the transformer. Unlike conventional full-parameter fine-tuning, TITAN restricts optimization to sparse, causally important circuits, enabling scalable and parameter-efficient adaptation while updating fewer than 5% of model parameters. Experimental results show that TITAN achieves BLEU improvements of 33.75% on Hindi-to-Urdu and 45.86% on Hindi-to-Sindhi over the strongest multilingual baselines. Human expert evaluation further confirms substantial gains in adequacy and fluency across both language pairs. Additional analyses reveal that multilingual translation is implemented through sparse, structured circuits operating over a shared semantic subspace rather than through an English-centric pivot representation. Overall, our findings demonstrate that mechanistic circuit discovery can directly improve low-resource multilingual translation while providing interpretable insights into multilingual transformer behavior.
Figures & tables
Pair
Split
#Sent
Hindi
Target Language
Avg. Len.
#Tokens
#Types
#Tokens
#Types
Hi–Sd
SFT Train
20K
404.6k
36.1K
398.2K
49.7K
20.1
DPO Train
15K
302.8K
30.5K
297.8K
41.6K
20.0
Contrastive-From SFT
10K
220.6K
15.8K
217.6K
20.5K
21.9
Hi–Ur
SFT Train
30K
562.3K
47.3K
599.2K
43.7K
19.4
DPO Train
30K
567.1K
47.4K
603.1K
43.8K
19.5
Table 1: Corpus statistics for the Hindi–Sindhi (Devanagari) and Hindi–Urdu parallel corpora used in TITAN. Each corpus is partitioned into SFT, DPO, and contrastive analysis splits. We sample a 10K contrastive subset from the SFT split of each language pair to construct contrastive-based intervention prompts for causal translation circuit identification. Hi–Sd : Hindi–Sindhi (Devanagari); Hi–Ur : Hindi–Urdu. Avg. Len. denotes the mean number of whitespace-tokenized tokens per sentence.
Hindi–Urdu
Hindi–Sindhi (Devanagari)
H → U
U → H
H → S
S → H
Models
BL
CF2
TER
COM
BL
CF2
TER
COM
BL
CF2
TER
COM
BL
CF2
TER
COM
ZS
NLLB-200
0.01
0.09
105.65
57.08
0.31
0.61
108.63
55.39
0.44
1.05
101.23
41.34
2.31
10.58
91.36
43.08
IndicTrans2
4.33
25.19
80.69
62.85
2.12
23.63
87.16
60.57
0.59
3.08
119.88
41.76
1.15
7.38
109.90
55.16
Mistral-3
0.32
2.40
103.46
33.70
0.19
2.56
102.95
30.12
0.17
1.92
296.8
33.43
0.68
5.21
238.11
27.63
LLaMA-3.1
1.46
3.78
117.57
43.83
0.11
4.20
100.79
44.42
0.51
3.63
273.22
37.52
1.21
6.93
250.01
33.43
Table 2: Translation performance comparison across Hindi-Urdu and Hindi-Sindhi (Devanagari) under zero-shot (ZS), supervised fine-tuning (SFT), and Direct Preference Optimization (DPO) settings. TITAN consistently outperforms multilingual translation baselines across both language pairs, demonstrating the effectiveness of sparse causal translation circuits for low-resource multilingual translation. All CIRCE and TITAN results use Gemma-2 9B as the underlying base model. BL : BLEU, CF2 : ChrF2, TER : Translation Edit Rate, COM : COMET.
Setting
Direction
BLEU
chrF++
TER
COMET
Hindi–English
SFT
Hi → En
27.96
52.36
70.24
73.71
DPO
Hi → En
29.33
54.37
66.35
75.43
CIRCE
Hi → En
33.48
58.11
66.34
76.39
TITAN
Hi → En
34.21
60.04
60.22
76.72
SFT
En → Hi
26.08
50.88
70.49
78.15
Table 3: Cross-lingual generalization on unseen high-resource Hindi–English and English–French translation tasks. TITAN consistently outperforms SFT and DPO baselines, using Gemma-2 9B as the base model; CIRCE applies targeted SFT, while TITAN further applies DPO to the same identified components.
Setting
Direction
BLEU
chrF++
TER
COMET
Hindi–Tamil
SFT
Hi → Ta
22.61
40.66
81.66
77.20
DPO
Hi → Ta
23.01
42.87
77.20
79.00
CIRCE
Hi → Ta
25.04
49.46
75.80
81.20
TITAN
Hi → Ta
26.71
50.60
72.20
82.70
English–Irish
Table 4: Performance on additional typologically diverse low-resource translation tasks. TITAN consistently outperforms SFT, DPO, and CIRCE on Hindi–Tamil and English–Irish, demonstrating that the observed gains extend beyond the closely related Indic language pairs used for primary evaluation.
Figure 1: Progressive knockout analysis for Hindi → Sindhi translation. CIRCE-identified components produce substantially larger degradation than randomly selected components.
Direction
System
Adequacy
Fluency
Hindi → Urdu
Gemma-2 (SFT)
Baseline
33.3
20.0
Gemma-2 (DPO)
Baseline
36.7
53.3
CIRCE
Ours
56.7
56.7
Titan
Ours
76.7
86.7
Hindi → Sindhi
Table 5: Human expert evaluation (% of ratings ≥4 on a 1–5 scale) for adequacy and fluency across Hindi–Urdu and Hindi–Sindhi translation tasks. Best results per direction are highlighted in bold.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Perturbation Type
Direction
X+ / X− Example
Ablated Component
Target Nullification
Hi → Sd
Hindi: ... → Sindhi:
Target language identity
Hindi: ... → Nothing:
Hi → Ur
Hindi: ... → Urdu:
Target language identity
Hindi: ... → Nothing:
Action Distortion
Hi → Sd
Hindi: ... → Sindhi:
Source-side instruction frame
Eat the following: ... → Sindhi:
Appendix
Table 6: Contrastive perturbation taxonomy with representative examples. Each X− alters a single semantic component of the directive while preserving structural isomorphism with X+ .
Role
Tokens
Function
IND
(Indicator)
Hindi , Sindhi , Urdu , → , :
Translation direction and structural scaffold
SRC
(Source)
Content tokens of the source sentence
Semantic payload
TGT
(Target)
First generated token(s) at query position
Translation output signal
Appendix
Table 7: Token role definitions used in attention and SAE analyses.
Figure 2: Progressive knockout analysis for Hindi → Urdu translation. CIRCE-identified components produce substantially larger degradation than randomly selected components.
Figure 3: Dominant CIRCE-identified attention heads for Hindi → Sindhi translation. Different heads specialize in structural anchoring, source-language identification, translation-direction routing, and target-language commitment.
Figure 4: Dominant CIRCE-identified attention heads for Hindi → Urdu translation. Compared to Hindi → Sindhi, the Urdu circuit exhibits broader mid-layer routing behavior associated with script-conversion complexity.
Figure 5: MLP representation dynamics for Hindi → Sindhi translation. Top: source-language reception behavior. Middle: target-language generation behavior. Bottom: latent pivot analysis across Hindi, Sindhi, and English representation directions.
Figure 6: MLP representation dynamics for Hindi → Urdu translation. Similar to Hindi → Sindhi, translation behavior evolves through shared Indic semantic directions rather than explicit English pivots.
Figure 7: Sparse feature analysis for Hindi → Sindhi translation. Top: layer-wise distribution of Hindi-specific and Sindhi-specific SAE features. Middle: mean feature selectivity across layers. Bottom: joint selectivity–lift feature space highlighting sparse causally relevant translation features.
Figure 8: Sparse feature analysis for Hindi → Urdu translation. Similar to Hindi → Sindhi, causally important translation behavior is concentrated within a sparse subset of high-lift features.
Large Language Models (LLMs) offer new potential for translation tasks but often experience performance degradation when handling low-resource languages. To address this limitation, we propose an approach for fine-tuning LLMs on a low-resource language, Kupang Malay. Our approach involves designing a set of instructions by leveraging explicit lexical and semantic features from a bilingual dictionary, and introducing Continual Instruction Tuning (CIT), a training paradigm that enables iterative instruction-based training. Experimental results demonstrate that our model, named Lius, yields notable improvements over standard instruction-tuned models by outperforming 4-6 points, and surpassing both Neural Machine Translation (NMT) and Multilingual LLM models by 10-13 points on several evaluation metrics. These findings highlight the potential of our approach to mitigate the reliance on large-scale parallel data in low-resource language translation.
Large Language Models (LLMs) have achieved strong performance across many downstream tasks, yet their effectiveness in extremely low-resource machine translation remains limited. Standard adaptation techniques typically rely on large-scale parallel data or extensive fine-tuning, which are infeasible for the long tail of underrepresented languages. In this work, we investigate a more constrained question: in data-scarce settings, to what extent can linguistically similar pivot languages and few-shot demonstrations provide useful guidance for on-the-fly adaptation in LLMs? We study a data-efficient experimental setup that combines linguistically related pivot languages with few-shot in-context examples, without any parameter updates, and evaluate translation behavior under controlled conditions. Our analysis shows that while pivot-based prompting can yield improvements in certain configurations, particularly in settings where the target language is less well represented in the model's vocabulary, the gains are often modest and sensitive to few shot example construction. For closely related or better represented varieties, we observe diminishing or inconsistent gains. Our findings provide empirical guidance on how and when inference-time prompting and pivot-based examples can be used as a lightweight alternative to fine-tuning in low-resource translation settings.
Neural machine translation (NMT) from Chinese to low-resource Southeast Asian languages remains severely constrained by the extreme scarcity of clean parallel corpora and the pervasive noise in existing mined data. This chronic shortage not only impedes effective model training but also sustains a large performance gap with high-resource directions, leaving millions of speakers of languages such as Lao, Burmese, and Tagalog with persistently low-quality translation systems despite recent advances in large multilingual models. We introduce Multilingual Expert-Reward Informed Tuning (MERIT), a unified translation framework that transforms the traditional English-centric ALT benchmark into a Chinese-centric evaluation suite for five Southeast Asian low-resource languages (LRLs). Our framework combines language-specific token prefixing (LTP) with supervised fine-tuning (SFT) and a novel group relative policy optimization (GRPO) guided by the semantic alignment reward (SAR). These results confirm that, in LRL{\textrightarrow}Chinese translation, targeted data curation and reward-guided optimization dramatically outperform mere model scaling.