Fine-tuning LLMs with in-context examples (IC-Train) lets translation models adapt to new parallel data at deployment, but it is unclear whether they actually learn the translations of rare terms from those examples. We formalize this capability as in-context term learning (ICTL). We show that standard IC-Train develops this capability only weakly, and propose RareCipherFT, which strengthens it without any lexical resources. RareCipherFT pairs each training target with in-context examples that share a rare term with it. It then replaces that term with a character-shuffled cipher in both the context and the reference, so the term can only be recovered by copying from context. The ciphered fraction of training data is annealed to preserve translation quality. Across seven translation directions between English and Tamil, Assamese, Mizo, Nepali and Hausa, and four models from 1B to 7B parameters, RareCipherFT raises term recall over standard IC-Train by 5-15 absolute points without lowering COMET. Probes using synonyms, nonce words and controls for shared surface context show that these gains come from linking source terms to their in-context translations rather than from shortcuts.
Figures & tables
Figure 2: Overview of RareCipherFT. (i) In-context examples are retrieved from the few-shot pool with BM25. (ii) A shared rare span , occurring in both the gold reference y∗ and the ICL target yi , is replaced by its cipher , a random permutation of its characters, at every occurrence in the prompt and the reference; distractor spans , occurring only in yi , are likewise replaced by their ciphers . (iii) Training is annealed linearly from ciphered to original instances.
Pair
Training
Llama-3.2-1B
Qwen2.5-1.5B
Qwen2.5-3B
OLMo-3-7B
Support
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
Lus-En
Base
45.56
54.68
45.57
55.53
46.88
54.26
46.40
50.43
2000
235
ICFT
54.85 (0.46)
53.69 (2.56)
56.37 (0.29)
59.93 (1.81)
58.28 (0.02)
62.70 (1.09)
59.68 (0.12)
69.48 (1.78)
RareCipherFT
55.08 (0.39)
61.63 (1.07)
56.46 (0.86)
65.93 (1.74)
58.43 (0.44)
68.65 (0.54)
60.23 (0.38)
77.73 (0.54)
As-En
Base
64.44
66.55
63.74
57.92
69.72
72.39
65.75
71.70
2000
568
ICFT
68.53 (0.20)
69.68 (0.64)
67.92 (0.14)
67.75 (0.95)
71.48 (0.12)
73.55 (1.77)
71.38 (0.08)
77.85 (0.77)
Table 1: COMET and Term Recall of every model and language pair. Each cell is the mean over 3 training seeds, along with standard deviation. Support is the number of instances used to score each metric.
Training
Llama-3.2-1B / Ha-En
Qwen2.5-1.5B / As-En
Qwen2.5-1.5B / Ne-En
Qwen2.5-3B / Ta-En
OLMo-3-7B / En-Ha
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
ICFT
66.64 (0.12)
49.55 (1.23)
67.92 (0.14)
67.75 (0.95)
79.97 (0.09)
58.55 (1.62)
70.44 (0.16)
54.26 (1.58)
74.78 (0.19)
51.70 (0.20)
RareCipherFT
68.97 (0.07)
59.35 (1.73)
68.92 (0.10)
75.39 (1.41)
81.00 (0.40)
65.74 (2.55)
72.20 (0.07)
65.11 (0.76)
76.99 (0.15)
57.32 (0.30)
NoCipher
68.93 (0.15)
55.11 (1.02)
68.34 (0.03)
71.91 (0.71)
81.09 (0.14)
62.60 (2.81)
71.96 (0.03)
63.39 (0.57)
77.01 (0.15)
56.66 (0.25)
NoAnneal
68.41 (0.14)
59.29 (1.15)
67.19 (0.34)
78.66 (0.40)
80.06 (0.31)
70.64 (2.14)
66.68 (0.76)
70.35 (1.65)
76.76 (0.15)
57.90 (0.10)
NoSelect
68.76 (0.15)
53.52 (1.11)
68.63 (0.09)
72.49 (1.55)
81.11 (0.18)
65.47 (0.79)
71.77 (0.20)
63.21 (0.04)
76.20 (0.24)
55.15 (1.21)
Table 2: Ablations of RareCipherFT (Section 6.2 ). Removing the cipher ( NoCipher ) or the rarity threshold ( NoSelect ) lowers term recall, while removing annealing ( NoAnneal ) lowers COMET. Each cell is the mean over 3 training seeds, along with standard deviation.
Figure 3: Term recall under the probes of Section 6.3 : (a) parametric memory (synonym), (b, c) semantic similarity (shuffle, nonce), and (d) without shared left-side context. The models shown are Llama-3.2-1B (Ha-En), OLMo-3-7B (Lus-En, En-Ha), Qwen2.5-1.5B (As-En, Ne-En) and Qwen2.5-3B (Ta-En). Mean over 3 training seeds, error bars are the standard deviation.
Training
Llama-3.2-1B / Ha-En
OLMo-3-7B / Lus-En
Qwen2.5-1.5B / As-En
Qwen2.5-1.5B / Ne-En
Qwen2.5-3B / Ta-En
Llama-3.2-1B / En-Ha
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
COMET
TermRecall
RareCipherFT
68.97 (0.07)
69.28 (1.13)
60.23 (0.38)
78.17 (0.87)
68.92 (0.10)
76.72 (1.35)
81.00 (0.40)
67.65 (0.31)
72.20 (0.07)
68.16 (1.16)
75.99 (0.20)
60.22 (0.44)
DragFT
68.34 (0.00)
63.40 (0.75)
58.00 (0.29)
82.88 (0.81)
66.66 (0.04)
79.32 (0.71)
80.51 (0.17)
69.50 (1.53)
70.06 (0.15)
67.42 (0.89)
75.29 (0.08)
59.25 (0.16)
Support
1997
102
2000
184
2000
483
500
169
1998
322
1997
1384
Table 3: COMET and Term Recall of RareCipherFT and DragFT (Section 6.4 ). Each cell is the mean over 3 training seeds, along with standard deviation. Support is the number of test instances in each column.
Figure 4: A wrong dictionary entry copied by DragFT, Nepali → English. DragFT’s dictionary maps the name [Nepali text: compile with XeLaTeX/LuaLaTeX] ( Kusum ) to shivasgirl . Transliterations in Figure 9 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Split
LP
Source
Size
Few-shot pool
Train
Ta-En
Samanantar
15k
100k
As-En/En-As
IndicNE
11k/9k
25k
Lus-En
IndicNE
9k
25k
Ne-En
ISHAAN
15k
100k
Ha-En/En-Ha
WMT22-Afr
13k/11k
100k
Test
Ta-En
EnTam
2k
1k
Appendix
Table 4: Datasets Used for MT. IndicNE = IndicNE-Corp1.0; WMT22-Afr = WMT22 African. Sources: Samanantar from Ramesh et al. (2022) ; EnTam from Ramasamy et al. (2012) ; IndicNE-Corp1.0 from Pakray et al. (2024) ; ISHAAN from ISHAAN Consortium (2026) ; WMT22 African from Adelani et al. (2022) ; WMT21 from Akhbardeh et al. (2021) .
Figure 5: Term recall (left) and COMET (right) as the rarity threshold β varies over {100,250,500,∞} . β=∞ removes the threshold (NoSelect). Without the threshold, term recall falls below its β=250 value for every pair. Mean is taken over 3 training seeds, shading is the standard deviation.
Figure 6: The prompt used to generate a synonym for a rare target term. {lang} is the target language of the pair.
Pair
Training
Llama-3.2-1B
Qwen2.5-1.5B
Qwen2.5-3B
OLMo-3-7B
Support
Lus-En
Base
4.78
4.79
5.81
5.79
2000
ICFT
11.04 (0.36)
12.31 (0.32)
14.15 (0.20)
15.64 (0.32)
RareCipherFT
11.53 (0.29)
12.54 (0.76)
14.43 (0.41)
16.36 (0.21)
As-En
Base
8.95
8.15
11.18
11.04
2000
ICFT
11.30 (0.16)
10.71 (0.14)
13.56 (0.14)
14.67 (0.21)
RareCipherFT
11.68 (0.24)
11.44 (0.27)
14.30 (0.30)
15.27 (0.18)
Appendix
Table 5: BLEU of every model and language pair. Each cell is the mean over 3 training seeds, along with standard deviation. Support is the number of instances used to score each metric.
Figure 7: Terminology copying from context, Mizo → English. The source term is highlighted in green and its English rendering in yellow . The stem ngaihdam in the test source also occurs in Example 1, whose target supplies forgiveness ; RareCipherFT copies it and otherwise differs from the reference only in dropping the opening clause, while ICFT replaces the term with the vaguer in his purpose .
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Where there is growing interest in in-context language learning (ICLL) for unseen languages with large language models, such languages usually suffer from the lack of NLP tools, data resources, and researcher expertise. This means that progress is difficult to assess, the field does not allow for cheap large-scale experimentation, and findings on ICLL are often limited to very few languages and tasks. In light of such limitations, we introduce a framework (Rashid), for studying ICLL wherein we reversibly cipher high-resource languages (HRLs) to construct truly unseen languages with access to a wide range of resources available for HRLs, unlocking previously impossible exploration of ICLL phenomena. We use our framework to assess current methods in the field with SOTA evaluation tools and manual analysis, explore the utility of potentially expensive resources in improving ICLL, and test ICLL strategies on rich downstream tasks beyond machine translation. These lines of exploration showcase the possibilities enabled by our framework, as well as providing actionable insights regarding current performance and future directions in ICLL.
Niyati Bafna, Ryan Soh-Eun Shim, Barbara Plank +2
Johns Hopkins University, Center for Language and Speech Processing · MaiNLP, Center for Information and Language Processing, LMU Munich
In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks from a few examples, making it promising for languages underrepresented in pre-training. Recent work on many-shot ICL suggests that modern LLMs can further benefit from larger ICL examples enabled by their long context windows. However, such gains depend on careful example selection, and the inference cost can be prohibitive for low-resource language communities. In this paper, we present an empirical study of many-shot ICL for machine translation from English into ten truly low-resource languages recently added to FLORES+. We analyze the effects of retrieving more informative examples, using out-of-domain data, and ordering examples by length. Our findings show that many-shot ICL becomes more effective as the number of examples increases. More importantly, we show that BM25-based retrieval substantially improves data efficiency: 50 retrieved examples roughly match 250 many-shot examples, while 250 retrieved examples perform similarly to 1,000 many-shot examples. We further show that ICL provides additional gains on top of fine-tuning.
Yinhan Lu, Gaganpreet Jhajj, Chen Zhang +2
Mila – Quebec AI Institute · McGill University · Athabasca University +2
Large Language Models (LLMs) can be trained to perform chain-of-thoughts reasoning in order to improve the reliability of their responses. In this work, we investigate how explicit reasoning can be leveraged for LLM-Based Machine Translation (MT) with in-context samples. We introduce a novel fragment-based reasoning framework in which the model first extracts parallel source-target fragments from retrieved similar exemplars, and uses these fragments as intermediate reasoning traces to produce the final translation. To train our model, we distill silver fragments and drafts from a large teacher model. Our experiments with the Qwen3 model family, over 6 languages, including up to 5 domains per language, demonstrate that fragment-based MT significantly outperforms alternative methods like standard k-shot or basic drafting.
Maxime Bouthors, Josep Crego, François Yvon
SYSTRAN by ChapsVision, 5 rue Feydeau, F-75002 Paris, France · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France