cs.CLOct 17, 2026
SaveTraining with Ciphers to Enhance In-Context Terminology Learning For Translation from Rare Languages
Organizations: IIT Bombay
Abstract
Fine-tuning LLMs with in-context examples (IC-Train) lets translation models adapt to new parallel data at deployment, but it is unclear whether they actually learn the translations of rare terms from those examples. We formalize this capability as in-context term learning (ICTL). We show that standard IC-Train develops this capability only weakly, and propose RareCipherFT, which strengthens it without any lexical resources. RareCipherFT pairs each training target with in-context examples that share a rare term with it. It then replaces that term with a character-shuffled cipher in both the context and the reference, so the term can only be recovered by copying from context. The ciphered fraction of training data is annealed to preserve translation quality. Across seven translation directions between English and Tamil, Assamese, Mizo, Nepali and Hausa, and four models from 1B to 7B parameters, RareCipherFT raises term recall over standard IC-Train by 5-15 absolute points without lowering COMET. Probes using synonyms, nonce words and controls for shared surface context show that these gains come from linking source terms to their in-context translations rather than from shortcuts.
Figures & tables
Figure 2: Overview of RareCipherFT. (i) In-context examples are retrieved from the few-shot pool with BM25. (ii) A shared rare span , occurring in both the gold reference and the ICL target , is replaced by its cipher , a random permutation of its characters, at every occurrence in the prompt and the reference; distractor spans , occurring only in , are likewise replaced by their ciphers . (iii) Training is annealed linearly from ciphered to original instances.
| Pair | Training | Llama-3.2-1B | Qwen2.5-1.5B | Qwen2.5-3B | OLMo-3-7B | Support | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | ||
| Lus-En | Base | 45.56 | 54.68 | 45.57 | 55.53 | 46.88 | 54.26 | 46.40 | 50.43 | 2000 | 235 |
| ICFT | 54.85 (0.46) | 53.69 (2.56) | 56.37 (0.29) | 59.93 (1.81) | 58.28 (0.02) | 62.70 (1.09) | 59.68 (0.12) | 69.48 (1.78) | |||
| RareCipherFT | 55.08 (0.39) | 61.63 (1.07) | 56.46 (0.86) | 65.93 (1.74) | 58.43 (0.44) | 68.65 (0.54) | 60.23 (0.38) | 77.73 (0.54) | |||
| As-En | Base | 64.44 | 66.55 | 63.74 | 57.92 | 69.72 | 72.39 | 65.75 | 71.70 | 2000 | 568 |
| ICFT | 68.53 (0.20) | 69.68 (0.64) | 67.92 (0.14) | 67.75 (0.95) | 71.48 (0.12) | 73.55 (1.77) | 71.38 (0.08) | 77.85 (0.77) | |||
Table 1: COMET and Term Recall of every model and language pair. Each cell is the mean over 3 training seeds, along with standard deviation. Support is the number of instances used to score each metric.
| Training | Llama-3.2-1B / Ha-En | Qwen2.5-1.5B / As-En | Qwen2.5-1.5B / Ne-En | Qwen2.5-3B / Ta-En | OLMo-3-7B / En-Ha | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | |
| ICFT | 66.64 (0.12) | 49.55 (1.23) | 67.92 (0.14) | 67.75 (0.95) | 79.97 (0.09) | 58.55 (1.62) | 70.44 (0.16) | 54.26 (1.58) | 74.78 (0.19) | 51.70 (0.20) |
| RareCipherFT | 68.97 (0.07) | 59.35 (1.73) | 68.92 (0.10) | 75.39 (1.41) | 81.00 (0.40) | 65.74 (2.55) | 72.20 (0.07) | 65.11 (0.76) | 76.99 (0.15) | 57.32 (0.30) |
| NoCipher | 68.93 (0.15) | 55.11 (1.02) | 68.34 (0.03) | 71.91 (0.71) | 81.09 (0.14) | 62.60 (2.81) | 71.96 (0.03) | 63.39 (0.57) | 77.01 (0.15) | 56.66 (0.25) |
| NoAnneal | 68.41 (0.14) | 59.29 (1.15) | 67.19 (0.34) | 78.66 (0.40) | 80.06 (0.31) | 70.64 (2.14) | 66.68 (0.76) | 70.35 (1.65) | 76.76 (0.15) | 57.90 (0.10) |
| NoSelect | 68.76 (0.15) | 53.52 (1.11) | 68.63 (0.09) | 72.49 (1.55) | 81.11 (0.18) | 65.47 (0.79) | 71.77 (0.20) | 63.21 (0.04) | 76.20 (0.24) | 55.15 (1.21) |
Table 2: Ablations of RareCipherFT (Section 6.2 ). Removing the cipher ( NoCipher ) or the rarity threshold ( NoSelect ) lowers term recall, while removing annealing ( NoAnneal ) lowers COMET. Each cell is the mean over 3 training seeds, along with standard deviation.
Figure 3: Term recall under the probes of Section 6.3 : (a) parametric memory (synonym), (b, c) semantic similarity (shuffle, nonce), and (d) without shared left-side context. The models shown are Llama-3.2-1B (Ha-En), OLMo-3-7B (Lus-En, En-Ha), Qwen2.5-1.5B (As-En, Ne-En) and Qwen2.5-3B (Ta-En). Mean over 3 training seeds, error bars are the standard deviation.
| Training | Llama-3.2-1B / Ha-En | OLMo-3-7B / Lus-En | Qwen2.5-1.5B / As-En | Qwen2.5-1.5B / Ne-En | Qwen2.5-3B / Ta-En | Llama-3.2-1B / En-Ha | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | COMET | TermRecall | |
| RareCipherFT | 68.97 (0.07) | 69.28 (1.13) | 60.23 (0.38) | 78.17 (0.87) | 68.92 (0.10) | 76.72 (1.35) | 81.00 (0.40) | 67.65 (0.31) | 72.20 (0.07) | 68.16 (1.16) | 75.99 (0.20) | 60.22 (0.44) |
| DragFT | 68.34 (0.00) | 63.40 (0.75) | 58.00 (0.29) | 82.88 (0.81) | 66.66 (0.04) | 79.32 (0.71) | 80.51 (0.17) | 69.50 (1.53) | 70.06 (0.15) | 67.42 (0.89) | 75.29 (0.08) | 59.25 (0.16) |
| Support | 1997 | 102 | 2000 | 184 | 2000 | 483 | 500 | 169 | 1998 | 322 | 1997 | 1384 |
Table 3: COMET and Term Recall of RareCipherFT and DragFT (Section 6.4 ). Each cell is the mean over 3 training seeds, along with standard deviation. Support is the number of test instances in each column.
Figure 4: A wrong dictionary entry copied by DragFT, Nepali English. DragFT’s dictionary maps the name [Nepali text: compile with XeLaTeX/LuaLaTeX] ( Kusum ) to shivasgirl . Transliterations in Figure 9 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Split | LP | Source | Size | Few-shot pool |
|---|---|---|---|---|
| Train | Ta-En | Samanantar | 15k | 100k |
| As-En/En-As | IndicNE | 11k/9k | 25k | |
| Lus-En | IndicNE | 9k | 25k | |
| Ne-En | ISHAAN | 15k | 100k | |
| Ha-En/En-Ha | WMT22-Afr | 13k/11k | 100k | |
| Test | Ta-En | EnTam | 2k | 1k |
Table 4: Datasets Used for MT. IndicNE = IndicNE-Corp1.0; WMT22-Afr = WMT22 African. Sources: Samanantar from Ramesh et al. (2022) ; EnTam from Ramasamy et al. (2012) ; IndicNE-Corp1.0 from Pakray et al. (2024) ; ISHAAN from ISHAAN Consortium (2026) ; WMT22 African from Adelani et al. (2022) ; WMT21 from Akhbardeh et al. (2021) .
Figure 5: Term recall (left) and COMET (right) as the rarity threshold varies over . removes the threshold (NoSelect). Without the threshold, term recall falls below its value for every pair. Mean is taken over 3 training seeds, shading is the standard deviation.
Figure 6: The prompt used to generate a synonym for a rare target term. {lang} is the target language of the pair.
| Pair | Training | Llama-3.2-1B | Qwen2.5-1.5B | Qwen2.5-3B | OLMo-3-7B | Support |
|---|---|---|---|---|---|---|
| Lus-En | Base | 4.78 | 4.79 | 5.81 | 5.79 | 2000 |
| ICFT | 11.04 (0.36) | 12.31 (0.32) | 14.15 (0.20) | 15.64 (0.32) | ||
| RareCipherFT | 11.53 (0.29) | 12.54 (0.76) | 14.43 (0.41) | 16.36 (0.21) | ||
| As-En | Base | 8.95 | 8.15 | 11.18 | 11.04 | 2000 |
| ICFT | 11.30 (0.16) | 10.71 (0.14) | 13.56 (0.14) | 14.67 (0.21) | ||
| RareCipherFT | 11.68 (0.24) | 11.44 (0.27) | 14.30 (0.30) | 15.27 (0.18) |
Table 5: BLEU of every model and language pair. Each cell is the mean over 3 training seeds, along with standard deviation. Support is the number of instances used to score each metric.
Figure 7: Terminology copying from context, Mizo English. The source term is highlighted in green and its English rendering in yellow . The stem ngaihdam in the test source also occurs in Example 1, whose target supplies forgiveness ; RareCipherFT copies it and otherwise differs from the reference only in dropping the opening clause, while ICFT replaces the term with the vaguer in his purpose .
More in this topic
Machine Translation
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
See all 259 papers