Reasoning about In-Context Samples for Machine-Translation

Maxime Bouthors; Josep Crego; François Yvon

cs.CLAug 27, 2026

Reasoning about In-Context Samples for Machine-Translation

Authors: Maxime Bouthors, Josep Crego, François Yvon

Organizations: SYSTRAN by ChapsVision, 5 rue Feydeau, F-75002 Paris, France · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France

Abstract

Large Language Models (LLMs) can be trained to perform chain-of-thoughts reasoning in order to improve the reliability of their responses. In this work, we investigate how explicit reasoning can be leveraged for LLM-Based Machine Translation (MT) with in-context samples. We introduce a novel fragment-based reasoning framework in which the model first extracts parallel source-target fragments from retrieved similar exemplars, and uses these fragments as intermediate reasoning traces to produce the final translation. To train our model, we distill silver fragments and drafts from a large teacher model. Our experiments with the Qwen3 model family, over 6 languages, including up to 5 domains per language, demonstrate that fragment-based MT significantly outperforms alternative methods like standard k-shot or basic drafting.

Figures & tables

Figure 1: Overview of the training pipeline of the model: (1) exemplars are retrieved from the TM; (2) A teacher model performs FE from the exemplars; (3) A student model is trained to reproduce FE as a reasoning trace.

Figure 2: Illustration of the fragment extraction (FE) step, with

k=1

exemplar. Extracted fragments are colored while generated parts are grayed.

		BLEU $\uparrow$				COMET $\uparrow$				MetricX $\downarrow$
Model	think	k=0	k=1	k=2	k=3	k=0	k=1	k=2	k=3	k=0	k=1	k=2	k=3
(I)	✗	28.6	33.0	35.6	36.0	82.3	79.9	81.9	82.1	7.43	2.15	2.12	2.10
(B)	✗	38.3	45.3	46.0	46.3	85.3	86.3	86.4	86.5	2.27	2.11	2.09	2.07
(D)	✗	36.7	44.7	45.5	45.8	85.0	86.1	86.2	86.4	2.36	2.16	2.13	2.11
(D)	✓	36.8	44.1	45.0	46.0	84.9	85.0	85.6	85.6	2.34	2.66	2.43	2.11
(F)	✗	37.4	44.8	45.5	45.9	85.1	86.2	86.3	86.4	2.33	2.13	2.11	2.09

Table 1: Average BLEU , COMET , and MetricX scores on the multi-domain test set for the different models and number of examples

k

. Note that fine-tuned models can be run with or without reasoning. Results significantly better (p<1%) than the baseline (B) are in bold . The significance is assessed with a paired t-test for COMET and MetricX , and with bootstrapping for BLEU with SacreBLEU (n=1000).

$k=$	0	1	2	3
BLEU	8/16	12/16	14/16	14/16
COMET	14/16	14/16	14/16	14/16
MetricX	6/16	13/16	15/16	14/16

Table 2: Win rates of (F) (with thinking on) against (B) using BLEU , COMET , and MetricX across 16 domains.

		BLEU $\uparrow$				COMET $\uparrow$				MetricX $\downarrow$
Model	think	k=0	k=1	k=2	k=3	k=0	k=1	k=2	k=3	k=0	k=1	k=2	k=3
(I)	✗	44.6	55.7	56.9	57.4	85.6	84.3	85.8	85.7	1.68	2.21	2.01	2.07
(B)	✗	45.9	64.4	65.2	65.4	86.0	89.4	89.8	89.8	1.65	1.27	1.22	1.20
(D)	✗	44.9	64.5	65.5	65.8	85.8	89.2	89.8	89.9	1.74	1.31	1.26	1.26
(D)	✓	46.2	63.6	65.8	66.1	86.1	86.4	89.7	89.9	1.68	1.89	1.26	1.24
(F)	✗	45.5	63.7	64.4	65.1	86.3	89.2	89.7	89.9	1.67	1.30	1.25	1.23

Table 3: BLEU , COMET , and MetricX scores on the GNOME domain, for several models and number of exemplars

k

. Results significantly better (p<5%) than the baseline (B) are in bold . The significance is assessed with a paired t-test for COMET and MetricX , and with bootstrapping for BLEU (using SacreBLEU,

n=1000

fragments	Prec.	Rec.	BLEU
silver	98.5	81.1	77.1
(F)	99.7	89.2	80.8
(F+D)	99.7	89.1	80.8

Table 4: Assessing the faithfulness of the source fragments w.r.t. the source sentence.

	BLEU $\uparrow$				COMET $\uparrow$				MetricX $\downarrow$
Model	k=0	k=1	k=2	k=3	k=0	k=1	k=2	k=3	k=0	k=1	k=2	k=3
(D)	+2.2	+1.2	+0.7	0.0	+0.2	+1.0	+0.5	+0.5	-0.2	-0.6	-0.4	0.0
(F)	+1.1	+0.3	+0.1	0.0	0.0	-0.1	-0.4	-0.4	0.0	0.0	+0.1	+0.1
(F+D)	+1.3	+0.1	0.0	+0.1	+0.1	-0.1	-0.1	0.0	-0.1	0.0	0.0	0.0

Table 5: BLEU , COMET , and MetricX scores average gains on the multi-domain test set when prefixing the silver drafts and/or fragments instead of having them be inferred during reasoning.

Figure 3: Differences in COMET score gain between (F) and (B) w.r.t the coverage of the reference sentence.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

domain	ECB	JRC	Wiki	JRC	KDE	TED	ECB	EMEA	Epp	JRC	Wiki	KDE	EMEA	JRC	Kor	Sub
en-xx pair	pl	pl	pl	es	uk	uk	fr	fr	fr	fr	fr	de	de	de	de	de
size	46K	1.0M	111K	480K	93K	177K	140K	242K	1.9M	475K	538K	223K	18K	467K	248K	500K

Appendix

Table 6: List of domains with the size of the full available training datasets.

Panel of Figure 4 — Figure 4: FE prompt used to generate the silver fragments an the draft with Qwen3-32B . Thinking is disabled.

Panel of Figure 5 — Figure 5: Prompt used to train the baseline model (B) . The thinking process is empty, as the model is only trained to generate a translation.

Panel of Figure 6 — Figure 6: Prompt used to train the draft model (D) .

Panel of Figure 7 — Figure 7: Prompt used to train the fragments model (F) .

Panel of Figure 8 — Figure 8: Prompt used to train the draft+fragments model (F+D) .

		ECB	JRC	Wiki	JRC	KDE	TED	ECB	EMEA	Epp	JRC	Wiki	KDE	EMEA	JRC	Kor	Sub
think	k	pl	pl	pl	es	uk	uk	fr	fr	fr	fr	fr	de	de	de	de	de
		BLEU $\uparrow$
✗	0	21.2	27.2	28.2	35.3	22.1	18.7	40.0	39.5	32.8	38.9	33.4	30.0	33.0	26.1	9.8	20.9
✗	1	32.2	40.9	25.6	43.0	25.1	17.7	48.4	48.4	27.8	49.1	34.6	29.0	37.8	36.9	13.4	18.0
✗	2	35.0	43.4	29.0	46.9	26.5	18.6	52.7	51.3	31.5	52.2	36.8	32.0	40.4	40.6	14.5	19.0
✗	3	35.5	43.2	28.7	46.8	27.7	18.7	53.4	51.1	31.6	52.7	37.9	32.5	40.3	41.0	15.1	19.2

Appendix

Table 7: BLEU , COMET , and MetricX scores on the multi-domain test sets for the pre-trained Qwen3-8B Instruct model (I) .

		ECB	JRC	Wiki	JRC	KDE	TED	ECB	EMEA	Epp	JRC	Wiki	KDE	EMEA	JRC	Kor	Sub
think	k	pl	pl	pl	es	uk	uk	fr	fr	fr	fr	fr	de	de	de	de	de
		BLEU $\uparrow$
✗	0	40.8	44.5	36.5	50.4	37.6	21.4	51.3	49.0	36.1	52.2	40.3	34.5	39.4	37.0	19.4	23.1
✗	1	48.3	54.7	38.0	60.7	42.9	21.7	63.6	64.0	36.6	65.8	43.4	41.1	47.2	53.6	22.2	20.9
✗	2	48.8	55.2	38.2	61.3	43.3	21.6	64.8	64.9	37.0	67.1	43.7	41.7	47.8	54.3	23.3	23.5
✗	3	49.0	55.5	38.6	61.9	43.4	21.7	65.2	65.2	36.9	67.5	43.9	42.2	47.8	54.8	24.0	23.6

Appendix

Table 8: BLEU , COMET , and MetricX scores on the multi-domain test sets for the fine-tuned Qwen3-8B baseline model (B) .

		ECB	JRC	Wiki	JRC	KDE	TED	ECB	EMEA	Epp	JRC	Wiki	KDE	EMEA	JRC	Kor	Sub
think	k	pl	pl	pl	es	uk	uk	fr	fr	fr	fr	fr	de	de	de	de	de
		BLEU $\uparrow$
✗	0	36.3	40.5	34.6	48.0	34.8	21.2	49.6	47.5	35.7	50.5	39.4	33.9	39.2	36.4	17.6	21.1
✗	1	46.3	53.2	36.9	59.4	40.9	21.8	63.3	63.6	36.2	64.8	43.5	41.1	47.6	52.7	20.6	23.7
✗	2	47.0	54.0	37.5	60.3	41.3	21.6	64.7	64.7	36.3	66.7	44.0	41.6	48.1	54.2	21.6	24.0
✗	3	47.5	54.5	37.8	60.7	41.5	21.8	65.1	64.9	36.6	67.1	44.2	41.7	48.4	54.6	22.7	24.3

Appendix

Table 9: BLEU , COMET , and MetricX scores on the multi-domain test sets for the fine-tuned Qwen3-8B drafting model (D) .

		ECB	JRC	Wiki	JRC	KDE	TED	ECB	EMEA	Epp	JRC	Wiki	KDE	EMEA	JRC	Kor	Sub
think	k	pl	pl	pl	es	uk	uk	fr	fr	fr	fr	fr	de	de	de	de	de
		BLEU $\uparrow$
✗	0	38.0	42.5	35.4	49.1	36.5	21.5	50.8	47.9	35.8	51.4	36.5	33.8	39.0	37.3	18.9	23.2
✗	1	46.5	53.9	37.3	59.5	41.7	21.6	63.1	63.5	36.2	65.0	43.4	40.8	47.0	52.7	21.8	23.3
✗	2	47.4	54.7	37.7	60.7	42.3	21.6	64.3	64.2	36.2	66.6	43.9	41.3	47.9	54.1	22.5	23.4
✗	3	48.1	55.1	38.0	61.2	42.3	21.5	64.9	64.7	36.2	66.9	44.2	41.7	48.0	54.8	23.4	23.5

Appendix

Table 10: BLEU , COMET , and MetricX scores on the multi-domain test sets for the fine-tuned Qwen3-8B fragment-based model (F) .

		ECB	JRC	Wiki	JRC	KDE	TED	ECB	EMEA	Epp	JRC	Wiki	KDE	EMEA	JRC	Kor	Sub
think	k	pl	pl	pl	es	uk	uk	fr	fr	fr	fr	fr	de	de	de	de	de
		BLEU $\uparrow$
✗	0	37.9	41.8	35.1	49.0	35.9	21.1	50.2	49.0	35.5	51.2	41.2	33.9	39.2	37.0	18.6	23.2
✗	1	46.4	53.7	37.1	59.6	41.3	21.3	63.1	63.0	35.9	64.3	43.5	41.3	46.7	52.4	21.7	23.4
✗	2	47.4	54.5	37.4	60.6	42.0	21.3	64.2	64.3	36.1	66.3	43.8	41.6	47.7	53.9	22.7	23.7
✗	3	47.8	55.0	37.8	61.1	42.0	21.4	64.8	64.5	36.1	66.7	43.9	41.6	48.1	54.3	23.3	23.6

Appendix

Table 11: BLEU , COMET , and MetricX scores on the multi-domain test sets for the fine-tuned Qwen3-8B fragment-based drafting model (F+D) .

Figure 9: Illustration of the traceability provided by the FE step.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 259 papers

Unlocking Reasoning Capability on Machine Translation in Large Language Models
Feb 16, 2026Sara Rajaee, Sebastian Vincent, Alexandre Berard +3Structured Reasoning in Language Models Large Reasoning Models
University of Amsterdam · Cohere · Cohere Labs
When Does Reasoning Help in Machine Translation? A Hierarchical Analysis of LRM Reasoning Traces
Sep 18, 2026Yuxiang Liu, Jiaming Luo, Eleftheria Briakou +1Machine Translation Large Reasoning Models
University of Illinois at Urbana-Champaign · Google DeepMind
Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?
Jun 2, 2026Renhao Pei, Yihong Liu, Sampo Pyysalo +2CoT Reasoning Low-Resource MT
ELLIS Institute Finland · University of Turku · Center for Information and Language Processing, LMU Munich +1

Feb 16, 2026J/K move · Enter open · S save

Unlocking Reasoning Capability on Machine Translation in Large Language Models

Sara Rajaee, Sebastian Vincent, Alexandre Berard, Marzieh Fadaee+2

University of Amsterdam · Cohere · Cohere Labs

Structured Reasoning in Language Models Large Reasoning Models LLM Fine-Tuning Cross-Lingual Reasoning in Language Models

Open paper AskPDF arXiv

Reasoning-oriented large language models (RLMs) achieve strong gains on tasks such as mathematics and coding by generating explicit intermediate reasoning. However, their impact on machine translation (MT) remains underexplored. We systematically evaluate several open- and closed-weights RLMs on the WMT24++ benchmark and find that enabling explicit reasoning consistently degrades translation quality across languages and models. Analysis reveals that MT reasoning traces are highly linear, lacking revision, self-correction and exploration of alternative translations, which limits their usefulness. Furthermore, injecting higher-quality reasoning traces from stronger models does not reliably improve weaker models' performance. To address this mismatch, we propose a structured reasoning framework tailored to translation, based on multi-step drafting, adequacy refinement, fluency improvement, and selective iterative revision. We curate a synthetic dataset of dynamic structured reasoning traces and post-train a large reasoning model on this data. Experiments show significant improvements over standard translation fine-tuning and injected generic reasoning baselines. Our findings demonstrate that reasoning must be task-structured to benefit MT.

Abstract

Machine Translation

Explore similar work

Unlocking Reasoning Capability on Machine Translation in Large Language Models

When Does Reasoning Help in Machine Translation? A Hierarchical Analysis of LRM Reasoning Traces

Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?