Large Reasoning Models increasingly use intermediate traces for machine translation, but it remains unclear when such reasoning helps or hurts. We analyze reasoning traces across models, languages, domains, and datasets, focusing on reasoning language, length, and structure. We find that the best reasoning language is model-specific, reasoning length has a non-monotonic relationship with quality, and traces exhibit recurring functional patterns. To uncover these patterns, we introduce Hierarchical Meta-Summarization (HMS), a scalable framework that induces coarse- and fine-grained reasoning structures without predefined taxonomies. HMS reveals a shared organization--understanding/planning, translating/drafting, and refining/verifying--alongside domain-specific variation. Our results suggest that MT reasoning should be controlled in a model-aware, length-aware, and pattern-aware manner rather than uniformly encouraged.
Figures & tables
Inst. → Reason
WMT
CMT
DRT
Mean
en-de
en-es
en-ja
en-ru
en-zh
en-es
en-fr
en-zh
en-zh
en → en (%)
78.6
93.3
29.4
66.8
64.2
92.5
92.9
57.0
63.8
71.0
zh → zh (%)
97.2
98.3
92.1
94.8
99.2
97.5
97.8
99.8
99.0
97.3
tgt → tgt (%)
27.7
7.2
69.3
55.0
–
4.6
7.3
–
–
28.5
Table 1: Reasoning-Language Instruction-Adherence (RLIA) rates for Qwen-32B. The model follows Chinese reasoning-language instructions most reliably, while target-language instructions yield much lower adherence.
Metric
Inst.
WMT
CMT
DRT
Mean
en-de
en-es
en-ja
en-ru
en-zh
en-es
en-fr
en-zh
en-zh
COMET ↑
en
0.7613
0.7774
0.8186
0.7631
0.8297
0.8063
0.7676
0.8361
0.7742
0.7927
zh
0.7442
0.7976
0.8267
0.7662
0.8363
0.8135
0.7636
0.8404
0.7810
0.7966
Norm.MX ↑
en
0.8640
0.8293
0.7685
0.7728
0.8825
0.8456
0.8219
0.8634
0.8400
0.8320
zh
0.8710
0.8526
0.7842
0.7807
0.8894
0.8678
0.8296
0.8715
0.8502
0.8441
Norm.MX-QE ↑
en
0.8766
0.8358
0.7845
0.7977
0.8884
0.8605
0.8676
0.8769
0.8361
0.8471
Table 2: Translation quality for Qwen-32B by instructed reasoning language. Chinese reasoning yields higher mean scores across all three metrics, especially on Norm.MX and Norm.MX-QE.
Model
en → en (%)
zh → zh (%)
Qwen-14B
71.6
97.6
Qwen-32B
64.2
99.2
gpt-oss-20B
98.3
2.4
Llama-8B
77.8
99.9
Gemma-4-E4B
95.3
2.1
Table 3: RLIA on WMT (en-zh). Qwen and Llama favor Chinese reasoning instructions, whereas gpt-oss-20B and Gemma-4-E4B favor English.
Model
Inst.
COMET ↑
MX ↑
MX-QE ↑
Qwen-14B
en
0.8227
0.8644
0.8730
zh
0.8285
0.8749
0.8838
Qwen-32B
en
0.8297
0.8825
0.8884
zh
0.8363
0.8894
0.8960
gpt-oss-20B
en
0.8352
0.8837
0.8935
zh
0.8326
0.8791
0.8886
Table 4: Translation quality on WMT (en-zh). Chinese reasoning instructions perform better for Qwen and Llama, whereas English reasoning instructions perform better for gpt-oss-20B and Gemma-4-E4B.
Figure 1: Translation quality by ranked reasoning trace length for Qwen-32B on WMT.
Figure 2: Hierarchical Meta-Summarization (HMS). HMS is a data-driven framework for structural reasoning pattern analysis that abstracts raw reasoning traces into atomic steps and recursively applies meta-summarization to derive coarse Level-0 patterns and finer Level-1 sub-patterns.
Model
Induced HMS cluster
Function
Gemma-4-E4B
Analyzing Source Text
U/P
Drafting Initial Translation
T/D
Refining and Polishing Translation
R/V
Llama-8B
Initial Understanding and Planning
U/P
Executing Translation
T/D
Refining and Verifying
R/V
Table 5: Cross-model mapping of induced HMS clusters to shared functions. General-purpose LRMs largely recover the three-stage structure, whereas MT-specialized DRT-14B emphasizes critique and revision.
Figure 3: Level-0 reasoning allocation on WMT24 across four domains. Reasoning ratios remain stable, while reasoning budgets vary modestly by domain.
Figure 4: Qwen-32B COMET scores on WMT24 when the reasoning pattern is set to its maximum versus minimum observed length. Longer U/P is the most stable positive signal, while T/D and R/V effects vary more by domain.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Inst. → Reason
WMT
CMT
DRT
Mean
en-de
en-es
en-ja
en-ru
en-zh
en-es
en-fr
en-zh
en-zh
en → en (%)
87.7
97.3
43.5
72.8
71.6
95.9
96.7
64.0
66.8
77.4
zh → zh (%)
93.2
91.2
93.9
82.4
97.6
90.0
90.4
99.2
98.7
92.9
Appendix
Table 6: Reasoning-Language Instruction-Adherence (RLIA) rates for Qwen-14B. Chinese instructions yield higher overall adherence than English, though English is stronger on several English-to-European language pairs.
Metric
Inst.
WMT
CMT
DRT
Mean
en-de
en-es
en-ja
en-ru
en-zh
en-es
en-fr
en-zh
en-zh
COMET ↑
en
0.7520
0.7872
0.8051
0.7259
0.8227
0.8205
0.7713
0.8267
0.7600
0.7857
zh
0.7450
0.7940
0.8086
0.7192
0.8285
0.8176
0.7684
0.8329
0.7669
0.7868
Norm.MX ↑
en
0.8258
0.8071
0.7334
0.6925
0.8644
0.8572
0.8168
0.8420
0.8163
0.8062
zh
0.8346
0.8282
0.7555
0.7076
0.8749
0.8609
0.8175
0.8565
0.8272
0.8181
Norm.MX-QE ↑
en
0.8438
0.8138
0.7508
0.7204
0.8730
0.8749
0.8648
0.8579
0.8164
0.8240
Appendix
Table 7: Translation quality for Qwen-14B by instructed reasoning language. Chinese reasoning yields higher mean scores across all three metrics, with especially consistent gains on Norm.MX and Norm.MX-QE.
Model
Inst.
zh%
en%
COMET ↑
Norm.MX ↑
Norm.MX-QE ↑
Qwen-14B
English
28.4%
71.6%
0.8227
0.8644
0.8730
Chinese
97.6%
2.4%
0.8285
0.8749
0.8838
None
38.0%
62.0%
0.8235
0.8659
0.8745
Qwen-32B
English
35.8%
64.2%
0.8297
0.8825
0.8884
Chinese
99.2%
0.8%
0.8363
0.8894
0.8960
None
44.0%
56.0%
0.8306
0.8834
0.8894
Appendix
Table 8: Reasoning-language distributions and translation quality under English, Chinese, and no-language-control conditions on WMT24 en-zh.
Dataset
Model
Lang.
Ranked reasoning trace length
r16/r1
r1
r2
r3
r4
r5
r6
r7
r8
r9
r10
r11
r12
r13
r14
r15
r16
WMT
Qwen-14B
en
236
274
300
322
342
361
380
399
419
443
467
495
530
577
648
825
3.49×
zh
339
404
444
476
502
527
550
573
596
619
643
671
705
744
800
934
2.76×
Qwen-32B
en
276
325
359
389
416
442
469
496
525
556
592
635
685
752
856
1092
3.96×
zh
406
479
524
559
589
618
644
670
696
724
753
785
826
874
946
1147
2.83×
CMT
Qwen-14B
en
240
277
302
323
341
360
377
395
414
435
457
483
515
554
613
739
3.07×
Appendix
Table 9: Reasoning trace length by rank across datasets. For each input, 16 traces are sorted from shortest ( r1 ) to longest ( r16 ); cells report the mean length at each rank.
Model
Lang.
Ranked reasoning trace length
r16/r1
r1
r2
r3
r4
r5
r6
r7
r8
r9
r10
r11
r12
r13
r14
r15
r16
Qwen-14B
en
195
226
249
268
284
301
319
335
351
377
396
416
442
473
516
599
3.07×
zh
324
380
418
448
473
497
518
538
559
580
603
632
665
698
740
817
2.52×
Qwen-32B
en
212
253
281
305
327
348
369
392
416
439
466
496
531
575
655
796
3.76×
zh
369
432
478
513
543
572
598
625
649
675
702
730
765
807
866
996
2.70×
gpt-oss-20B
en
158
204
240
269
296
318
340
364
385
408
433
460
490
530
587
704
4.45×
Appendix
Table 10: Reasoning trace length by rank on WMT (en-zh). For each input, 16 traces are sorted from shortest ( r1 ) to longest ( r16 ); cells report the mean length at each rank.
Figure 5: Translation quality by ranked reasoning trace length for Qwen-32B on CMT.
Figure 6: Translation quality by ranked reasoning trace length for Qwen-32B on DRT.
Figure 7: Translation quality by ranked reasoning trace length for Qwen-14B on WMT.
Figure 8: Translation quality by ranked reasoning trace length for Qwen-14B on CMT.
Figure 9: Translation quality by ranked reasoning trace length for Qwen-14B on DRT.
Figure 10: Translation quality by ranked reasoning trace length on WMT (en-zh) across evaluated models. Adjacent length ranks are grouped into bins from shortest to longest traces.
Effort
COMET
Norm.MX
Norm.MX-QE
#words
Low
0.8665
0.8428
0.8501
93.3
Med
0.8696
0.8455
0.8530
143.6
High
0.8689
0.8469
0.8547
173.1
Appendix
Table 11: Controlled reasoning effort for gpt-oss-20B on WMT (en-zh). Higher effort increases trace length and slightly improves Norm.MX and Norm.MX-QE, while COMET peaks at medium effort.
HMS stage
API calls per trace
Input tokens per trace
Output tokens per trace
Step segmentation + abstraction
1
∼ 900–1,000
∼ 570–690
Meta-summarization
amortized, negligible
<10
<10
Step labeling
∼ 7
∼ 5,300–6,500
∼ 15–30
Total, step-wise labeling
∼ 8 calls
∼ 6.2k–7.5k
∼ 0.6k–0.7k
Appendix
Table 12: Approximate API call and token costs per trace for HMS step-wise labeling.
Evaluation
Yes rate
Fleiss’ κ
Summary faithfulness
92%
0.71
Level-0 label correctness
88%
0.68
Level-1 label correctness
79%
0.61
R/V corrective behavior
67%
0.65
R/V checking behavior
24%
0.60
Overall R/V validity
91%
0.74
Appendix
Table 13: Human evaluation of HMS summary and label quality with inter-annotator agreement.
Dataset
Stage
Level-0 Pattern
Level-1 Sub-patterns
WMT
U/P
Understanding & Planning
Understanding Task and Context
Identifying Linguistic & Cultural Nuances
Understanding User Intent and Context
Formulating & Planning Translation Strategy
Source Text Analysis and Deconstruction
T/D
Translating & Drafting
Initial Translation and Drafting
Appendix
Table 14: HMS-derived hierarchical reasoning pattern labels for Qwen-32B across three translation datasets. For DRT, HMS separates initial interpretation from translation-strategy planning, reflecting the stronger semantic and stylistic demands of literary translation.
Figure 11: Domain-specific allocation of WMT24 Level-1 reasoning patterns for Qwen-32B. Absolute budgets and relative ratios show how reasoning effort is distributed across fine-grained patterns within each functional stage.
Figure 12: Qwen-32B normalized reference-based MetricX scores under maximum versus minimum reasoning lengths on WMT24. Longer U/P is the most stable positive signal, while T/D and R/V effects vary more by domain.
Figure 13: Qwen-32B normalized MetricX-QE scores under maximum versus minimum reasoning lengths on WMT24. Longer U/P is generally favorable; longer T/D and R/V show more selective, domain-dependent patterns.
Figure 14: Llama-8B COMET scores under maximum versus minimum reasoning lengths on WMT24. The comparison broadly follows the Qwen-32B trend, with U/P more stable than T/D or R/V across domains.
Figure 15: Llama-8B normalized reference-based MetricX scores under maximum versus minimum reasoning lengths on WMT24. U/P shows the clearest positive pattern, whereas T/D and R/V depend more on domain.
Figure 16: Llama-8B normalized MetricX-QE scores under maximum versus minimum reasoning lengths on WMT24. Longer reasoning is not uniformly beneficial: U/P is comparatively robust, while T/D and R/V are more conditional.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Reasoning-oriented large language models (RLMs) achieve strong gains on tasks such as mathematics and coding by generating explicit intermediate reasoning. However, their impact on machine translation (MT) remains underexplored. We systematically evaluate several open- and closed-weights RLMs on the WMT24++ benchmark and find that enabling explicit reasoning consistently degrades translation quality across languages and models. Analysis reveals that MT reasoning traces are highly linear, lacking revision, self-correction and exploration of alternative translations, which limits their usefulness. Furthermore, injecting higher-quality reasoning traces from stronger models does not reliably improve weaker models' performance. To address this mismatch, we propose a structured reasoning framework tailored to translation, based on multi-step drafting, adequacy refinement, fluency improvement, and selective iterative revision. We curate a synthetic dataset of dynamic structured reasoning traces and post-train a large reasoning model on this data. Experiments show significant improvements over standard translation fine-tuning and injected generic reasoning baselines. Our findings demonstrate that reasoning must be task-structured to benefit MT.
Sara Rajaee, Sebastian Vincent, Alexandre Berard +3
Across multiple language pairings (English → {Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese}), we find reasoning errors in translation. To quantify how often these reasoning errors occur, we leverage an automated annotation protocol for reasoning evaluation wherein the goal is to detect if a reasoning step is any of three error categories: (1) source sentence-misaligned, (2) model hypothesis-misaligned, or (3) reasoning trace-misaligned. We probe the reasoning model with perturbed traces correcting for these identified reasoning errors using an array of weak-to-strong interventions: hedging, removal, re-reasoning after removal, hindsight, and oracle interventions. Experimenting with interventions on the reasoning traces suggests that small corrections to the reasoning have little impact on translation quality, but stronger interventions yield the highest resolution rates, despite translation quality gains being mixed. We find ultimately that reasoning errors in MT can be identified with high precision in Urdu but lower precision in Spanish, but that removing these reasoning errors does not resolve the initial errors significantly, suggesting limited reasoning faithfulness for machine translation.
Calvin Bao, Marine Carpuat
Department of Computer Science, University of Maryland, College Park, MD 20742, USA
Large Language Models (LLMs) can be trained to perform chain-of-thoughts reasoning in order to improve the reliability of their responses. In this work, we investigate how explicit reasoning can be leveraged for LLM-Based Machine Translation (MT) with in-context samples. We introduce a novel fragment-based reasoning framework in which the model first extracts parallel source-target fragments from retrieved similar exemplars, and uses these fragments as intermediate reasoning traces to produce the final translation. To train our model, we distill silver fragments and drafts from a large teacher model. Our experiments with the Qwen3 model family, over 6 languages, including up to 5 domains per language, demonstrate that fragment-based MT significantly outperforms alternative methods like standard k-shot or basic drafting.
Maxime Bouthors, Josep Crego, François Yvon
SYSTRAN by ChapsVision, 5 rue Feydeau, F-75002 Paris, France · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France