Across multiple language pairings (English → {Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese}), we find reasoning errors in translation. To quantify how often these reasoning errors occur, we leverage an automated annotation protocol for reasoning evaluation wherein the goal is to detect if a reasoning step is any of three error categories: (1) source sentence-misaligned, (2) model hypothesis-misaligned, or (3) reasoning trace-misaligned. We probe the reasoning model with perturbed traces correcting for these identified reasoning errors using an array of weak-to-strong interventions: hedging, removal, re-reasoning after removal, hindsight, and oracle interventions. Experimenting with interventions on the reasoning traces suggests that small corrections to the reasoning have little impact on translation quality, but stronger interventions yield the highest resolution rates, despite translation quality gains being mixed. We find ultimately that reasoning errors in MT can be identified with high precision in Urdu but lower precision in Spanish, but that removing these reasoning errors does not resolve the initial errors significantly, suggesting limited reasoning faithfulness for machine translation.
Figures & tables
Figure 1: A reasoning trace decision in Qwen3-8B (FLORES, en → es) maps a chick’s down to plumilla (quill/pen nib) instead of plumón (down feathers). The error propagates to the output.
Output Valid (O✓)
Output Invalid (O ✗ )
Reasoning Valid (R✓)
Potentially aligned : trace has no detected issues and the translation has no detected issues.
Silent translation error : translation is wrong, but the trace does not contain detectable issues.
Reasoning Invalid (R ✗ )
Right answer, wrong reason : translation is correct, but the trace contains a detected issue. This may cause reduced trust in good translations.
Wrong answer, wrong reason : translation is incorrect, and trace has semantic divergences. Highlighting this may help users reject imperfect translations.
Table 1: A 2 × 2 matrix framing the validity of reasoning and translation outputs. In this work, we analyze the detection of invalid outputs, conditioned on invalid reasoning R ✗ .
Category
Question
Example
Connection to Lee and Hockenmaier (2025)
Input– Reasoning
Does the trace make claims supported by the source sentence?
Source : “The bug crashed the whole system.” Step 3 : “The word bug here means a biological organism ; I’ll translate it as bacteria .” Explanation: The trace selects the wrong sense of bug ; the source context (software) does not support this interpretation.
In the taxonomy of Lee and Hockenmaier (2025) , this corresponds most directly to factuality , especially groundedness with respect to the source. Analogous to groundedness in NLG evaluation ( Honovich et al., 2022 ; Maynez et al., 2020 ) : just as a summary can hallucinate content absent from a source document, a trace can hallucinate translation equivalences.
Reasoning– Output
Does the output reflect the decisions stated in the trace?
Step 8 : “The phrase ‘pretending to listen’ maps naturally to haciendo como si estuviera escuchándonos .” Step 9 : “I’m confident in this phrasing. Finalizing the translation.” Output : “Ella estaba haciéndose pasar por si estuviera escuchándonos .” Explanation: The output uses a different phrase than the one committed to in Steps 8–9.
In the taxonomy of Lee and Hockenmaier (2025) , this aligns most closely with utility : whether a reasoning step actually contributes to the final answer. Prior work on CoT faithfulness asks a related question: whether reasoning reflects the model’s actual process ( Lanham et al., 2023 ; Jacovi and Goldberg, 2020 ) .
Reasoning- Internal
Is the trace internally consistent and coherent?
Step 5 : “ bank means el banco .” Step 8 : “Wait—since it is next to ‘river’, it should be la orilla .” Step 9 : “Alright, going with el banco as decided earlier.” Explanation: Step 9 reverts to Step 5 without acknowledging the correction in Step 8.
In the taxonomy of Lee and Hockenmaier (2025) , this aligns with both coherence and validity . Coherence is concerned with whether a reasoning step’s preconditions are established consistently, while validity is concerned with whether the step contains a logical inconsistency. Draws also from detection of self-contradictory reasoning in LLM outputs ( Liu et al., 2024 ) .
Table 2: Error categories for reasoning errors in machine translation, grounded in the reasoning-trace evaluation taxonomy of Lee and Hockenmaier (2025) . Highlighted spans in the example indicate the error span to focus on.
Model
Language pair
# Samples
# w/ errors
Error rate
Avg. steps / trace
Avg. errors / sample
Qwen3-8B
en → es
100
56
56.0%
39.12
1.40
en → zh
100
54
54.0%
32.69
1.86
en → ja
100
86
86.0%
40.88
3.51
en → fr
100
73
73.0%
42.31
2.81
en → de
100
81
81.0%
45.64
2.94
Table 3: LLM-as-a-judge reasoning errors on FLORES devtest, n=100 per direction.
English → Urdu
English → Spanish
Study size
Samples annotated
30
30
Issues annotated
189
58
Translation correctness (majority vote per sample)
Yes (no major error)
0/30 (0.0%)
24/30 (80.0%)
No (major error)
28/30 (93.3%)
5/30 (16.7%)
Table 4: Human validation summary for English → Urdu and English → Spanish. Translation correctness and reasoning-error validation are reported at the sample and issue level respectively, using majority vote across three annotators. Error reflection rates are computed over annotations labeled Yes or Borderline .
Qwen3-8B
Ministral-3-8B
Intervention
Resolved / total
Rate ↑
Δ COMET ↑
Resolved / total
Rate ↑
Δ COMET ↑
en-de
hedging
2 / 149
1.3%
0.0000
48 / 116
41.4%
-0.0010
hindsight
125 / 149
83.9%
+0.0088
103 / 116
88.8%
+0.0001
oracle-1
47 / 149
31.5%
+0.0038
78 / 116
67.2%
+0.0006
oracle-k
45 / 149
30.2%
+0.0025
77 / 116
66.4%
+0.0027
Table 5: Intervention results grouped by language pair. Bold marks the highest resolution rate and the highest Δ COMET within each language section. Positive Δ COMET values are shown with an explicit + sign. Resolved / total preserves the model-specific denominators from the original results.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Bilingual human annotation protocol used to validate final translation correctness and detected reasoning errors. Part (A) is applied once per translation. Part (B) is applied once per detected reasoning error. Q3 is multi-select. For cost reasons, we run this protocol on English → Spanish and English → Urdu.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.