cs.CLOct 11, 2026
SaveFeedback as an Interface: Controlling Form and Detail in LLM Translation Refinement
Organizations: University of Manchester · Osapiens · University of Sheffield · University of Technology Nuremberg
Abstract
Feedback presentation shapes how LLMs use evaluator judgments during translation refinement. We introduce a machine translation protocol that keeps the evaluator score, main issue, and recommendation fixed while varying feedback form and supporting detail. Across seven models and six translation directions, low-load feedback outperforms high-load feedback in all 42 translator-direction settings under xCOMET. On the 29,400-record rewriting subset, mean xCOMET scores multiplied by 100 are 83.30 for low-load feedback and 80.67 for high-load feedback; the full-matrix original-translation reference mean is 83.46. MetricX-24 supports the low-load advantage on a stratified subset. A no-feedback second pass yields a small improvement over the original translations, and direct evaluator prompting reproduces the low-load advantage without a separate rewriting stage. These results show that compact feedback limits the quality loss from evaluator-guided refinement in the evaluated high-resource MT settings.
Figures & tables
Figure 1: Controlled comparison for feedback presentation. The score, main issue, and recommendation are fixed, while the feedback is rewritten by tone, format, or information load before revision by the same translator.
| Factor | Values and change |
| Tone | Positive, neutral, negative. Changes evaluative wording. |
| Response format | Structured, free-text. Changes organization. |
| Information load | Low load, high load. Changes supporting detail. |
Table 1: Controlled feedback dimensions. Tone and response format are surface-form manipulations; information load changes the amount of supporting detail shown to the reviser.
| Variant | Feedback shown to the reviser |
| Raw | The translation captures the main idea of the German sentence but has issues with accuracy and naturalness. The message about heart disease as a leading cause of death is conveyed. However, “more than any other single factor” misrenders “als alles andere zusammen,” which means “than all others combined.” The last sentence is also off: the German says heart disease is preventable for almost all people, while “almost preventable in everyone” changes the emphasis. A score around 75 reflects the general idea with inaccuracies and awkward phrasing. |
| Low load | Score: 75/100. Main issue: two meanings are inaccurate, “than all others combined” and “preventable for almost all people.” Recommendation: revise those phrases for accuracy and naturalness. |
| High load | The translation captures the main idea, but several accuracy and naturalness issues remain. The opening correctly conveys that heart disease is a leading cause of death worldwide. The comparative phrase weakens the source meaning: “als alles andere zusammen” means “than all others combined”; the translation gives “more than any other single factor.” The final sentence also changes the emphasis. The German says heart disease is preventable for almost all people; “almost preventable in everyone” sounds awkward and suggests a different meaning. A score around 75 is appropriate because the message is understandable, but these two places need revision. |
| Structured | Overall: acceptable with notable inaccuracies. Key reasons: the core message is correct; “more than any other single factor” weakens “than all others combined”; “almost preventable in everyone” changes the final sentence. Recommendation: revise the comparative phrase and final sentence. Rating: 75/100. |
Table 2: Feedback example for one German-to-English item with a 75/100 evaluator score. The table lightly shortens logged feedback for space. All rows keep the same score, main issue, and recommendation, while varying how the critique is organized and how much supporting detail is included. Bold labels in the structured variant reflect the labeled-field format; other variants are shown without added formatting.
Table 3: Base translation quality (xCOMET ) by translator and language pair.
| Translator | Original | Self Feedback | Best Cross-Critic | Avg. over Critics |
| Mistral-7B | 79.26 | 75.16 | 80.53 | 78.47 |
| Llama3.1-8B | 83.56 | 80.70 | 83.21 | 81.86 |
| Qwen2.5-7B | 81.66 | 81.89 | 83.70 | 82.52 |
| Qwen2.5-14B | 83.76 | 82.49 | 84.06 | 83.15 |
| Gemma3-12B | 84.84 | 82.72 | 84.08 | 82.68 |
| Gemma3-27B | 85.60 | 83.47 | 84.06 | 83.41 |
Table 4: Mean refinement quality (xCOMET ) across six translation directions. Self feedback uses the same model as translator and evaluator. For each translator and translation direction, Best Cross-Critic selects the non-identical evaluator with the highest mean refinement score, then averages the selected scores across directions. Selection uses the evaluated outputs. Bold values exceed the original translation baseline.
| Condition | xCOMET |
| Original translation (full matrix) | 83.46 |
| Raw refinement (full matrix) | 82.13 |
| Low-load feedback | 83.30 |
| High-load feedback | 80.67 |
Table 5: Mean xCOMET scores. Original translation and raw refinement are full-matrix reference means; low-load and high-load feedback are means on the 29,400-record rewriting subset.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Model Name | Name of Checkpoint | Training Data Cutoff | Before Nov. 2024? |
| Llama-3.1-8B | meta-llama/Llama-3.1-8B-Instruct | 2023-12 | ✓ |
| Qwen2.5-7B | Qwen/Qwen2.5-7B-Instruct | Not publicly disclosed (before 2024-09) | ✓ |
| Qwen2.5-14B | Qwen/Qwen2.5-14B-Instruct | Not publicly disclosed (before 2024-09) | ✓ |
| Mistral-7B | mistralai/Mistral-7B-Instruct-v0.3 | Not publicly disclosed (before 2024-05) | ✓ |
| Gemma3-12B | google/gemma-3-12b-it | 2024-08 | ✓ |
| Gemma3-27B | google/gemma-3-27b-it | 2024-08 | ✓ |
Table 6: Selected models used as both translators and evaluators.
| Dimension | What changes | Values | Held fixed |
| Tone | Surface wording and sentiment | Positive, neutral, negative | Score, main issue, recommendation |
| Response format | Organization of the same critique | Structured, free-text | Score, main issue, recommendation |
| Information load | Amount of supporting detail shown to the reviser | Low load, high load | Score, main issue, recommendation |
Table 7: Controlled feedback dimensions used in the final experiments. Tone and response format manipulate surface form; information load manipulates accessible detail while keeping the core judgment fixed.
Figure 2: Appendix heatmaps for English-source directions. The top panel shows English German and the bottom panel shows English Spanish. Positive cells indicate that refinement helps relative to the original translation; negative cells indicate degradation.
Figure 3: Appendix heatmaps for English Chinese (top) and German English (bottom). These panels contrast one harder Chinese-target setting with one easier English-target setting under the same translator and evaluator grid.
Figure 4: Appendix heatmaps for the remaining German-source directions: German Spanish (top) and German Chinese (bottom). Across all appendix panels, warmer colors indicate mean improvement and cooler colors indicate mean degradation relative to the original translation.
| Condition | Mean | vs. Raw |
| Raw refinement (reference) | 82.13 | 0.00 |
| Structured | 83.69 | +1.56 |
| Free-text | 83.43 | +1.30 |
| Neutral | 83.50 | +1.37 |
| Positive tone | 81.78 | -0.35 |
| Negative tone | 81.69 | -0.45 |
Table 8: Tone and response-format results under xCOMET on the rewriting data. Raw refinement is shown as a full-matrix reference mean; differences are relative to that reference.
Figure 5: Setting-level xCOMET differences relative to the raw-refinement reference across 42 translator-direction settings.
Figure 6: Mean xCOMET scores across two refinement rounds on the English German and German Chinese subset. Raw feedback has a higher round-one score, and the conditions reach similar scores in round two. Table 9 reports results by translator.
| Translator | Raw | Near-id. | Low load |
| Mistral-7B | 79.7 74.2 (-5.5) | 75.5 74.6 (-0.9) | 77.1 75.4 (-1.7) |
| Qwen2.5-14B | 82.1 78.9 (-3.2) | 78.5 79.3 (+0.8) | 78.1 76.9 (-1.2) |
| Gemma3-27B | 84.0 83.2 (-0.8) | 84.3 82.8 (-1.5) | 83.1 83.1 (+0.1) |
| All translators | 81.9 78.8 (-3.2) | 79.4 78.9 (-0.5) | 79.4 78.5 (-0.9) |
Table 9: Two-round xCOMET results on the English German and German Chinese subset. Cells show round-one and round-two scores; parentheses show the change from round one to round two.
More in this topic
Machine Translation
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
See all 267 papers