cs.CLAug 30, 2026

Confident but Wrong: A Constrained Decoding Diagnostic for Low-Resource Automatic Post-Editing

Authors: Isuru Wijesiri, Nisansa de Silva, Kavindu Warnakulasuriya, Aloka Fernando, Surangika Ranathunga

Organizations: WSO2, Sri Lanka · Dept. of CSE, University of Moratuwa, Sri Lanka · National University of Singapore · Massey University, New Zealand

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Automatic Post-Editing (APE) for low-resource languages (LRLs) often fails to improve Machine Translation (MT), and the score alone cannot say why: whether more training would help, or whether the training data is too inconsistent to learn from. We introduce a black-box, inference-time diagnostic that tells these two cases apart without retraining or annotation. It varies an edit-distance penalty λλ that drives the model from free editing towards copying the MT, and reads two signals: (1) the shape of the Translation Edit Rate (TER)-vs-λλ curve, U-shaped if edits from the model reduce error and monotonically decreasing if none does; and (2) the ordering of constraint variants that trust model confidence to increasing degrees, which shows whether confidence tracks edit quality. Across decoder-only and encoder-decoder models on English-Sinhala, the diagnostic exposes two failure modes consistent with a heterogeneous post-edit signal as the underlying cause: Binary Collapse, where the model copies the MT or makes off-target edits, and Confident Miscalibration, where the confidence signals we test do not separate useful edits from unnecessary ones. The pattern holds on English-Marathi and English-Tamil, with the failure modes tracking the post-edit distribution rather than MT quality or language family. Beyond diagnosis, the curve shape prescribes a concrete next step for practitioners; in the favorable case, a static constraint yields a free inference-time accuracy gain. We release the first English-Sinhala (~66k) and a new English-Tamil (~39k) APE datasets with all code.

Figures & tables

 D1D2D3D4D5
 Lit.NewsWikiMath/HNLLB
Train5,72813,30517,4715,75410,332
Valid7161,6632,1847191,291
Test7161,6642,1847201,292
Total7,16016,63221,8397,19312,915
MT TER24.987.9314.8924.3059.19
Table 1: Experimental splits D1–D5 ( ∼{\sim} 66k triplets). D1–D4 use Google Translate MT; D5 is a contrastive low-quality NLLB-1.3B condition. MT TER is test-set TER of unedited MT vs. PE. Full statistics in Appendix Table 12 .
Figure 1: TER vs. λ\lambda curves (Gemma 3 1B, seed 42). Left: D1 (TER 24.98): monotonic. Center: D5 (TER 59.19): U-shaped, min λ=2.0\lambda{=}2.0 . Right: En-Mr (WMT22, TER 22.93): U-shaped, min λ=2.0\lambda{=}2.0 . En-Mr’s MT quality matches D1 yet its curve is U-shaped: the pattern is consistent with a post-edit-distribution effect, not an MT-quality effect.
MethodLogicTER ↓\downarrowΔ\Delta TER
MT Baseline–24.98–
Vanilla FT–25.85 ±\pm 0.02−- 0.87
λentropy\lambda_{\text{entropy}}Trust if confident25.72 ±\pm 0.09−- 0.74
λpmt\lambda_{\text{pmt}}Trust if hates MT25.37 ±\pm 0.25−- 0.39
λseq-NLL\lambda_{\text{seq-NLL}}Trust if seq. confident25.32 ±\pm 0.21−- 0.34
λstatic\lambda_{\text{static}}Don’t trust model24.98 ±\pm 0.080.00
Table 2: Constrained decoding on D1 (Gemma 3 1B, mean ±\pm std across 3 seeds). The Static >> Probability >> Entropy ordering is the signature of Confident Miscalibration (zero-shot: Appendix A , Table 9 ). The sequence-level variant patterns with the token-level ones, so the ordering is not an artifact of token-level signals.
Token Category%Mean ppMean HH
Correct copy64.50.950.30
Missed edit29.00.950.33
Unnecessary edit5.20.800.91
Correct edit1.20.810.84
Table 3: Token-level confidence for Gemma 3 1B on D1 (vanilla decoding). pp : mean token probability; HH : mean output entropy.
Token CategoryD1D2D5
Correct copy64.5 (.95)87.0 (.99)29.4 (.90)
Missed edit29.0 (.95)7.6 (.97)36.9 (.86)
Unnecessary edit5.2 (.80)3.7 (.90)28.4 (.77)
Correct edit1.2 (.81)1.7 (.95)5.3 (.84)
Table 4: The Table 3 breakdown repeated on D2 (clean MT) and the U-shaped D5 (Gemma 3 1B, seed 42). Each cell gives the token share in % (mean pp in parentheses).
ModelMethodTER ↓\downarrowBLEU ↑\uparrowΔ\Delta TER
Gemma 3 1B (decoder-only, LoRA):
 Base (zero-shot)190.318.88−- 165.33
 Vanilla FT25.8567.16−- 0.87
 λentropy\lambda_{\text{entropy}}25.7267.23−- 0.74
 λpmt\lambda_{\text{pmt}}25.3767.45−- 0.39
 λstatic\lambda_{\text{static}}24.9867.380.00
Table 5: Cross-model constrained decoding on D1. All three architectures show the same ordering: Static >> Probability >> Entropy.
  TER ↓\downarrowBLEU ↑\uparrowΔ\Delta TER
D1: High-quality MT (Google Translate):
 MT Baseline24.9867.38–
 Vanilla FT25.8567.16−- 0.87
 λstatic\lambda_{\text{static}}24.9867.380.00
D5: Low-quality MT (NLLB-1.3B), Gemma 1B:
 MT Baseline59.1930.54–
Table 6: D1 vs. D5 (seed 42). On D5 (genuine error correction) both Gemma 1B and NLLB-600M improve over MT; constrained decoding further improves Gemma 1B, and Static >> PMT >> Entropy still holds.
SystemTER ↓\downarrowBLEU ↑\uparrowΔ\Delta TER
MT Baseline22.9367.01–
Vanilla FT22.2469.81+0.69
λentropy\lambda_{\text{entropy}}22.0669.95+0.87
λpmt\lambda_{\text{pmt}}21.8369.96+1.10
λstatic\lambda_{\text{static}}21.7369.72+1.20
Table 7: English–Marathi validation (Gemma 3 1B, seed 42, WMT22 APE dev set; test refs unavailable). Static >> PMT >> Entropy holds cross-lingually. Full sweeps in Appendix Table 19 .
SystemTER ↓\downarrowBLEU ↑\uparrowΔ\Delta TER
MT Baseline20.5773.17–
Vanilla FT21.7771.39−- 1.21
λstatic\lambda_{\text{static}}20.6873.00−- 0.11
Table 8: English–Tamil validation (News, Gemma 3 1B, seed 42, static constraint λ=5.0\lambda{=}5.0 ). The monotonic λ\lambda curve approaches but never crosses the MT baseline. Full sweep in Appendix Table 20 ; per-domain in Table 22 .
Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

StrategyTER ↓\downarrowBLEU ↑\uparrowchrF++ ↑\uparrow
MT Baseline28.8263.2477.96
Zero-shot37.5154.9072.84
Fixed few-shot34.9357.6174.44
RAG ( k=5k{=}5 )37.9254.4772.06
Two-stage (gate+edit)36.7555.2873.47
Appendix
Table 9: GPT-5.2 APE results on 200 stratified D1 test samples (same samples as edit-type analysis, Section 3.2 ). All prompting strategies degrade substantially over the MT baseline. RAG performs worse than both fixed few-shot and zero-shot; the two-stage decomposition (Section A.3 ) is intermediate and shows the same failure mode for structured prompting.
MetricValueBaseline
TER ↓\downarrow36.75MT 28.82
Gate precision0.75“always edit” 0.73
Gate recall0.99–
Gate F10.85–
Edits called192/200 (96%)–
Appendix
Table 10: Two-stage decomposition with GPT-5.2 on 200 stratified D1 test samples. Confusion matrix: TP=144, FP=48, TN=6, FN=2. Gold edit rate (positive class proportion) is 0.73; the “always edit” baseline therefore achieves gate precision 0.73 by definition. GPT-5.2’s gate barely improves on this (0.75), and triggers an edit on 96% of inputs, so the pipeline collapses to vanilla zero-shot APE for almost all samples.
#User (Source + MT)Assistant (Post-Edit)Type
1Source: When he saw who his visitor was his eyes brightened. MT: තම අමුත්තා කවුදැයි දුටු විට ඔහුගේ දෑස් දීප්තිමත් විය.තම අමුත්තා කවුදැයි දුටු විට ඔහුගේ දෑස් දීප්තිමත් විය.Copy
2Source: They went to the football field and started their game. MT: ඔවුන් පාපන්දු පිටියට ගොස් ඔවුන්ගේ ක්‍රීඩාව ආරම්භ කළහ.ඔවුන් පාපන්දු පිටියට ගොස් ඔවුන්ගේ ක්‍රීඩාව ආරම්භ කළහ.Copy
3Source: Monte Cristo bowed, and went to Madame de Villefort. MT: මොන්ටේ ක්‍රිස්ටෝ හිස නමා මැඩම් ඩි විල්ෆෝර්ට් වෙත ගියේය.මොන්ටේ ක්‍රිස්ටෝ හිස නමා , මැඩම් ඩි විල්ෆෝර්ට් වෙත ගියේය.Minor
4Source: “I do not know her name; but it is she, sir, it is she!” MT: “මම ඇගේ නම දන්නේ නැහැ; නමුත් ඒ ඇයයි, සර්, මේ ඇයයි!”“මම ඇගේ නම දන්නේ නැහැ; නමුත් ඒ ඇයයි, මහත්මයා, ඒ මේ ඇයයි!”Minor
5Source: He melted in the hot sun. MT: ඔහු උණුසුම් හිරු තුළ දිය වී ගියේය.ඔහු උණුසුම් හිරු එළියේ දිය වී ගියේය.Major
Appendix
Table 11: Fixed few-shot examples for GPT-5.2. Examples 1–2 show “copy” behavior (TER == 0), 3–4 show minor edits, and 5 shows a major edit. All from D1 training set.
DomainTripletsMean TEREdit Dist.
Literature7,20925.730/21/49
News16,6329.951/33/16
Wikipedia21,83914.925/45/30
Maths6,48723.329/26/45
Health1,00042.113/18/69
NLLB Corpus12,91561.02/5/93
Appendix
Table 12: English–Sinhala APE dataset release statistics. Mean TER is the corpus-level mean TER(MT, PE) over all triplets before cleaning. Edit Dist. shows none/minor/major percentages (TER thresholds: 0, 20). Literature, News, Wikipedia, Maths, and Health use Google Translate MT; NLLB Corpus uses NLLB-1.3B MT. Experimental splits (D1–D5) derived from these domains after cleaning are in Table 1 .
DomainTripletsMean TEREdit Dist.
Wiki19,86412.741/37/23
News10,00020.244/20/36
Literature3,50126.842/13/45
Maths3,42323.625/30/44
Health1,8707.369/17/15
Total38,658––
Appendix
Table 13: English–Tamil APE dataset statistics. Edit Dist. shows the percentage of none/minor/major edits (TER thresholds: 0, 20). All domains use Google Translate MT. Experiments in this paper use only News; remaining domains are released for future research.
  Seed 42Seed 52Seed 62
ModelMethodTERBLEUchrF++TERBLEUchrF++TERBLEUchrF++
Gemma 1BVanilla25.8467.3479.5825.8467.0679.6525.8867.0779.49
HSA25.2667.4279.7825.8967.2079.5825.6067.2479.62
λentropy\lambda_{\text{entropy}}25.8167.3079.5525.6567.2479.7125.6967.1479.54
λpmt\lambda_{\text{pmt}}25.6567.3479.6525.1867.5979.8825.2967.4379.72
λstatic\lambda_{\text{static}}24.9767.3179.8625.0767.4379.8624.9167.3979.86
Appendix
Table 14: Full per-seed results on D1 (Literature). Most constrained decoding methods improve over vanilla fine-tuning, with static constraint consistently achieving the best results; entropy is the exception, trailing vanilla on NLLB. † Gemma 4B seed 62 diverged during QLoRA training; seed 72 used instead.
Figure 2: Token-level confidence vs. edit correctness for Gemma 3 1B on D1. 59.4% of unnecessary edits have p>0.9p>0.9 , indistinguishable from the 54.8% of correct edits at p>0.9p>0.9 : per-token probability does not separate net-positive edits from unnecessary ones. The 84.8% of missed edits at p>0.9p>0.9 shows confidence is equally uninformative about whether to edit at all.
MethodTER ( T=1T{=}1 )TER ( T=2T{=}2 )
MT Baseline24.9824.98
Vanilla FT ( λ=0\lambda{=}0 )25.8427.63
λentropy\lambda_{\text{entropy}} (best)25.8127.40
λpmt\lambda_{\text{pmt}} (best)25.6525.66
λstatic\lambda_{\text{static}} (best)24.9725.08
Appendix
Table 15: Effect of temperature scaling ( T∗=2.0T^{*}{=}2.0 , tuned on val NLL) on D1, seed 42. Calibration worsens unconstrained generation (25.84 →{\to} 27.63) and prevents the static constraint from recovering the MT baseline. The ordering Static >{>} PMT >{>} Entropy is preserved.
λ\lambdaStaticEntropyPMT
0.025.84––
0.125.81––
0.325.66––
0.525.6525.8125.79
1.025.0025.8125.69
2.024.9825.8125.65
Appendix
Table 16: TER vs. λ\lambda sweep (D1, Gemma 3 1B, seed 42). Static converges to MT-level quality at λ≥1.0\lambda{\geq}1.0 . Entropy is flat ( ∼{\sim} 25.81 at all λ\lambda ): the constraint barely activates. PMT shows a real curve with minimum at λ=2.0\lambda{=}2.0 , but never approaches the MT baseline.
λ\lambdaTER ↓\downarrowΔ\Delta vs MT
MT Baseline59.19–
0.057.16+2.03
0.156.99+2.20
0.357.12+2.07
0.557.03+2.16
0.756.75+2.44
Appendix
Table 17: TER vs. λ\lambda sweep (D5, Gemma 3 1B, seed 42, static constraint). Unlike D1’s monotonic curve, D5 shows a U-shape: TER dips below both vanilla FT (56.89) and MT baseline at λ=2.0\lambda{=}2.0 , then rises at λ=5.0\lambda{=}5.0 as the constraint forces copying of low-quality MT.
λ\lambdaTER ↓\downarrowΔ\Delta vs MT
MT Baseline59.19–
−- 5.077.37−- 18.18
−- 3.071.24−- 12.05
−- 2.066.05−- 6.87
−- 1.061.10−- 1.91
−- 0.558.87+0.31
Appendix
Table 18: TER vs. λ\lambda sweep (D5, NLLB-600M, seed 42, static constraint). The curve is U-shaped with its minimum at λ=0\lambda{=}0 (vanilla FT). Negative λ\lambda penalizes MT tokens (encouraging hallucination); positive λ\lambda forces MT copying. Both directions degrade performance, with the negative direction steeper, consistent with low-quality MT structure providing useful scaffolding. Values saturate at ∣λ∣≥12|\lambda|{\geq}12 .
λ\lambdaλstatic\lambda_{\text{static}}λpmt\lambda_{\text{pmt}}λentropy\lambda_{\text{entropy}}
MT Baseline22.93
Vanilla FT22.24
0.521.9822.2522.21
1.021.8322.1922.15
2.021.7322.0522.10
3.021.8321.9022.09
Appendix
Table 19: TER vs. λ\lambda sweep (English–Marathi, Gemma 3 1B, seed 42). All three constraint methods shown. The static constraint produces a U-shaped curve with best TER 21.73 at λ=2.0\lambda{=}2.0 . The ordering Static >> PMT >> Entropy holds at every λ\lambda value tested. Entropy TER is nearly flat across λ\lambda (same pattern as D1 and D5), confirming that the entropy-scaled constraint barely activates.
λ\lambdaλstatic\lambda_{\text{static}} TER
MT Baseline20.57
Vanilla FT21.77
0.524.24
1.023.34
2.021.46
3.020.89
Appendix
Table 20: TER vs. λ\lambda sweep (English–Tamil News, Gemma 3 1B, seed 42, static constraint). The curve is monotonic, approaching but never crossing the MT baseline (20.57). Plotted in Figure 3 (a).
λ\lambdaTER ↓\downarrowΔ\Delta vs MT
MT Baseline47.54–
Vanilla FT50.90−- 3.36
0.553.42−- 5.87
1.051.08−- 3.54
2.048.16−- 0.62
3.047.72−- 0.18
Appendix
Table 21: TER vs. λ\lambda sweep (English–Tamil News with NLLB-1.3B MT, Gemma 3 1B, seed 42, static constraint). Same post-edits as Table 20 but with synthetic NLLB MT. The curve is U-shaped with minimum at λ=3.0\lambda{=}3.0 , then plateaus at ∼{\sim} 48.2 for λ≥7\lambda{\geq}7 . Plotted in Figure 3 (b).
Figure 3: English–Tamil News λ\lambda curves: same post-edits, different MT. (a) Google Translate MT (TER MT=20.57{}_{\mathrm{MT}}{=}20.57 ): monotonic, never crosses MT baseline. (b) NLLB-1.3B MT (TER MT=47.54{}_{\mathrm{MT}}{=}47.54 ): U-shaped with minimum at λ=3.0\lambda{=}3.0 (TER  = 47.72\,{=}\,47.72 ). The within-language contrast is consistent with a task-difficulty effect rather than a language-specific effect. Corresponding sweep values in Tables 20 and 21 .
DomainMT TERFT TERΔ\Delta TERTriplets
Wiki12.6712.95−- 0.2823,880
News20.5721.77−- 1.2110,000
Combined †19.0718.96++ 0.118,794
Appendix
Table 22: English–Tamil per-domain vanilla fine-tuning results (Gemma 3 1B, seed 42). MT source is Google Translate for all domains. Constrained decoding experiments are conducted only on the News domain (Section 5.5 ). † Literature (3,501), Maths (3,423), and Health (1,870) are combined into a single split because individually each is too small for stable fine-tuning; Literature alone failed to converge.
λ\lambdaTER ↓\downarrowΔ\Delta vs MT
MT Baseline7.93–
0.07.91+0.02
0.59.17−- 1.24
1.08.32−- 0.39
2.08.25−- 0.33
2.58.12−- 0.19
Appendix
Table 23: TER vs. λ\lambda sweep (D2 News, Gemma 3 1B, seed 42, static constraint). Vanilla FT already marginally improves over MT; the constraint monotonically worsens TER, confirming the model already makes net-positive edits and no inference-time correction is needed.
λ\lambdaTER ↓\downarrowΔ\Delta vs MT
MT Baseline14.89–
0.014.88+0.01
0.515.01−- 0.12
1.014.71+0.19
2.014.45+0.45
2.514.45+0.44
Appendix
Table 24: TER vs. λ\lambda sweep (D3 Wikipedia, Gemma 3 1B, seed 42, static constraint). The constraint beats the MT baseline at λ≥1.0\lambda{\geq}1.0 , with best TER 14.45 at λ=2.0\lambda{=}2.0 ( Δ\Delta TER =+0.45{=}+0.45 ). Mild U-shape with uptick at λ=3.0\lambda{=}3.0 .
λ\lambdaTER ↓\downarrowΔ\Delta vs MT
MT Baseline24.30–
0.024.99−- 0.68
0.525.93−- 1.63
1.024.57−- 0.27
2.024.09+0.21
2.524.58−- 0.27
Appendix
Table 25: TER vs. λ\lambda sweep (D4 Maths/Health, Gemma 3 1B, seed 42, static constraint). Despite vanilla FT degrading TER by 0.680.68 , the U-shaped curve reveals genuine correctable errors, with best TER 24.09 at λ=2.0\lambda{=}2.0 ( Δ\Delta TER =+0.21{=}+0.21 , beats MT baseline).
DatasetSystemTER ↓\downarrowBLEU ↑\uparrowΔ\Delta TER
D2 (News)MT Baseline7.9389.25–
Vanilla FT7.9189.56+0.02
λstatic\lambda_{\text{static}}8.1289.39−- 0.19
D3 (Wiki)MT Baseline14.8979.11–
Vanilla FT14.8878.95+0.01
λstatic\lambda_{\text{static}}14.4579.20+0.45
Appendix
Table 26: Domain robustness (Gemma 3 1B, seed 42). Vanilla FT degrades TER on D1 ( −- 0.87) and D4 ( −- 0.68), but the static constraint reveals different underlying patterns: D4 shows a U-shaped curve that beats MT ( Δ\Delta TER =+0.21{=}+0.21 ), while D1 only recovers to MT level. D3’s constraint similarly reveals correctable errors ( Δ\Delta TER =+0.45{=}+0.45 ). D2 is the opposite case: vanilla FT already marginally improves, and any constraint worsens TER ( −- 0.19), so no tested constraint improves this result. Only on D1 (Literature) do no net-positive edits survive.
Figure 4: TER vs. λ\lambda curves across model architectures. Left: Gemma 4B on D1 (seed 42): weakly U-shaped, the only model to dip below the MT baseline ( Δ\Delta TER =+0.15{=}+0.15 at λ=2.0\lambda{=}2.0 ). Compare with the monotonic 1B curve in Figure 1 (a). Center: NLLB-600M on D1 (seed 42): decreasing until λ=5\lambda{=}5 , then plateauing at TER ≈29.8{\approx}29.8 ( λ=15\lambda{=}15 and λ=20\lambda{=}20 give identical results, confirming constraint saturation). Best TER 29.59 at λ=5.0\lambda{=}5.0 , still 4.61 above MT; the encoder–decoder’s over-editing is too severe for an inference-time constraint to undo. Right: NLLB-600M on D5 (seed 42): U-shaped, centered at λ≈0\lambda{\approx}0 ; neither constraint direction improves performance. Negative λ\lambda penalizes MT-window tokens; the steeper left arm is consistent with MT structure providing useful scaffolding. The D1/D5 contrast is consistent with task differences rather than architecture alone.
Figure 5: TER vs. λ\lambda curves for D2–D4 (Gemma 3 1B, seed 42, static constraint). Left: D2 (News, TER MT=7.93{}_{\mathrm{MT}}{=}7.93 ): vanilla FT marginally improves over MT; the constraint monotonically worsens TER, confirming the model’s edits are already net-positive. Center: D3 (Wikipedia, TER MT=14.89{}_{\mathrm{MT}}{=}14.89 ): vanilla FT shows negligible change ( Δ\Delta TER =+0.01{=}+0.01 ), but the static constraint reveals genuine correctable errors, reaching TER  = 14.45{\,=\,}14.45 at λ=2.0\lambda{=}2.0 ( Δ\Delta TER =+0.45{=}+0.45 ); mild U-shape with uptick at λ=3.0\lambda{=}3.0 . Right: D4 (Maths/Health, TER MT=24.30{}_{\mathrm{MT}}{=}24.30 ): vanilla FT degrades TER ( −- 0.68), but unlike D1 the curve is U-shaped with best TER  = 24.09{\,=\,}24.09 at λ=2.0\lambda{=}2.0 ( Δ\Delta TER =+0.21{=}+0.21 ), revealing genuine correctable errors despite similar MT quality.
ParameterGemma 1BGemma 4BNLLB
Learning rate3e-53e-52e-4
SchedulerLinearLinearLinear
Warmup ratio0.100.100.10
Batch (eff.)6416 ×\times 448 ×\times 1
Max epochs10108
Eval steps5050100
Appendix
Table 27: Training hyperparameters and compute. Batch size is per-device ×\times gradient accumulation = effective. Decoder-only Gemma models are fine-tuned to generate the post-edit from a Source: {src} \n MT: {mt} prompt; NLLB-600M takes src </s> mt as encoder input.
Edit TypeExample (MT →\to PE)
Stylistic (MT semantically correct):
Loanword →\to nativeස්පයිඩර් →\to මකුළුවා ( “Spider” →\to native term )
Register shiftදනිමු →\to දන්නවා ( literary →\to spoken “know” ; shifting between diglossic forms de Silva (2026) )
Word order (SVO →\to SOV)Clause reordering to Sinhala-preferred order
RetranslationComplete rewrite, same meaning
Correction (actual MT error):
Appendix
Table 28: Edit type taxonomy from manual analysis of 90 (MT, PE) pairs across D1–D3 (30 per domain). Edits span both stylistic and corrective categories with no single type dominating, particularly in D1 (Literature).
DomainSource (English)Key Edit TypeTER
D1 (Lit.)Spider replied, “Yes, I’ll take you on an adventure…”Loanword: ස්පයිඩර් →\to මකුළුවා (“Spider” →\to native term) + verb formality27.3
D1 (Lit.)“I didn’t think,” she sobbed.Mistranslation: කෑගැසුවාය (“screamed”) →\to වැළපුණා ය (“sobbed”)33.3
D1 (Lit.)Thank you, Mrs. Goose.Loanword: මිසිස් ගූස් →\to පාත්ත මහත්මිය (English title →\to native)66.7
D1 (Lit.)“Ahmed?” drowsily.Morphological: නිදිමත →\to නිදිමතින් (missing instrumental case -ින්)50.0
Appendix
Table 29: Representative examples from D1, illustrating the heterogeneity of edit types. The examples include loanword replacement, mistranslation correction, and morphological fixes.
MQM CategoryHumanCopiedFixedCopy%
Accuracy5948081.4
Fluency4126763.4
Style8256168.3
Retranslation1712070.6
Total200143871.5
Appendix
Table 30: Edit-type decomposition of Binary Collapse on D1 (Gemma 3 1B, seed 42, 200 samples). Human : number of samples where humans made this type of edit. Copied : model copied MT exactly. Fixed : model output matches PE. Binary Collapse is not selective: the model copies at high rates across all categories. Of 8 successful corrections, 7 are punctuation. (Summarized in Section 4.3 .)
Edit TypeHumanModelGap
Accuracy
mistranslation441045
addition6–6
untranslated423
polysemy_sense3–1
omission212
Appendix
Table 31: Per-type edit classification for D1 (200 samples, Gemma 3 1B, seed 42). Human : MT →{\to} PE edits. Model : MT →{\to} predicted actions. Gap : predicted →{\to} PE remaining differences. The model’s repertoire is limited to punctuation (40) and a few mistranslation fixes (10); all other edit types are effectively abandoned.
 Model Action Category
HumanNoneAcc.Flu.Sty.Ret.
Accuracy48542–
Fluency26–15––
Style566191–
Retranslation12221–
Appendix
Table 32: Cross-tabulation of human edit category vs. model action category on D1 (200 samples). Rows: what humans edited. Columns: what the model did. The “None” column (exact copies) dominates every row. When the model does act on style edits, it typically makes fluency (punctuation) changes instead.
 D2D3D1
 NewsWikiLit.
Human edit distribution (%)
Accuracy7.519.029.5
Fluency68.033.020.5
Style19.041.041.0
Retranslation5.57.08.5
Appendix
Table 33: Cross-domain edit-type profiles and model behavior (200 samples per domain, Gemma 3 1B, seed 42). Subjective % = style + retranslation. Domains ordered by increasing subjectivity. Success rate = model output exactly matches PE.
DomainnnEditEditsper-edit
  rate≠\neq PEΔ\Delta TER
D1 Literature71623.688.8++ 6.3
D2 News166437.772.6−- 0.9
D3 Wikipedia218441.680.6−- 1.1
D5 (low-qual. MT)129279.898.4−- 3.2
Appendix
Table 34: Test-time edit behavior on the full test sets (Gemma 3 1B, seed 42). Edit rate : % of sentences where the output differs from the MT. Edits ≠\neq PE : of those, % not matching the post-edit. per-edit Δ\Delta TER : mean [TER(output,PE)−TER(MT,PE)][\text{TER(output,PE)}-\text{TER(MT,PE)}] over edited sentences; positive means editing moves the output away from the post-edit. Editing is net-negative only on D1, the domain with the most inconsistent post-edit signal; D1, D2, D3 share the same MT (Google Translate), D5 uses NLLB and is shown as a contrast. TER computed with Asian-aware tokenization.
Example 1: Unnecessary Edit (MT == PE, TER ref =0=0 )
Source“I must buy you.”
MT Output“මම ඔබව මිලදී ගත යුතුයි.”
Human PE“මම ඔබව මිලදී ගත යුතුයි.” (identical to MT)
Vanilla FT“මම ඔබව මිලදී ගන්නවා .” (changed ගත යුතුයි “must buy” →\to ගන්නවා “will buy”: unnecessary modality shift)
Constrained“මම ඔබව මිලදී ගත යුතුයි.” (correctly preserved MT, TER =0=0 )
MT is correct. The model changes obligation to simple future at high confidence, altering the meaning.
Appendix
Table 35: Qualitative examples from D1 (all text is real model output from Gemma 3 1B, seed 42; constrained =λstatic=\lambda_{\text{static}} at λ=2.0\lambda{=}2.0 ). Example 1: the model edits perfect MT. Example 2: the model ignores PE’s minor stylistic fixes and instead corrupts a content word. Example 3: the model attempts the same correction as PE but hallucinates the wrong transliteration. Constrained decoding preserves the MT output in all three cases.
FieldContentLabel
Source“I didn’t think,” she sobbed. 
MT“මම හිතුවේ නැහැ,” ඇය කෑගැසුවාය . 
PE“මම හිතුවේ නැහැ,” ඇය වැළපුණා ය . 
Diffකෑගැසුවාය (screamed) →\to වැළපුණා ය (sobbed)ACCURACY
SourceSpider replied, “Yes…” 
MTස්පයිඩර් පිළිතුරු දුන්නේ… 
Appendix
Table 36: Calibration examples for the ACCURACY/STYLE boundary. The first is ACCURACY (“screamed” is a mistranslation of “sobbed”). The second is STYLE (both are valid Sinhala renderings; the native term is a stylistic preference).
CategoryGPT (%)Corrected (%)Agree (%)κ\kappa
Accuracy2629730.64
Fluency2627770.72
Style3128650.62
Retranslation1716820.92
Overall100100730.70
Appendix
Table 37: GPT-5.2 classification validation. GPT : original automated distribution. Corrected : distribution after human corrections. Agree : proportion confirmed by all three annotators. κ\kappa : per-category binary Fleiss’ κ\kappa across the three annotators (overall: multiclass Fleiss’ κ\kappa ).
GPT label →\toAcc.Flu.Sty.Ret.
Human: Accuracy26030
Human: Fluency02610
Human: Style00271
Human: Retranslation00016
Appendix
Table 38: Confusion matrix between GPT-5.2 labels (columns) and consensus human labels (rows) on 100 validation samples. Diagonal entries indicate agreement; off-diagonal entries indicate systematic confusions. GPT overclassifies Style: 3 cases relabeled as Accuracy, 1 as Fluency.
DatasetΔ\Delta TERΔ\Delta chrF++Δ\Delta BLEU
D1 (monotonic)+0.03+0.07+0.05
D5 (U-shaped)+3.98+0.97+2.58
En-Mr (U-shaped)+1.20+0.90+3.03
Appendix
Table 39: Best improvement over the MT baseline along the static λ\lambda sweep, per metric (Gemma 3 1B, seed 42). All deltas are oriented so that positive is better. The monotonic/U-shaped distinction survives under chrF++ and BLEU.
TagGold (%)Predicted (%)Δ\Delta
none29.117.7−- 11.4
minor20.90.1−- 20.8
major50.082.1++ 32.1
Tag accuracy53.9%
Appendix
Table 40: CoT self-tagging: gold vs. predicted edit-level distribution on D1 test set. The minor class is virtually eliminated ( −- 20.8pp), absorbed almost entirely by the major class ( ++ 32.1pp).
MethodTER ↓\downarrowΔ\Delta TERσ\sigmaNote
MT Baseline24.98–––
Vanilla FT25.84−- 0.860.02–
HSA (mean)25.58−- 0.600.26High σ\sigma
ELC (oracle)27.29−- 2.31–Noise on none class
ELC (minor)25.88−- 0.90–≈\approx vanilla
CoT29.55−- 4.57–82% predict [major]
Appendix
Table 41: Failed interventions on D1 (Literature), Gemma 3 1B. HSA is seed-dependent ( σ=0.26\sigma{=}0.26 ); ELC degrades even with oracle tags; CoT collapses to predicting [major] for 82% of samples.
AnchorTargetα\boldsymbol{\alpha}TER ↓\downarrowΔ\Delta
Vanilla FT––25.84–
Zero-scalarhˉmt→0\bar{h}_{\text{mt}}{\to}00.125.26+0.58
MT-vectorh→hfrz\mathbf{h}{\to}\mathbf{h}^{\text{frz}}0.00125.73+0.11
Zero-vectorh→0\mathbf{h}{\to}\mathbf{0}0.00125.69+0.15
Appendix
Table 42: HSA anchor ablation on D1 (seed 42). Zero-scalar performs best but is modest ( ++ 0.58 TER) and highly seed-dependent ( σ=0.26\sigma{=}0.26 ).
Algorithm 1: Constrained Decoding
Input: Model MM , MT tokens xmt\mathbf{x}^{\text{mt}} , type, λ\lambda , window kk , γ\gamma or β\beta
Output: Post-edited tokens y\mathbf{y}
Initialize MT pointer ptr←0\text{ptr}\leftarrow 0
for t=1,2,…t=1,2,\ldots do
lt←M.logits(y<t)\mathbf{l}_{t}\leftarrow M.\text{logits}(\mathbf{y}_{<t})
pt←softmax(lt)p_{t}\leftarrow\text{softmax}(\mathbf{l}_{t})
Appendix
Table 48
ParameterSearch RangeOptimal
λ\lambda (all)0.1, 0.3, 0.5, 1.0, 2.0, 2.5, 3.02.5
γ\gamma (entropy)0.5, 1.0, 2.0, 3.02.0
β\beta (probability)0.5, 1.0, 2.01.0
kk (window)1, 2, 32
Appendix
Table 43: Hyperparameter search ranges and optimal values (tuned on D1 validation set, Gemma 3 1B).
Curve signatureWhat it meansRecommended actionExample
U-shaped: minimum at λ>0\lambda{>}0 , dipping below MTNet-positive edits survived trainingDeploy the static constraint at that λ\lambda as a free inference-time gain (§ 6.3 )D5; English–Marathi; Gemma 4B on D1 (marginal)
Minimum at λ=0\lambda{=}0The model is already net-positive without any constraintDeploy the vanilla fine-tuned checkpoint directlyD2 (News)
Monotonic, reaching the MT baselineNo net-positive edits survived; copying is optimalCopy MT for now; make the post-edits more consistent before training againD1 (Literature)
Monotonic, plateauing above the MT baselineOver-editing is too severe to undo at inferenceAddress data or MT quality before training againNLLB-600M on D1
Appendix
Table 44: The four outcomes read off a single λ\lambda sweep, with the next step each implies. The first two checkpoints are deployable as is; the last two call for upstream work on the training signal (§ 6.3 ).
QE ApproachBest AUROCNote
LaBSE (pretrained)0.45Below chance
COMET-QE (pretrained)0.58Near chance
XLM-R regression0.745Best overall
XLM-R 3-class0.703F1 macro =0.43{=}0.43
XLM-R binary0.732Precision =0.44{=}0.44
XLM-R (train on test)0.707Cannot memorize
Appendix
Table 45: QE approaches on D1. Fine-tuned models plateau at AUROC ∼{\sim} 0.73–0.75, insufficient for reliable gating.
 D1 (heterogeneous) D5 (genuine errors)
MQM CategorynnMean grad norm nnMean grad norm
Accuracy581.58 322.68
Fluency411.46 193.60
Style821.21 1191.53
Retranslation172.04 292.97
Appendix
Table 46: Mean gradient norms (norm of group mean LoRA gradient vector) by MQM category on D1 (heterogeneous edits) and D5 (genuine error correction). Loss CV (within-group coefficient of variation of per-sample loss) on D1: Accuracy 0.88, Fluency 0.86, Style 1.11, Retranslation 1.64.
 D1 (heterogeneous)D5 (genuine errors)
 Acc.Flu.Sty.Ret.Acc.Flu.Sty.Ret.
Accuracy1.00−- 0.010.280.141.00−- 0.020.030.03
Fluency−- 0.011.000.020.00−- 0.021.000.030.00
Style0.280.021.000.140.030.031.000.05
Retranslation0.140.000.141.000.030.000.051.00
Appendix
Table 47: Pairwise cosine similarity between group mean LoRA gradients on D1 (left) and D5 (right), Gemma 3 1B, seed 42. On D1 the matrix is far from orthogonal; on D5 all off-diagonal entries approach zero (see text).
λ\lambdaFilteredRandom ctrl.
MT Base.24.9824.98
0.026.7426.37
0.526.5225.45
1.026.4625.15
2.026.0624.97
3.026.0624.97
Appendix
Table 48: TER vs. λ\lambda sweep on D1 (Gemma 3 1B, seed 42 only, static constraint; cross-seed replication pending) for the two matched-size retraining conditions of Appendix L : Filtered and Random control . λ=0.0≡\lambda{=}0.0\equiv vanilla fine-tuning; MT Base. is the shared MT baseline. Best (lowest) TER per condition in bold .
More in this topic

Low-Resource MT

Low-resource machine translation develops translation systems for languages with scarce parallel text, tools, or standardized evaluation.

See all 87 papers

Explore similar work

CardsList
May 20, 2026

Smarter edits? Post-editing with error highlights and translation suggestions

As MT quality increases, interest in enhanced post-editing features such as QE-derived error highlights is growing, yet evidence for their usefulness remains limited. In this work, we explore the usefulness of LLM-derived error highlights and correction suggestions based on automatic post-editing (APE). We conduct a study where professional translators (En-Nl) post-edit translations using APE error highlights and correction suggestions and compare productivity, quality and user experience to regular PE and PE with QE-derived highlights. While no condition yielded productivity or quality gains compared to regular PE, APE highlights were better received than QE-derived highlights, and correction suggestions improved overall user experience.

Fleur V. J. van Tellingen, Gautam Ranka, Dora Žugčić +4

1Leiden University Centre for Linguistics · 2Visvesvaraya National Institute of Technology · Department of Bionanoscience, Faculty of Applied Sciences, Delft University of Technology +2

Machine Translation
AskPDF
Sep 19, 2026

Diagnose, Then Repair: A Two-Stage MQM-Guided Post-Editing Framework for Domain-Specific Machine Translation

LLM-based machine translation evaluation can closely match human judgments, but in practice it remains largely diagnostic, with the signals rarely translating into direct quality improvements under real production constraints. We propose a two-stage, evaluator-guided automatic post-editing framework that turns MQM-style evaluation into targeted repairs: a retrieval-augmented LLM evaluator outputs structured, span-level MQM diagnoses under an explicit edit contract, and a separate LLM post-editor applies minimal edits restricted to those diagnoses. This separation improves controllability and reduces paraphrastic drift compared to one-stage "judge-and-refine" baselines. In a systematic study involving seven LLMs spanning three model providers and seven languages, our best configuration consistently improves both COMET-22 and COMETKiwi scores over one-stage post-edit methods, while the evaluator's error spans and severities show strong agreement with human MQM annotations and human editor preferences.

Ji Hun Wang, Siyu Wu

Amazon

Machine TranslationMT Evaluation
AskPDF
Aug 17, 2026

IndicQE-APE: A Consolidated Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages

Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and language pairs on one footing. We consolidate the WMT 2020-2024 shared-task lineage with an extended English-Malayalam resource into IndicQE-APE: 126,754126{,}754 instances over nine directional pairs, with up to four label types aligned on the same segment, a direct assessment, a human post-edit, word-level tags and an error explanation, and a test set stratified over four difficulty axes. We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE. Two of the axes are defined partly on direct assessment and select a compressed slice of it. Segments whose segment-level and token-level signals disagree are ranked below equally scored segments of the same language. Four-shot prompting costs every model at or below 3.43.4B both correlation and output-format compliance. Unedited MT beats every APE system we run on three of the four pairs. The benchmark (https://huggingface.co/datasets/surrey-nlp/IndicQE-APE) and code (https://github.com/surrey-nlp/IndicQE-APE) are released.

Diptesh Kanojia, Archchana Sindhujan, Sourabh Deoghare +14

University of Surrey · 2IIT Bombay · University of Oslo +9

Low-Resource Language ProcessingMT Evaluation
AskPDF