cs.CLSep 25, 2026
SaveImproving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting
Organizations: Department of Advanced Computing Sciences, Maastricht University · Warsaw University of Technology · IDEAS Research Institute
Abstract
Multimodal Machine Translation aims to incorporate additional signal from non-textual modalities to improve translations by resolving ambiguities. While models, through multimodal fusion, are able to accept images related to the source text, they can ignore this information. Therefore, increasing their visual sensitivity remains an active research area. In this work, we introduce a training method, Metric-based Loss Weighting, that improves visual grounding of translations by increasing the loss function for tokens that benefit from the accompanying image. We identify these tokens using the Point-wise Cross-mutual Information (PCXMI) metric, which compares the model's output probabilities with and without visual context. We introduce a Congruency-based PCXMI metric and experimentally show that both metrics working in combination yield the best results. We evaluate our method by fine-tuning three pretrained Multimodal Large Language Models on the task of Image-guided Machine Translation for three language directions. Metric-based Loss Weighting outperforms other tested methods on the CoMMuTE contrastive dataset, improving accuracy by up to more than 7 percentage points compared to standard fine-tuning, while maintaining strong general translation performance.
Figures & tables
Figure 1: The illustration of Metric-based Loss Weighting: (1) Calculate the image dependence metric for each token (PCXMI), based on the model’s output probabilities with and without an image (or with an incongruent image); (2) Fine-tune the model with increased loss (by ) for the identified tokens (with PCXMI larger than ).
| Model | test 2016 | test 2017 | MSCOCO | Average | CoMMuTE | ||||
| BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET | Accuracy | |
| Qwen3-VL 4B | 38.51 | 85.56 | 32.95 | 84.90 | 29.94 | 81.24 | 33.80 | 83.90 | 74.00% |
| Fine-tuning | 42.61 0.12 | 86.66 0.06 | 38.08 0.38 | 85.98 0.06 | 35.22 0.45 | 82.70 0.13 | 38.64 0.18 | 85.11 0.05 | 71.67 0.19% |
| VMLM | 42.52 0.19 | 86.79 0.03 | 38.48 0.43 | 86.34 0.06 | 36.47 0.09 | 82.95 0.17 | 39.16 0.06 | 85.36 0.05 | 71.67 0.38% |
| Metric Fine-tuning | 42.57 0.22 | 86.47 0.13 | 38.43 0.10 | 85.99 0.12 | 36.18 0.16 | 82.86 0.18 | 39.06 0.15 | 85.10 0.11 | 72.89 0.44% |
| Word Dropout | 42.20 0.19 | 86.62 0.06 | 37.17 0.19 | 85.80 0.01 | 35.00 0.22 | 82.69 0.16 | 38.13 0.18 | 85.03 0.04 | 76.89 0.44% |
Table 1: Results for English-to-German in terms of BLEU and COMET on the Multi30k testsets (separately and averaged over the testsets) and the accuracy on the CoMMuTE contrastive dataset. The results are averaged over three separate runs, and shows the standard error. Best results for each base model in each column are marked in bold , and the second best are underlined .
| Model | test 2016 | test 2017 | MSCOCO | Average | CoMMuTE | ||||
| BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET | Accuracy | |
| Qwen3-VL 4B | 43.77 | 86.04 | 43.47 | 86.31 | 43.24 | 83.69 | 43.49 | 85.35 | 75.00% |
| Fine-tuning | 67.03 0.36 | 89.81 0.05 | 61.62 0.13 | 89.25 0.10 | 50.90 0.43 | 85.37 0.12 | 59.85 0.08 | 88.14 0.03 | 81.17 0.37% |
| VMLM | 67.37 0.25 | 89.71 0.05 | 61.00 0.19 | 89.23 0.10 | 50.71 0.57 | 85.10 0.07 | 59.69 0.21 | 88.01 0.05 | 81.71 0.39% |
| Metric Fine-tuning | 66.83 0.41 | 89.43 0.12 | 60.98 0.11 | 88.97 0.02 | 49.73 0.09 | 84.71 0.15 | 59.18 0.13 | 87.70 0.09 | 81.17 0.19% |
| Word Dropout | 66.46 0.31 | 89.58 0.09 | 60.98 0.05 | 89.10 0.02 | 51.01 0.44 | 85.15 0.04 | 59.48 0.27 | 87.94 0.04 | 84.42 0.68% |
Table 2: Results for English-to-French in terms of BLEU and COMET on the Multi30k testsets (separately and averaged over the testsets) and the accuracy on the CoMMuTE contrastive dataset. The results are averaged over three separate runs, and shows the standard error. Best results for each base model in each column are marked in bold , and the second best are underlined .
| Model | test 2016 | CoMMuTE | |
| BLEU | COMET | Accuracy | |
| Qwen3-VL 4B | 23.12 | 83.43 | 67.21% |
| Fine-tuning | 35.82 0.25 | 88.63 0.05 | 74.35 0.37% |
| VMLM | 35.66 0.08 | 88.46 0.10 | 74.68 0.50% |
| Metric Fine-tuning | 35.19 0.16 | 88.39 0.06 | 75.54 1.07% |
| Word Dropout | 35.72 0.20 | 88.62 0.09 | 75.43 0.66% |
Table 3: Results for English-to-Czech in terms of BLEU and COMET on the Multi30k test 2016 testset and the accuracy on the CoMMuTE contrastive dataset. The results are averaged over three separate runs, and shows the standard error. Best results for each base model in each column are marked in bold , and the second best are underlined .
Figure 2: Two examples from the English-to-German CoMMuTE dataset with the per-token values of PCXMI and CPCXMI metrics for the Qwen3-VL 4B model. In red we mark the ambiguous words in the source side and the corresponding words in the target side.
Figure 3: Distributions of BLEU scores of Fine-tuning and Metric-based Loss Weighting on the MSCOCO testset for the English-to-French language direction.
| Model | Average | CoMMuTE | |
| BLEU | COMET | Accuracy | |
| English-to-German | |||
| Qwen3-VL 4B | 33.80 | 83.90 | 74.00% |
| Fine-tuning | 38.64 0.18 | 85.11 0.05 | 71.67 0.19% |
| Word Dropout | 38.13 0.18 | 85.03 0.04 | 76.89 0.44% |
| Metric Weighting | 38.30 0.15 | 85.34 0.07 | 77.00 0.33% |
Table 4: Results for combined Metric-based Loss Weighting and Word Dropout in terms of BLEU and COMET averaged over the test sets (and three separate runs) for English-to-German and English-to-French, and for test 2016 for English-to-Czech, and CoMMuTE accuracy.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Epoch Time |
| Qwen3-VL 4B | |
| Fine-tuning | 1:43 |
| VMLM | 3:30 |
| Metric Fine-tuning* | 0:20 |
| Word Dropout | 1:43 |
| Degradation | 1:43 |
Table 5: Average training times per epoch for the tested models (in hours:minutes). *The time for Metric-based Fine-tuning is only for the second stage of the training.
| Dataset | # Examples |
| Multi30k | |
| Train | 29,000 |
| Validation | 1014 |
| Test 2016 | 1000 |
| Test 2017 | 1000 |
| MSCOCO | 461 |
Table 6: The sizes of the used datasets in terms the number of examples.
| Hyper-parameter | Value |
| Optimizer | AdamW |
| Learning Rate | 0.0001* |
| LR Scheduler | Cosine |
| LR Warmup Steps | 500 |
| Weight Decay | 0.01 |
| Max Grad Norm | 1.0 |
Table 7: The shared hyper-parameters of all tested fine-tuning methods. *We tuned the Learning Rate hyper-parameter for the second stage of the Metric-based Fine-tuning method.
| Model | En-De | En-Fr | En-Cs |
| Qwen3-VL 4B | |||
| VMLM (p) | 0.3 | 0.1 | 0.1 |
| Metric Fine-tuning (lr) | 1e-4 | 1e-4 | 1e-5 |
| Word Dropout (p) | 0.1 | 0.1 | 0.1 |
| Degradation (p) | 0.3 | 0.2 | 0.1 |
| Metric Weighting ( ) | 5 | 6 | 6 |
Table 8: Found best hyper-parameters for each method and language direction. For VMLM, Word Dropout and Degradation, the hyper-parameter is the probability of masking or dropout p. For Metric-based Fine-tuning, the hyper-parameter is the learning rate (lr) of the second stage training. For Metric-based Weighting, the hyper-parameter is the threshold .
Figure 4: The results of applying the Metric-based Loss Weighting method to Qwen3-VL 4B model on the English-to-German language direction with different values of hyper-parameters. The results are in terms of the BLEU and COMET scores averaged over all used testsets, and the accuracy on CoMMuTE dataset. We averaged the results over three separate runs and error bars show the standard error.
Figure 5: Kernel Density Estimate visualization of distributions of metrics (PCXMI, CPCXMI, and the maximum of both) in the Multi30k training dataset for each tested model and language direction. We take the maximum token-level metric for each example in the dataset.
Figure 6: Kernel Density Estimate visualization of distributions of Maximum of PCXMI and CPCXMI in the Multi30k training and CoMMuTE datasets for each tested model and language direction. We take the maximum token-level metric for each example in the dataset.
Figure 7: Selected examples from the English-to-German CoMMuTE dataset with the per-token values of PCXMI and CPCXMI metrics for the Qwen3-VL 4B model. In red we mark the ambiguous words the source side and the corresponding words on the target side.
Figure 8: Selected examples with the highest difference in BLEU between Fine-tuning and Metric-based Loss Weighting (Fine-tuning scoring higher) in the MSCOCO testset on the English-to-French language direction of Qwen3-VL 8B. We also report BLEU and COMET scores for the translations by the methods (and their differences), and our description of errors.
Figure 9: Selected examples with the highest difference in BLEU between Metric-based Loss Weighting and Fine-tuning (Weighting scoring higher) in the MSCOCO testset on the English-to-French language direction of Qwen3-VL 8B. We also report BLEU and COMET scores for the translations by the methods (and their differences), and our description of errors.
| Model | test 2016 | test 2017 | MSCOCO | Average | CoMMuTE | ||||
| BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET | Accuracy | |
| English-to-German | |||||||||
| Qwen3-VL 4B | 38.51 | 85.56 | 32.95 | 84.90 | 29.94 | 81.24 | 33.80 | 83.90 | 74.00% |
| Fine-tuning | 42.61 0.12 | 86.66 0.06 | 38.08 0.38 | 85.98 0.06 | 35.22 0.45 | 82.70 0.13 | 38.64 0.18 | 85.11 0.05 | 71.67 0.19% |
| Word Dropout | 42.20 0.19 | 86.62 0.06 | 37.17 0.19 | 85.80 0.01 | 35.00 0.22 | 82.69 0.16 | 38.13 0.18 | 85.03 0.04 | 76.89 0.44% |
| Metric Weighting | 41.91 0.33 | 86.80 0.04 | 37.29 0.14 | 86.14 0.04 | 35.71 0.19 | 83.09 0.13 | 38.30 0.15 | 85.34 0.07 | 77.00 0.33% |
Table 9: Results for combined Metric-based Loss Weighting and Word Dropout for all testsets in English-to-German and English-to-French. The results are averaged over three separate runs.