Text-Only vs. Image-Aware VLM Judges for Manga Translation Evaluation

Subaru Kimura; Jun Suzuki

cs.CLOct 13, 2026

Text-Only vs. Image-Aware VLM Judges for Manga Translation Evaluation

Authors: Subaru Kimura, Jun Suzuki

Organizations: Tohoku University

Abstract

In manga translation, visual context outside speech-bubble text, including speaker identities, referents, and scene tone, is depicted in page images, making it natural to refer to images during translation evaluation. However, most conventional manga translation evaluations rely solely on text-only automatic metrics, leaving the impact of image input insufficiently explored. We propose a practical validation framework that employs the same vision-language model (VLM) as both a text-only judge (TOJ) and an image-aware judge (IAJ) to investigate evaluation divergences associated with image-aware evaluation procedures. We use a Japanese-English manga dataset of official translations aligned at the page and speech-bubble levels. First, adding page images to TOJ improved the primary VLM's detection of deliberately injected, visually dependent degradations. Second, we analyzed evaluation divergences between TOJ and IAJ, finding different error counts and error-category distributions. Human verification did not reveal a consistent preference between TOJ and IAJ.

Figures & tables

Figure 1: Overview of the proposed validation framework.

Degradation Type	Description
speaker_or_ addressee	Confuses speaker or addressee, shifting subjects or honorifics
referent_or_ object	Translates referenced objects/entities in image as something else
tone_emotion_or_ style	Translates with tone or style contradicting facial expression or scene
action_or_ state	Translates with actions or physical states contradicting depicted artwork

Table 1: Degradation operation types used in Experiment 1 to create translations contradicting manga page image content.

Condition	Page Image	Grounding	Mean (%)	5-run Range (%)	Mean flags
TOJ	None	None	57.9%	56.0–58.7%	1.11
IAJ-blank	Blank image	Yes	52.0%	50.7–53.3%	1.03
IAJ-minimal	Actual page	None	81.6%	80.0–82.7%	1.40
IAJ-full	Actual page	Yes	80.5%	78.7–82.7%	1.20

Table 2: Experiment 1 results. Semantic match rates are summarized by their mean and range across five runs. Mean flags denotes the average number of reported errors per instance.

	Semantic match rate			Mean flags
MQM Judge	TOJ	IAJ-full	Difference	TOJ	IAJ-full
gemini-3-flash-preview	57.9% (56.0–58.7)	80.5% (78.7–82.7)	+22.7 pp	1.11	1.20
gpt-5.4-mini	24.8% (24.0–25.3)	30.7% (30.7–30.7)	+5.9 pp	1.05	0.89
qwen3-vl-32b-instruct	45.6% (44.0–48.0)	59.2% (57.3–61.3)	+13.6 pp	1.43	1.53

Table 3: Experiment 1 cross-model comparison. Semantic match rates are five-run means with ranges in parentheses. Differences are IAJ-full minus TOJ, computed before rounding. Mean flags is defined as in Table 2 .

Panel of Figure 2 — Figure 2: Representative example ( bj_vol1_113 ) of degraded translation contradicting manga image content missed by TOJ but detected by IAJ-full. Left: JA/EN page images; Right: degraded segment, degradation intent, and IAJ-full error explanation. The EN page image is shown for illustration only and was not provided to IAJ. The patient’s beard and the doctor’s face clarify the referent error changing “HIS” to “MY”. Give My Regards to Black Jack / SHUHO SATO

All 180 Pages
Metric	TOJ	IAJ-full
Total error flags	318	262
Mean / page	1.77	1.46
Major error flags	36	24
Mean major / page	0.20	0.13
Minor error flags	268	233

Table 4: Automatic evaluation results reported by judges on official translations across 180 pages (based on gemini-3-flash-preview ). The table includes total error flags, severity counts, mean model-reported quality_score , and mean score deficit (

100-S_{p}

). Total error flags include major , minor , and neutral .

Category	TOJ-only	IAJ-only	TOJ-only Share
Overall	152	96	61.3%
meaning_accuracy	42	32	56.8%
naturalness_and_register	30	19	61.2%
omission_or_addition	31	17	64.6%
tone_and_emotion	19	10	65.5%
terminology_and_named_entities	20	12	62.5%

Table 5: Breakdown of TOJ/IAJ unique error flags by MQM category across 180 pages.

Individual votes ( $n=240$ )
Metric	Value
Annotators / Verified Pages	6 / 40
Prefer IAJ / TOJ / Tie	73 / 87 / 80
Percentage	30.4 / 36.3 / 33.3%
Medium or high confidence	228 (95.0%)
Mean Pairwise Agreement	0.512

Table 6: Experiment 2: Main judgment agreement and page-level consensus among 6 annotators over 40 TOJ/IAJ-full MQM outputs.

Output validity ( $n=240$ each)	TOJ	IAJ
	Count (%)	Count (%)
Valid	168 (70.0)	172 (71.7)
Partially valid	46 (19.2)	28 (11.7)
Invalid	15 (6.3)	12 (5.0)
Unclear	11 (4.6)	28 (11.7)
Valid or partially valid	214 (89.2)	200 (83.3)

Table 7: Experiment 2 human ratings. Left: output validity and reasons for ties. Right: perceived image usefulness and observed effects on IAJ outputs. Values are counts with percentages in parentheses. IAJ denotes IAJ-full. Positive effects include valid detection, overflagging prevention, or rationale strengthening; negative effects include visual misrecognition/overinterpretation or missed textual issues. Effects are counted per response, with two responses reporting both positive and negative effects.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Base Dataset
Target Manga Work	Give My Regards to Black Jack Vol. 1
Page Images (JA)	196
Page Images (EN)	196
Pages with Established Segment Alignment	180
Total Segment Count (180 pages)	974

Appendix

Table 8: Dataset statistics.

Run	TOJ (%)	IAJ-min. (%)	Diff (pp)	TOJ-only	IAJ-min.-only	McNemar $p$	Cluster $p$	Cluster 95% CI
1	58.7%	81.3%	+22.7	2	19	0.000221	0.000740	[+11.0, +36.1]
2	58.7%	80.0%	+21.3	2	18	0.000403	0.001700	[+8.3, +35.1]
3	56.0%	81.3%	+25.3	1	20	$<0.0001$	$<0.0001$	[+13.3, +38.6]
4	57.3%	82.7%	+25.3	1	20	$<0.0001$	$<0.0001$	[+13.9, +37.8]
5	58.7%	82.7%	+24.0	1	19	$<0.0001$	0.000390	[+11.5, +37.7]

Appendix

Table 9: Significance test results for TOJ vs. IAJ-minimal in Experiment 1. Differences per run are percentage-point differences (IAJ-minimal

-

TOJ). “TOJ-only detected” indicates instances detected only by TOJ; “IAJ-min.-only detected” indicates instances detected only by IAJ-minimal, aggregated across 75 paired degraded translation instances. McNemar

p

provides exact instance-level paired reference values. Cluster

p

represents sign-flip test

p

-values over 54 page-level clusters, and Cluster 95% confidence interval (CI) reflects bootstrap confidence intervals resampling source pages.

Panel of Figure 3 — Figure 3: Representative examples ( bj_vol1_114 , bj_vol1_116 ) of degraded translations contradicting manga image content in Experiment 1. Format matches Figure 2 (JA/EN page images + degraded segment + degradation intent + IAJ-full error explanation). The EN page images are shown for illustration only and were not provided to IAJ. Give My Regards to Black Jack / SHUHO SATO

Figure 4: Annotation UI screenshot. Left panel: JA/EN page images and comparison of source and translation segments. Middle panel: TOJ/IAJ flags. Right panel: annotation fields including main judgment ( prefer_IAJ / prefer_TOJ / tie ), confidence, page-level output validity, image effect, image usefulness, and comments. Images expand on click.

Model	Condition	Pearson $r$	Spearman $\rho$
gemini-3-flash-preview (Main)	TOJ	0.402	0.762
gemini-3-flash-preview (Main)	IAJ-full	0.348	0.717
gpt-5.4-mini (Auxiliary)	TOJ	0.557	0.873
gpt-5.4-mini (Auxiliary)	IAJ-full	0.752	0.881
qwen3-vl-32b-instruct (Auxiliary)	TOJ	0.467	0.803
qwen3-vl-32b-instruct (Auxiliary)	IAJ-full	0.391	0.693

Appendix

Table 10: Page-level correlation between quality_score and sign-flipped normalized MQM penalty (

-E_{p}^{\mathrm{word}}

). Each model and condition has

n=180

; pooled values have

n=1080

Model	IAJ Higher	TOJ Higher	Tied	Mean IAJ	Mean TOJ	Mean Diff
gemini-3-flash-preview	81 (45.0%)	32 (17.8%)	67 (37.2%)	91.9	89.8	+2.2
gpt-5.4-mini	90 (50.0%)	43 (23.9%)	47 (26.1%)	87.9	86.4	+1.5
qwen3-vl-32b-instruct	102 (56.7%)	24 (13.3%)	54 (30.0%)	85.5	80.1	+5.4

Appendix

Table 11: Experiment 2 score summary across the primary and auxiliary models. For 180 official translation pages, the table shows page counts where the IAJ-full model-reported score exceeded TOJ, vice versa, ties, and mean model-reported scores ( quality_score ). Mean difference reflects average unrounded score difference (IAJ

-

TOJ).

All 180 Pages
Metric	gpt TOJ	gpt IAJ-full	qwen TOJ	qwen IAJ-full
Total error flags	463	444	568	547
Mean / page	2.57	2.47	3.16	3.04
Major error flags	136	123	212	146
Mean major / page	0.76	0.68	1.18	0.81
Minor error flags	301	285	348	375

Appendix

Table 12: Error volume reported by auxiliary models on 180 official translation pages. Total error flags include major , minor , and neutral . gpt represents gpt-5.4-mini ; qwen represents qwen3-vl-32b-instruct .

Metric / Page Group	Value / Pages
Micro-Average Coverage (166 / 318 flags)	52.2%
Page-Level Macro-Average Coverage (n=160)	53.1%
Page-Level Median Coverage	50.0%
100% Coverage (Fully Retained) Pages	54 pages (33.8%)
Partial Coverage ( $0<c_{p}<1$ ) Pages	61 pages (38.1%)
0% Coverage (Fully Suppressed) Pages	45 pages (28.1%)

Appendix

Table 13: Coverage of TOJ error flags by IAJ-full and page-level distributions.

Multimodal MT

See all 11 papers

JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
Apr 1, 2026Issa Sugiura, Koki Maeda, Shuhei Kurita +3VLM Evaluation Benchmark Construction
Kyoto University · NII LLMC · Institute of Science Tokyo +2
It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
Sep 29, 2026Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem +1VLM Evaluation Figurative Language Understanding
Faculty of Data and Decision Sciences, Technion
Semantic Similarity is a Spurious Measure of Comic Understanding: Lessons Learned from Hallucinations in a Benchmarking Experiment
Mar 2, 2026Christopher Driggers-Ellis, Nachiketh Tibrewal, Rohit Bogulla +4VLM Evaluation VLM Hallucination
University of Florida, Gainesville, FL 32611

Apr 1, 2026J/K move · Enter open · S save

JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation

Issa Sugiura, Koki Maeda, Shuhei Kurita, Yusuke Oda+2

Kyoto University · NII LLMC · Institute of Science Tokyo+2

VLM Evaluation Benchmark Construction Visual Question Answering Multilingual VLM Evaluation

Open paper AskPDF arXiv

Reliable evaluation is essential for the development of vision-language models (VLMs). However, Japanese VQA benchmarks have undergone far less iterative refinement than their English counterparts. As a result, many existing benchmarks contain issues such as ambiguous questions, incorrect answers, and instances that can be solved without visual grounding, undermining evaluation reliability and leading to misleading conclusions in model comparisons. To address these limitations, we introduce JAMMEval, a refined collection of Japanese benchmarks for reliable VLM evaluation. It is constructed by systematically refining seven existing Japanese benchmark datasets through two rounds of human annotation, improving both data quality and evaluation reliability. In our experiments, we evaluate open-weight and proprietary VLMs on JAMMEval and analyze the capabilities of recent models on Japanese VQA. We further demonstrate the effectiveness of our refinement by showing that the resulting benchmarks yield evaluation scores that better reflect model capability, exhibit lower run-to-run variance, and improve the ability to distinguish between models of different capability levels. We release our dataset and code to advance reliable evaluation of VLMs.

Abstract

Multimodal MT

Explore similar work

JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation

It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

Semantic Similarity is a Spurious Measure of Comic Understanding: Lessons Learned from Hallucinations in a Benchmarking Experiment